Skip to contentSkip to conversation

x86: Select HARDIRQS_SW_RESEND on x86

RFC Applied Hans de Goede 4 participants 13 messages Jan 23, 2020 – Mar 12, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC v2 1/1 x86: Select HARDIRQS_SW_RESEND on x86

Modern x86 laptops are starting to use GPIO pins as interrupts more and more, e.g. touchpads and touchscreens have almost all moved away from PS/2 and USB to using I2C with a GPIO pin as interrupt. Modern x86 laptops also have almost all moved to using s2idle instead of using the system S3 ACPI power state to suspend.

The Intel and AMD pinctrl drivers do not define irq_retrigger handlers for the irqchips they register, this is causing edge triggered interrupts which happen while suspended using s2idle to get lost.

One specific example of this is the lid switch on some devices, lid switches used to be handled by the embedded-controller, but now the lid open/closed sensor is sometimes directly connected to a GPIO pin. On most devices the ACPI code for this looks like this:

Method (_E00, ...) {
	Notify (LID0, 0x80) // Status Change
}

Where _E00 is an ACPI event handler for changes on both edges of the GPIO connected to the lid sensor, this event handler is then combined with an _LID method which directly reads the pin. When the device is resumed by opening the lid, the GPIO interrupt will wake the system, but because the pinctrl irqchip doesn't have an irq_retrigger handler, the Notify will not happen. This is not a problem in the case the _LID method directly reads the GPIO, because the drivers/acpi/button.c code will call _LID on resume anyways.

But some devices have an event handler for the GPIO connected to the lid sensor which looks like this:

Method (_E00, ...) {
	if (LID_GPIO == One)
		LIDS = One
	else
		LIDS = Zero
	Notify (LID0, 0x80) // Status Change
}

And the _LID method returns the cached LIDS value, since on open we do not re-run the edge-interrupt handler when we re-enable IRQS on resume (because of the missing irq_retrigger handler), _LID now will keep reporting closed, as LIDS was never changed to reflect the open status, this causes userspace to re-resume the laptop again shortly after opening the lid.

The Intel GPIO controllers do not allow implementing irq_retrigger without emulating it in software, at which point we are better of just using the generic HARDIRQS_SW_RESEND mechanism rather then re-implementing software emulation for this separately in aprox. 14 different pinctrl drivers.

This commit selects HARDIRQS_SW_RESEND solving the problem of edge-triggered GPIO interrupts not being re-triggered on resume when they were triggered during suspend (s2idle) and/or when they were the cause of the wakeup.

---
I'm sending this out as a RFC since I'm not %100 sure this is the best solution and it seems like a somewhat big change to make.

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...
---
Changes in v2:
-v2 is really a resend because I forgot to add the pinctrl people to the Cc -While at it also fix some typos in the commit message

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index c1cbfc7b3ae8..8f8128047b49 100644
@@ -118,6 +118,7 @@ config X86
 	select GENERIC_IRQ_PROBE
 	select GENERIC_IRQ_RESERVATION_MODE
 	select GENERIC_IRQ_SHOW
+	select HARDIRQS_SW_RESEND
 	select GENERIC_PENDING_IRQ		if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select GENERIC_STRNCPY_FROM_USER
Signature · 1 line
2.24.1
#1 lore raw

Hans,

Quoted 5 lines · Hans de Goede

The Intel GPIO controllers do not allow implementing irq_retrigger without emulating it in software, at which point we are better of just using the generic HARDIRQS_SW_RESEND mechanism rather then re-implementing software emulation for this separately in aprox. 14 different pinctrl drivers.

Indeed.

Quoted 2 lines

I'm sending this out as a RFC since I'm not %100 sure this is the best solution and it seems like a somewhat big change to make.

It's not that bad. The only affected interrupt chips on x86 should be secondary interrupt chips like the GPIO controller.

ioapic/msi/... have irq_retrigger() functionality, so it won't do the software resend.

I just need to stare at the legacy PIC and the virt stuff.

Quoted 2 lines

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...

Yes. Let me stare at the couple of weird irqchips which might get surprised. I'll teach them not to do that :)

Thanks,

tglx

#2 lore raw

Hi Thomas,

Quoted 20 lines · Thomas Gleixner

Hans,

Quoted 5 lines · Hans de Goede

The Intel GPIO controllers do not allow implementing irq_retrigger without emulating it in software, at which point we are better of just using the generic HARDIRQS_SW_RESEND mechanism rather then re-implementing software emulation for this separately in aprox. 14 different pinctrl drivers.

Indeed.

Quoted 2 lines

I'm sending this out as a RFC since I'm not %100 sure this is the best solution and it seems like a somewhat big change to make.

It's not that bad. The only affected interrupt chips on x86 should be secondary interrupt chips like the GPIO controller.

ioapic/msi/... have irq_retrigger() functionality, so it won't do the software resend.

Quoted 7 lines

I just need to stare at the legacy PIC and the virt stuff.

Quoted 2 lines

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...

Yes. Let me stare at the couple of weird irqchips which might get surprised. I'll teach them not to do that :)

I know that you are very busy, still I'm wondering is there any progress on this ?

Regards,

Hans

#3 lore raw
Quoted 10 lines · Hans de Goede
Quoted 7 lines

I just need to stare at the legacy PIC and the virt stuff.

Quoted 2 lines

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...

Yes. Let me stare at the couple of weird irqchips which might get surprised. I'll teach them not to do that :)

I know that you are very busy, still I'm wondering is there any progress on this ?

Bah. That fell through the cracks, but actually I looked at this due to the PCI-E AER wreckage. So yes, this is fine, but we want:

https://lkml.kernel.org/r/[email protected]
https://lkml.kernel.org/r/[email protected]

if we want to backport this to stable.

Thanks,

tglx

#4 lore raw

Hi,

Quoted 12 lines · Thomas Gleixner
Quoted 10 lines · Hans de Goede
Quoted 7 lines

I just need to stare at the legacy PIC and the virt stuff.

Quoted 2 lines

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...

Yes. Let me stare at the couple of weird irqchips which might get surprised. I'll teach them not to do that :)

I know that you are very busy, still I'm wondering is there any progress on this ?

Quoted 7 lines

Bah. That fell through the cracks, but actually I looked at this due to the PCI-E AER wreckage. So yes, this is fine, but we want:

https://lkml.kernel.org/r/[email protected]
https://lkml.kernel.org/r/[email protected]

if we want to backport this to stable.

So far I have seen a few, but not a lot of devices which need this, so I'm not 100% sure what to do here.

Do you consider this change safe / suitable for stable if those 2 patches are backported and applied first?

Regards,

Hans

#5 lore raw
Quoted 19 lines · Hans de Goede
Quoted 18 lines · Thomas Gleixner
Quoted 10 lines · Hans de Goede
Quoted 7 lines

I just need to stare at the legacy PIC and the virt stuff.

Quoted 2 lines

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...

Yes. Let me stare at the couple of weird irqchips which might get surprised. I'll teach them not to do that :)

I know that you are very busy, still I'm wondering is there any progress on this ?

Bah. That fell through the cracks, but actually I looked at this due to the PCI-E AER wreckage. So yes, this is fine, but we want:

https://lkml.kernel.org/r/[email protected]
https://lkml.kernel.org/r/[email protected]

Quoted 7 lines
Quoted 1 line

if we want to backport this to stable.

So far I have seen a few, but not a lot of devices which need this, so I'm not 100% sure what to do here.

Do you consider this change safe / suitable for stable if those 2 patches are backported and applied first?

I think so. The two patches are on my list for backports anyway, but I wanted to give them some time to simmer.

Thanks,

tglx

#6 lore raw

Hi,

Quoted 22 lines · Thomas Gleixner
Quoted 21 lines · Hans de Goede
Quoted 19 lines · Thomas Gleixner
Quoted 10 lines · Hans de Goede
Quoted 7 lines

I just need to stare at the legacy PIC and the virt stuff.

Quoted 2 lines

Also maybe we should add a Cc: [email protected] ??? This seems like somewhat a big change for that but it does solve some real issues...

Yes. Let me stare at the couple of weird irqchips which might get surprised. I'll teach them not to do that :)

I know that you are very busy, still I'm wondering is there any progress on this ?

Bah. That fell through the cracks, but actually I looked at this due to the PCI-E AER wreckage. So yes, this is fine, but we want:

   https://lkml.kernel.org/r/[email protected]
   https://lkml.kernel.org/r/[email protected]

if we want to backport this to stable.

Quoted 8 lines
Quoted 5 lines

So far I have seen a few, but not a lot of devices which need this, so I'm not 100% sure what to do here.

Do you consider this change safe / suitable for stable if those 2 patches are backported and applied first?

I think so. The two patches are on my list for backports anyway, but I wanted to give them some time to simmer.

OK, I'll submit this patch for stable then once your backports have landed.

Regards,

Hans

merge bot Applied
#7 lore raw
In reply to Hans de Goede
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     17e5888e4e180b45af7bafe7f3a86440d42717f3
Gitweb:        https://git.kernel.org/tip/17e5888e4e180b45af7bafe7f3a86440d42717f3
Author:        Hans de Goede <[email protected]>
AuthorDate:    Thu, 23 Jan 2020 22:02:42 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 11 Mar 2020 22:39:39 +01:00

x86: Select HARDIRQS_SW_RESEND on x86

Modern x86 laptops are starting to use GPIO pins as interrupts more and more, e.g. touchpads and touchscreens have almost all moved away from PS/2 and USB to using I2C with a GPIO pin as interrupt. Modern x86 laptops also have almost all moved to using s2idle instead of using the system S3 ACPI power state to suspend.

The Intel and AMD pinctrl drivers do not define irq_retrigger handlers for the irqchips they register, this is causing edge triggered interrupts which happen while suspended using s2idle to get lost.

One specific example of this is the lid switch on some devices, lid switches used to be handled by the embedded-controller, but now the lid open/closed sensor is sometimes directly connected to a GPIO pin. On most devices the ACPI code for this looks like this:

Method (_E00, ...) {
	Notify (LID0, 0x80) // Status Change
}

Where _E00 is an ACPI event handler for changes on both edges of the GPIO connected to the lid sensor, this event handler is then combined with an _LID method which directly reads the pin. When the device is resumed by opening the lid, the GPIO interrupt will wake the system, but because the pinctrl irqchip doesn't have an irq_retrigger handler, the Notify will not happen. This is not a problem in the case the _LID method directly reads the GPIO, because the drivers/acpi/button.c code will call _LID on resume anyways.

But some devices have an event handler for the GPIO connected to the lid sensor which looks like this:

Method (_E00, ...) {
	if (LID_GPIO == One)
		LIDS = One
	else
		LIDS = Zero
	Notify (LID0, 0x80) // Status Change
}

And the _LID method returns the cached LIDS value, since on open we do not re-run the edge-interrupt handler when we re-enable IRQS on resume (because of the missing irq_retrigger handler), _LID now will keep reporting closed, as LIDS was never changed to reflect the open status, this causes userspace to re-resume the laptop again shortly after opening the lid.

The Intel GPIO controllers do not allow implementing irq_retrigger without emulating it in software, at which point we are better of just using the generic HARDIRQS_SW_RESEND mechanism rather then re-implementing software emulation for this separately in aprox. 14 different pinctrl drivers.

Select HARDIRQS_SW_RESEND to solve the problem of edge-triggered GPIO interrupts not being re-triggered on resume when they were triggered during suspend (s2idle) and/or when they were the cause of the wakeup.

This requires

008f1d60fe25 ("x86/apic/vector: Force interupt handler invocation to irq context") c16816acd086 ("genirq: Add protection against unsafe usage of generic_handle_irq()")

to protect the APIC based interrupts from being wreckaged by a software resend.

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index beea770..9128932 100644
@@ -128,6 +128,7 @@ config X86
 	select GENERIC_GETTIMEOFDAY
 	select GENERIC_VDSO_TIME_NS
 	select GUP_GET_PTE_LOW_HIGH		if X86_PAE
+	select HARDIRQS_SW_RESEND
 	select HARDLOCKUP_CHECK_TIMESTAMP	if X86_64
 	select HAVE_ACPI_APEI			if ACPI
 	select HAVE_ACPI_APEI_NMI		if ACPI
#8 lore raw
Quoted 4 lines · On Wed, Mar 11, 2020 at 10:42 PM tip-bot2 for Hans de Goede…

select GENERIC_GETTIMEOFDAY
select GENERIC_VDSO_TIME_NS
select GUP_GET_PTE_LOW_HIGH if X86_PAE
+ select HARDIRQS_SW_RESEND

Just help me understand the semantics of this thing...

According to the text in KConfig:

# Tasklet based software resend for pending interrupts on enable_irq() config HARDIRQS_SW_RESEND
bool

According to
commit a4633adcdbc15ac51afcd0e1395de58cee27cf92

[PATCH] genirq: add genirq sw IRQ-retrigger

Enable platforms that do not have a hardware-assisted
hardirq-resend mechanism
to resend them via a softirq-driven IRQ emulation mechanism.

so when enable_irq() is called, if the IRQ is already asserted, it will be distributed in the form of a software irq?

OK I give up I don't understand the semantics of this thing.

Maybe it's because I think of a register where the IRQ line is just a level IRQ bit thing that stays high as long as the IRQ is not handled.

So I suppose it is for any type of transient IRQ such as edge triggered that happened before the system came back online entirely and now the only remnant of it is a bit in the irchip status register?

I see that ARM and ARM64 simply just select this. What happens if you do that and why is x86 not selecting it in general?

Explain it to me and I promise to patch kernel/irq/Kconfig with the explanation so you don't have to give it again.

Yours,
Linus Walleij

#9 lore raw

Hi,

Quoted 30 lines · Linus Walleij
Quoted 4 lines · On Wed, Mar 11, 2020 at 10:42 PM tip-bot2 for Hans de Goede…

select GENERIC_GETTIMEOFDAY
select GENERIC_VDSO_TIME_NS
select GUP_GET_PTE_LOW_HIGH if X86_PAE
+ select HARDIRQS_SW_RESEND

Just help me understand the semantics of this thing...

According to the text in KConfig:

# Tasklet based software resend for pending interrupts on enable_irq() config HARDIRQS_SW_RESEND
bool

According to
commit a4633adcdbc15ac51afcd0e1395de58cee27cf92

[PATCH] genirq: add genirq sw IRQ-retrigger

Enable platforms that do not have a hardware-assisted
hardirq-resend mechanism
to resend them via a softirq-driven IRQ emulation mechanism.

so when enable_irq() is called, if the IRQ is already asserted, it will be distributed in the form of a software irq?

OK I give up I don't understand the semantics of this thing.

Quoted 8 lines

Maybe it's because I think of a register where the IRQ line is just a level IRQ bit thing that stays high as long as the IRQ is not handled.

So I suppose it is for any type of transient IRQ such as edge triggered that happened before the system came back online entirely and now the only remnant of it is a bit in the irchip status register?

The way I understand it is like this:

1. We have an edge triggered IRQ from a peripheral to a GPIO controller

2. We have a level triggered IRQ from the GPIO controller to the "root" IRQ controller.

3. With modern x86 suspend, we do not really put the entire system in a firmware-controller suspend state, instead the CPU is halted until any IRQ happens; and there is a power-management micro-controller which shuts various things down while the CPU is halted leading to similar power consumption as old S3 suspend.

The combination of these 3 means that we must ack the edge triggered IRQ at the GPIO controller level even while suspended (we briefly wake up for this) to make the level-triggered IRQ coming from the GPIO controller low so that we can go back to sleep.

When this happens we record in the kernel IRQ tracking data for the edge-triggered IRQ tied to the GPIO (there are no "remnants" of it in any chuip registers), that the IRQ needs to be replayed on resume. Some IRQ controllers allow writing a register to retrigger the IRQ without the level on the external GPIO actually changing.

Some IRQ controllers do not allow this, in this case we need to emulate this retriggering in software, this is what the HARDIRQS_SW_RESEND option is for, this handles this in a generic way, so that we do not have to add emulation to every IRQ-chip driver where the hardware lacks a "retrigger" register.

Quoted 2 lines

I see that ARM and ARM64 simply just select this. What happens if you do that and why is x86 not selecting it in general?

Erm, "selecting it in general" (well at least on x86) is what this patch is doing. But I guess you mean why is this not selected on all architectures? If I've understood tglx correctly the HARDIRQS_SW_RESEND option is incompatible with some irq-chip drivers. I think it was incompatible with using nested threaded handlers are some such...

tglx can probably explain this way better then I can :)

Regards,

Hans

#10 lore raw

On Thu, Mar 12, 2020 at 2:49 PM Hans de Goede <[email protected]> wrote:

[Me]

Quoted 5 lines
Quoted 2 lines

I see that ARM and ARM64 simply just select this. What happens if you do that and why is x86 not selecting it in general?

Erm, "selecting it in general" (well at least on x86) is what this patch is doing.

Sorry that I was unclear, what I meant to say is why wasn't this done ages ago since so many important architectures seem to have it enabled by default.

I suppose the reason would be something like "firmware/BIOS should handle that for us" and recently that has started to break apart and x86 platforms started to be more like ARM?

Yours,
Linus Walleij

#11 lore raw

Hi,

Quoted 9 lines · Linus Walleij

On Thu, Mar 12, 2020 at 2:49 PM Hans de Goede <[email protected]> wrote:

[Me]

Quoted 5 lines
Quoted 2 lines

I see that ARM and ARM64 simply just select this. What happens if you do that and why is x86 not selecting it in general?

Erm, "selecting it in general" (well at least on x86) is what this patch is doing.

Quoted 7 lines

Sorry that I was unclear, what I meant to say is why wasn't this done ages ago since so many important architectures seem to have it enabled by default.

I suppose the reason would be something like "firmware/BIOS should handle that for us" and recently that has started to break apart and x86 platforms started to be more like ARM?

That (x86 becoming more like ARM, sorta, kinda) as well as that turning it on on x86 was not safe until Thomas wrote the 2 patches which are marked as dependencies in the commit message for this patch.

Regards,

Hans

#12 lore raw
In reply to Linus Walleij
Quoted 14 lines · Linus Walleij

On Wed, Mar 11, 2020 at 10:42 PM tip-bot2 for Hans de Goede Just help me understand the semantics of this thing...

According to the text in KConfig:

# Tasklet based software resend for pending interrupts on enable_irq() config HARDIRQS_SW_RESEND
bool

According to
commit a4633adcdbc15ac51afcd0e1395de58cee27cf92

[PATCH] genirq: add genirq sw IRQ-retrigger

Quoted 8 lines

Enable platforms that do not have a hardware-assisted
hardirq-resend mechanism
to resend them via a softirq-driven IRQ emulation mechanism.

so when enable_irq() is called, if the IRQ is already asserted, it will be distributed in the form of a software irq?

OK I give up I don't understand the semantics of this thing.

Level type interrupts are "resending" in hardware as long as the device interrupt is still asserted.

The problem are edge interrupts.

When an edge interrupt is disabled via disable_irq() the core does not mask the chip because if the device raises an interrupt not all interrupt chips latch that and forward it to the CPU on unmask, i.e. some interrupt chips simply ignore an etch when the line is masked.

So when the device raises an edge while the interrupt is disabled the core still handles the hardware interrupt and:

- masks the interrupt line
- sets the pending bit
- does not invoke the device handler

On enable_irq() the pending bit is checked and if set the interrupt is tried to be retriggered or resent, but only if it's edge type.

So if the interrupt chip provides a irq_retrigger() callback the core uses that and only if this fails or is not available it resorts to software "resend" which means queueing it for execution in tasklet context.

Quoted 2 lines

I see that ARM and ARM64 simply just select this. What happens if you do that and why is x86 not selecting it in general?

irq resending on X86 is not really problem free for interrupts which are directly connect to the local APIC. The only way which is halfways safe is the hardware retrigger. See

    https://lkml.kernel.org/r/[email protected]
    https://lkml.kernel.org/r/[email protected]

for the gory details. The GPIO interrupts which hang off behind some slow bus or are multiplexed in other ways are not affected by this hardware design induced madness.

As I don't know how many other architectures have trainwrecked interrupt delivery mechanisms (IA64 definitely does), I'm more than reluctant to inflict this on the world unconditionally.

Hope that helps.

Thanks,

tglx