Skip to contentSkip to conversation

drm/i915: fix race when clearing RPS IIR bits

PATCH In review Imre Deak 4 participants 8 messages Mar 23, 2015 – Mar 24, 2015

Subsystems (2)
lore
Share & tools
lore

Imre Deak

author
#0 lore raw

PATCH 1/2 drm/i915: fix race when clearing RPS IIR bits

When disabling RPS interrupts there is a race where we disable RPS inerrupts while the interrupt handler is running and the handler has already latched the pending RPS interrupt from the master IIR register. Afterwards the disabling path clears the PM IIR bits, making the state of pending interrupts inconsistent from the interrupt handler's point of view. This triggers the following warning: "The master control interrupt lied (PM)!".

To fix this make sure that any running interrupt handler (which may have already latched the master IIR) finishes before clearing the IIR bits.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=87347

---
 drivers/gpu/drm/i915/i915_irq.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/gpu/drm/i915/i915_irq.c +7 -0
index 6d8340d..7afbde4 100644
@@ -330,6 +330,13 @@ void gen6_disable_rps_interrupts(struct drm_device *dev)
 	__gen6_disable_pm_irq(dev_priv, dev_priv->pm_rps_events);
 	I915_WRITE(gen6_pm_ier(dev_priv), I915_READ(gen6_pm_ier(dev_priv)) &
 				~dev_priv->pm_rps_events);
+
+	spin_unlock_irq(&dev_priv->irq_lock);
+
+	synchronize_irq(dev->irq);
+
+	spin_lock_irq(&dev_priv->irq_lock);
+
 	I915_WRITE(gen6_pm_iir(dev_priv), dev_priv->pm_rps_events);
 	I915_WRITE(gen6_pm_iir(dev_priv), dev_priv->pm_rps_events);
 
Signature · 6 lines
2.1.0

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#1 lore raw

PATCH 2/2 drm/i915: move clearing of RPS interrupt bits from disable to reset time

The logical place for clearing the RPS latched interrupt bits is when resetting the RPS interrupts, so move the corresponding part from the RPS disable function to the reset function. During resetting we already cleared the IIR bits, so the only thing missing there was clearing pm_iir.

Note that we call gen6_disable_rps_interrupts() also during driver load and resume time via intel_uncore_sanitize() when i915 interrupts are still not installed. If there are any pending RPS bits at this point (which after this patch wouldn't be cleared) they will be cleared by the reset code via the interrupt preinstall hooks.

---
 drivers/gpu/drm/i915/i915_irq.c | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)
drivers/gpu/drm/i915/i915_irq.c +1 -9
index 7afbde4..14ecb4d 100644
@@ -277,6 +277,7 @@ void gen6_reset_rps_interrupts(struct drm_device *dev)
 	I915_WRITE(reg, dev_priv->pm_rps_events);
 	I915_WRITE(reg, dev_priv->pm_rps_events);
 	POSTING_READ(reg);
+	dev_priv->rps.pm_iir = 0;
 	spin_unlock_irq(&dev_priv->irq_lock);
 }
 
@@ -334,15 +335,6 @@ void gen6_disable_rps_interrupts(struct drm_device *dev)
 	spin_unlock_irq(&dev_priv->irq_lock);
 
 	synchronize_irq(dev->irq);
-
-	spin_lock_irq(&dev_priv->irq_lock);
-
-	I915_WRITE(gen6_pm_iir(dev_priv), dev_priv->pm_rps_events);
-	I915_WRITE(gen6_pm_iir(dev_priv), dev_priv->pm_rps_events);
-
-	dev_priv->rps.pm_iir = 0;
-
-	spin_unlock_irq(&dev_priv->irq_lock);
 }
 
 /**
Signature · 6 lines
2.1.0

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

Re: [PATCH 2/2] drm/i915: move clearing of RPS interrupt bits from disable to reset time

Quoted 5 lines · Imre Deak

The logical place for clearing the RPS latched interrupt bits is when resetting the RPS interrupts, so move the corresponding part from the RPS disable function to the reset function. During resetting we already cleared the IIR bits, so the only thing missing there was clearing pm_iir.

Quoted 7 lines

Note that we call gen6_disable_rps_interrupts() also during driver load and resume time via intel_uncore_sanitize() when i915 interrupts are still not installed. If there are any pending RPS bits at this point (which after this patch wouldn't be cleared) they will be cleared by the reset code via the interrupt preinstall hooks.

Both merged to dinq, thanks.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Tested-by on v1 2/2
#3 lore raw

Re: [PATCH 2/2] drm/i915: move clearing of RPS interrupt bits from disable to reset time

In reply to Imre Deak

Task id: 6030 -------------------------------------Summary-------------------------------------
Platform Delta drm-intel-nightly Series Applied
PNV -1 275/275 274/275

ILK                                  303/303              303/303
SNB                                  304/304              304/304
IVB                                  339/339              339/339
BYT                                  287/287              287/287
HSW                                  361/361              361/361
BDW                                  310/310              310/310

-------------------------------------Detailed-------------------------------------
Platform Test drm-intel-nightly Series Applied
*PNV igt@gem_userptr_blits@minor-sync-interruptible PASS(2) DMESG_WARN(1)PASS(1)
Note: You need to pay more attention to line start with '*'
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw
In reply to Imre Deak
Quoted 8 lines · Imre Deak

When disabling RPS interrupts there is a race where we disable RPS inerrupts while the interrupt handler is running and the handler has already latched the pending RPS interrupt from the master IIR register. Afterwards the disabling path clears the PM IIR bits, making the state of pending interrupts inconsistent from the interrupt handler's point of view. This triggers the following warning: "The master control interrupt lied (PM)!".

Quoted 4 lines

To fix this make sure that any running interrupt handler (which may have already latched the master IIR) finishes before clearing the IIR bits.

Isn't this overkill for what is just a bogus WARN? If the WARN is a logical consequence of the code, let's just remove the WARN.

Or iow can you not find a cheaper way to fix this? -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#5 lore raw
Quoted 14 lines · Chris Wilson
Quoted 8 lines · Imre Deak

When disabling RPS interrupts there is a race where we disable RPS inerrupts while the interrupt handler is running and the handler has already latched the pending RPS interrupt from the master IIR register. Afterwards the disabling path clears the PM IIR bits, making the state of pending interrupts inconsistent from the interrupt handler's point of view. This triggers the following warning: "The master control interrupt lied (PM)!".

Quoted 4 lines

To fix this make sure that any running interrupt handler (which may have already latched the master IIR) finishes before clearing the IIR bits.

Quoted 4 lines

Isn't this overkill for what is just a bogus WARN? If the WARN is a logical consequence of the code, let's just remove the WARN.

Or iow can you not find a cheaper way to fix this?

We only run this on suspend/resume afaik, overhead should be acceptable. And we've had that much overhead before we've done all the runtime pm unification, it's still less synchronization than disabling interrupts completely.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw
Quoted 18 lines · Daniel Vetter
Quoted 17 lines · Chris Wilson
Quoted 8 lines · Imre Deak

When disabling RPS interrupts there is a race where we disable RPS inerrupts while the interrupt handler is running and the handler has already latched the pending RPS interrupt from the master IIR register. Afterwards the disabling path clears the PM IIR bits, making the state of pending interrupts inconsistent from the interrupt handler's point of view. This triggers the following warning: "The master control interrupt lied (PM)!".

Quoted 4 lines

To fix this make sure that any running interrupt handler (which may have already latched the master IIR) finishes before clearing the IIR bits.

Isn't this overkill for what is just a bogus WARN? If the WARN is a logical consequence of the code, let's just remove the WARN.

Quoted 6 lines
Quoted 1 line

Or iow can you not find a cheaper way to fix this?

We only run this on suspend/resume afaik, overhead should be acceptable. And we've had that much overhead before we've done all the runtime pm unification, it's still less synchronization than disabling interrupts completely.

Hmm, I thought this was in conjunction with RPS pm masking (i.e. fired everytime we no longer expect to receive RPS interrupts). If it is only the infrequent, then yeah I can't complain too much - I still think it is slightly fishy, but I can accept that it is just a quirk of the buffering the interrupt does.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#7 lore raw
Quoted 21 lines · Chris Wilson
Quoted 20 lines · Daniel Vetter
Quoted 18 lines · Chris Wilson
Quoted 8 lines · Imre Deak

When disabling RPS interrupts there is a race where we disable RPS inerrupts while the interrupt handler is running and the handler has already latched the pending RPS interrupt from the master IIR register. Afterwards the disabling path clears the PM IIR bits, making the state of pending interrupts inconsistent from the interrupt handler's point of view. This triggers the following warning: "The master control interrupt lied (PM)!".

Quoted 4 lines

To fix this make sure that any running interrupt handler (which may have already latched the master IIR) finishes before clearing the IIR bits.

Isn't this overkill for what is just a bogus WARN? If the WARN is a logical consequence of the code, let's just remove the WARN.

Or iow can you not find a cheaper way to fix this?

Quoted 7 lines
Quoted 4 lines

We only run this on suspend/resume afaik, overhead should be acceptable. And we've had that much overhead before we've done all the runtime pm unification, it's still less synchronization than disabling interrupts completely.

Hmm, I thought this was in conjunction with RPS pm masking (i.e. fired everytime we no longer expect to receive RPS interrupts).

Yes, it affects only the driver loading (where it's a nop) and the runtime/system suspend path.

Quoted 3 lines

If it is only the infrequent, then yeah I can't complain too much - I still think it is slightly fishy, but I can accept that it is just a quirk of the buffering the interrupt does.

The reason for sync_irq is to prevent any subsequent gen6_enable_rps_interrupts() call to trigger spurious interrupts. Without it we could have:

CPU 0 CPU 1
gen8_gt_irq_handler()
tmp = I915_READ(GEN8_GT_IIR(2))
<tmp has pending RPS bits>

                                    gen6_disable_rps_interrupts()
                                    gen6_reset_rps_interrupts()
                                    <clear GEN8_GT_IIR(2)>
                                    gen6_enable_rps_interrupts()

gen6_rps_irq_handler(tmp)
<handle the now stale RPS events>

Now I admit the above is quite unlikely, or even impossible due to what happens between gen6_disable_rps_interrupts() and gen6_enable_rps_interrupts() in our code atm. I still thought it's safer not to rely on those.

One alternative would have been to extend the irq_lock in the interrupt handler, so that we read GEN8_GT_IIR(2) and handle any events in it in one critical section, but that would add the overhead where it actually matters.

--Imre

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx