Skip to content

RFC drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

Denis Pisarev amd-gfx dri-devel lkml 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 RFC Latest1 patch 4 repliesIn review
  2. v2 RFC1 patch 0 repliesSuperseded
  3. v1 RFC1 patch 2 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/1drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is unresponsivedrivers/gpu/drm/amd/amdgpu/amdgpu.h drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +2 more+114 -330 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [RFC PATCH v3 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

Hi,

v3 addresses the two findings from the sashiko-bot review of v2. Failure data and trigger isolation are unchanged (bugzilla 219492):
S4 resume on Cezanne (gmc_v9, GFXOFF) wedges KIQ TLB flushes at 80-140/hour for 9+ hours with sched.ready true throughout; holding GFXOFF off across the S4 cycle produces zero errors.

1. [High] "VFs and interrupt contexts silently drop TLB flushes once the threshold is reached" - correct, and fixed. In v3 the latch only reroutes bare metal process context to MMIO. VFs and IRQ contexts keep submitting to KIQ exactly as before this patch, with per-failure logging, because they have no MMIO alternative; there is no longer any code path that drops a flush without attempting and logging.

2. [High] "KIQ and MMIO race on the same invalidation engine if KIQ recovers" - this remains the documented open question; no code change in v3. Our analysis: once latched, this path submits no new KIQ commands, so the exposure is limited to already-queued stale commands and the recovery transition window. The engine serializes requests internally, so the realistic worst case is a lost flush request caught by the existing ACK timeout ("Timeout waiting for VM flush ACK!"), not silent state corruption. If maintainers consider a fence necessary (or a dedicated invalidate engine for the MMIO path), guidance on the preferred mechanism would be welcome.

Full patch history: v1 (initial fallback+counter), v2 (GFXOFF hold, per-instance counter, irqsave, VF/IRQ restrictions, MES error propagation) - all from bot review; v3 (this one) fixes the VF/IRQ drop regression the bot found in v2.

Also still open from the cover letters: the alternative direction of fixing the S4 resume ordering itself (RLC/ME vs GFXOFF) instead of a runtime fallback. Happy to run tracing on the affected hardware.

Patch 1/1 follows.

Denis Pisarev

Denis Pisarev (1):
drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is
unresponsive

 drivers/gpu/drm/amd/amdgpu/amdgpu.h     |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c |  18 ++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |   2 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 123 +++++++++++++++++++-----
 5 files changed, 114 insertions(+), 33 deletions(-)
Signature · 1 line
2.55.0