Skip to contentSkip to conversation

Enable OA unit for Gen 8 and 9 in i915 perf

PATCH In review Robert Bragg 5 participants 87 messages Apr 12, 2017 – Jun 1, 2017 Page 1 of 2

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover Enable OA unit for Gen 8 and 9 in i915 perf

Updates based on latest review feedback from Matthew and Lionel and includes an update to the TestOa register config for SKL GT2 compared the last series (based on the latest XML files from VPG)

Although the _[SUB]SLICE_MASK GETPARM patches were reviewed, it's worth mentioning there was a TODO comment in the last patches about rebasing the parameter ID before upstreaming which is removed now, and there's a minimal comment for what the new parameters are, consistent with other more recently added parameters. Conveniently the actual IDs didn't need rebasing so there's no last moment uapi change for gputop, mesa and igt.

The series is longer just because I've included the gen7 prep patches (already reviewed) that I haven't landed yet but the gen8+ bits depend on.

Regards,
- Robert

Robert Bragg (15):
drm/i915/perf: fix gen7_append_oa_reports comment drm/i915/perf: avoid poll, read, EAGAIN busy loops drm/i915/perf: avoid read back of head register drm/i915/perf: no head/tail ref in gen7_oa_read drm/i915/perf: improve tail race workaround drm/i915/perf: improve invalid OA format debug message drm/i915/perf: better pipeline aged/aging tail updates drm/i915/perf: rate limit spurious oa report notice drm/i915: expose _SLICE_MASK GETPARM
drm/i915: expose _SUBSLICE_MASK GETPARM drm/i915/perf: Add 'render basic' Gen8+ OA unit configs drm/i915/perf: Add OA unit support for Gen 8+ drm/i915/perf: Add more OA configs for BDW, CHV, SKL + BXT drm/i915/perf: per-gen timebase for checking sample freq drm/i915/perf: remove perf.hook_lock

 drivers/gpu/drm/i915/Makefile           |    8 +-
 drivers/gpu/drm/i915/i915_drv.c         |   10 +
 drivers/gpu/drm/i915/i915_drv.h         |  121 +-
 drivers/gpu/drm/i915/i915_gem_context.h |    1 +
 drivers/gpu/drm/i915/i915_oa_bdw.c      | 5154 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bdw.h      |   38 +
 drivers/gpu/drm/i915/i915_oa_bxt.c      | 2541 +++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bxt.h      |   38 +
 drivers/gpu/drm/i915/i915_oa_chv.c      | 2730 ++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_chv.h      |   38 +
 drivers/gpu/drm/i915/i915_oa_hsw.c      |   58 +-
 drivers/gpu/drm/i915/i915_oa_sklgt2.c   | 3302 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt2.h   |   38 +
 drivers/gpu/drm/i915/i915_oa_sklgt3.c   | 2856 +++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt3.h   |   38 +
 drivers/gpu/drm/i915/i915_oa_sklgt4.c   | 2910 +++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt4.h   |   38 +
 drivers/gpu/drm/i915/i915_perf.c        | 1341 ++++++--
 drivers/gpu/drm/i915/i915_reg.h         |   22 +
 drivers/gpu/drm/i915/intel_lrc.c        |    5 +
 include/uapi/drm/i915_drm.h             |   27 +-
 21 files changed, 21076 insertions(+), 238 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.h
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

PATCH v4 1/15 drm/i915/perf: fix gen7_append_oa_reports comment

If I'm going to complain about a back-to-front convention then the least I can do is not muddle the comment up too.

---
 drivers/gpu/drm/i915/i915_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_perf.c +1 -1
index 060b171480d5..78fef53b45c9 100644
@@ -431,7 +431,7 @@ static int append_oa_sample(struct i915_perf_stream *stream,
  * userspace.
  *
  * Note: reports are consumed from the head, and appended to the
- * tail, so the head chases the tail?... If you think that's mad
+ * tail, so the tail chases the head?... If you think that's mad
  * and back-to-front you're not alone, but this follows the
  * Gen PRM naming convention.
  *
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

PATCH v4 2/15 drm/i915/perf: avoid poll, read, EAGAIN busy loops

In reply to Robert Bragg

If the function for checking whether there is OA buffer data available (during a poll or blocking read) has false positives then we want to avoid a situation where the subsequent read() returns EAGAIN (after a more accurate check) followed by a poll() immediately reporting the same false positive POLLIN event and effectively maintaining a busy loop until there really is data.

This makes sure that we clear the .pollin event status whenever we return EAGAIN to userspace which will throttle subsequent POLLIN events and repeated attempts to read to the 5ms intervals of the hrtimer callback we have.

---
 drivers/gpu/drm/i915/i915_perf.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_perf.c +9 -1
index 78fef53b45c9..f59f6dd20922 100644
@@ -1352,7 +1352,15 @@ static ssize_t i915_perf_read(struct file *file,
 		mutex_unlock(&dev_priv->perf.lock);
 	}
 
-	if (ret >= 0) {
+	/* We allow the poll checking to sometimes report false positive POLLIN
+	 * events where we might actually report EAGAIN on read() if there's
+	 * not really any data available. In this situation though we don't
+	 * want to enter a busy loop between poll() reporting a POLLIN event
+	 * and read() returning -EAGAIN. Clearing the oa.pollin state here
+	 * effectively ensures we back off until the next hrtimer callback
+	 * before reporting another POLLIN event.
+	 */
+	if (ret >= 0 || ret == -EAGAIN) {
 		/* Maybe make ->pollin per-stream state if we support multiple
 		 * concurrent streams in the future.
 		 */
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw

PATCH v4 3/15 drm/i915/perf: avoid read back of head register

In reply to Robert Bragg

There's no need for the driver to keep reading back the head pointer from hardware since the hardware doesn't update it automatically. This way we can treat any invalid head pointer value as a software/driver bug instead of spurious hardware behaviour.

This change is also a small stepping stone towards re-working how the head and tail state is managed as part of an improved workaround for the tail register race condition.

---
 drivers/gpu/drm/i915/i915_drv.h  | 11 ++++++++++
 drivers/gpu/drm/i915/i915_perf.c | 46 ++++++++++++++++++----------------------
 2 files changed, 32 insertions(+), 25 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 1af4e6f5410c..2f8a7a4f29df 100644
@@ -2466,6 +2466,17 @@ struct drm_i915_private {
 				u8 *vaddr;
 				int format;
 				int format_size;
+
+				/**
+				 * Although we can always read back the head
+				 * pointer register, we prefer to avoid
+				 * trusting the HW state, just to avoid any
+				 * risk that some hardware condition could
+				 * somehow bump the head pointer unpredictably
+				 * and cause us to forward the wrong OA buffer
+				 * data to userspace.
+				 */
+				u32 head;
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
drivers/gpu/drm/i915/i915_perf.c +21 -25

104 lines not on this page · raw message

Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw

PATCH v4 4/15 drm/i915/perf: no head/tail ref in gen7_oa_read

In reply to Robert Bragg

This avoids redundantly passing an (inout) head and tail pointer to gen7_append_oa_reports() from gen7_oa_read which doesn't need to reference either itself.

Moving the head/tail reads and writes into gen7_append_oa_reports should have no functional effect except to avoid some redundant head pointer writes in cases where nothing was copied to userspace.

This is a stepping stone towards updating how the head and tail pointer state is managed to improve the workaround for the OA unit's tail pointer race. It reduces the number of places we need to read/write the head and tail pointers.

---
 drivers/gpu/drm/i915/i915_perf.c | 51 +++++++++++++++-------------------------
 1 file changed, 19 insertions(+), 32 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +19 -32
index f47d1cc2144b..83dc67a635fb 100644
@@ -420,8 +420,6 @@ static int append_oa_sample(struct i915_perf_stream *stream,
  * @buf: destination buffer given by userspace
  * @count: the number of bytes userspace wants to read
  * @offset: (inout): the current position for writing into @buf
- * @head_ptr: (inout): the current oa buffer cpu read position
- * @tail: the current oa buffer gpu write position
  *
  * Notably any error condition resulting in a short read (-%ENOSPC or
  * -%EFAULT) will be returned even though one or more records may
@@ -439,9 +437,7 @@ static int append_oa_sample(struct i915_perf_stream *stream,
 static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 				  char __user *buf,
 				  size_t count,
-				  size_t *offset,
-				  u32 *head_ptr,
-				  u32 tail)
+				  size_t *offset)
 {
 	struct drm_i915_private *dev_priv = stream->dev_priv;
 	int report_size = dev_priv->perf.oa.oa_buffer.format_size;
@@ -449,14 +445,15 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 	int tail_margin = dev_priv->perf.oa.tail_margin;
 	u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
 	u32 mask = (OA_BUFFER_SIZE - 1);
-	u32 head;
+	size_t start_offset = *offset;
+	u32 head, oastatus1, tail;
 	u32 taken;
 	int ret = 0;
 
 	if (WARN_ON(!stream->enabled))
 		return -EIO;
 
-	head = *head_ptr - gtt_offset;
+	head = dev_priv->perf.oa.oa_buffer.head - gtt_offset;
 
 	/* An out of bounds or misaligned head pointer implies a driver bug
 	 * since we are in full control of head pointer which should only
@@ -467,7 +464,8 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 		      "Inconsistent OA buffer head pointer = %u\n", head))
 		return -EIO;
 
-	tail -= gtt_offset;
+	oastatus1 = I915_READ(GEN7_OASTATUS1);
+	tail = (oastatus1 & GEN7_OASTATUS1_TAIL_MASK) - gtt_offset;
 
 	/* The OA unit is expected to wrap the tail pointer according to the OA
 	 * buffer size
@@ -477,8 +475,6 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 			  tail);
 		dev_priv->perf.oa.ops.oa_disable(dev_priv);
 		dev_priv->perf.oa.ops.oa_enable(dev_priv);
-		*head_ptr = I915_READ(GEN7_OASTATUS2) &
-			GEN7_OASTATUS2_HEAD_MASK;
 		return -EIO;
 	}
 
@@ -542,7 +538,18 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 		report32[0] = 0;
 	}
 
-	*head_ptr = gtt_offset + head;
+
+	if (start_offset != *offset) {
+		/* We removed the gtt_offset for the copy loop above, indexing
+		 * relative to oa_buf_base so put back here...
+		 */
+		head += gtt_offset;
+
+		I915_WRITE(GEN7_OASTATUS2,
+			   ((head & GEN7_OASTATUS2_HEAD_MASK) |
+			    OA_MEM_SELECT_GGTT));
+		dev_priv->perf.oa.oa_buffer.head = head;
+	}
 
 	return ret;
 }
@@ -570,8 +577,6 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 {
 	struct drm_i915_private *dev_priv = stream->dev_priv;
 	u32 oastatus1;
-	u32 head;
-	u32 tail;
 	int ret;
 
 	if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
@@ -579,9 +584,6 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 
 	oastatus1 = I915_READ(GEN7_OASTATUS1);
 
-	head = dev_priv->perf.oa.oa_buffer.head;
-	tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
-
 	/* XXX: On Haswell we don't have a safe way to clear oastatus1
 	 * bits while the OA unit is enabled (while the tail pointer
 	 * may be updated asynchronously) so we ignore status bits
@@ -621,9 +623,6 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 		dev_priv->perf.oa.ops.oa_enable(dev_priv);
 
 		oastatus1 = I915_READ(GEN7_OASTATUS1);
-
-		head = dev_priv->perf.oa.oa_buffer.head;
-		tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
 	}
 
 	if (unlikely(oastatus1 & GEN7_OASTATUS1_REPORT_LOST)) {
@@ -635,19 +634,7 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 			GEN7_OASTATUS1_REPORT_LOST;
 	}
 
-	ret = gen7_append_oa_reports(stream, buf, count, offset,
-				     &head, tail);
-
-	/* Note: we update the head pointer here even if an error
-	 * was returned since the error may represent a short read
-	 * where some some reports were successfully copied.
-	 */
-	I915_WRITE(GEN7_OASTATUS2,
-		   ((head & GEN7_OASTATUS2_HEAD_MASK) |
-		    OA_MEM_SELECT_GGTT));
-	dev_priv->perf.oa.oa_buffer.head = head;
-
-	return ret;
+	return gen7_append_oa_reports(stream, buf, count, offset);
 }
 
 /**
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#5 lore raw

PATCH v4 5/15 drm/i915/perf: improve tail race workaround

In reply to Robert Bragg

There's a HW race condition between OA unit tail pointer register updates and writes to memory whereby the tail pointer can sometimes get ahead of what's been written out to the OA buffer so far (in terms of what's visible to the CPU).

Although this can be observed explicitly while copying reports to userspace by checking for a zeroed report-id field in tail reports, we want to account for this earlier, as part of the _oa_buffer_check to avoid lots of redundant read() attempts.

Previously the driver used to define an effective tail pointer that lagged the real pointer by a 'tail margin' measured in bytes derived from OA_TAIL_MARGIN_NSEC and the configured sampling frequency. Unfortunately this was flawed considering that the OA unit may also automatically generate non-periodic reports (such as on context switch) or the OA unit may be enabled without any periodic sampling.

This improves how we define a tail pointer for reading that lags the real tail pointer by at least %OA_TAIL_MARGIN_NSEC nanoseconds, which gives enough time for the corresponding reports to become visible to the CPU.

The driver now maintains two tail pointers:
1) An 'aging' tail with an associated timestamp that is tracked until we can trust the corresponding data is visible to the CPU; at which point it is considered 'aged'.
2) An 'aged' tail that can be used for read()ing.

The two separate pointers let us decouple read()s from tail pointer aging.

The tail pointers are checked and updated at a limited rate within a hrtimer callback (the same callback that is used for delivering POLLIN events) and since we're now measuring the wall clock time elapsed since a given tail pointer was read the mechanism no longer cares about the OA unit's periodic sampling frequency.

The natural place to handle the tail pointer updates was in gen7_oa_buffer_is_empty() which is called as part of blocking reads and the hrtimer callback used for polling, and so this was renamed to oa_buffer_check() considering the added side effect while checking whether the buffer contains data.

---
 drivers/gpu/drm/i915/i915_drv.h  |  60 ++++++++-
 drivers/gpu/drm/i915/i915_perf.c | 277 ++++++++++++++++++++++++++-------------
 2 files changed, 241 insertions(+), 96 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +55 -5
index 2f8a7a4f29df..088c4c60bd38 100644
@@ -2094,7 +2094,7 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_is_empty: Check if OA buffer empty (false positives OK)
+	 * @oa_buffer_check: Check for OA buffer data + update tail
 	 *
 	 * This is either called via fops or the poll check hrtimer (atomic
 	 * ctx) without any locks taken.
@@ -2107,7 +2107,7 @@ struct i915_oa_ops {
 	 * here, which will be handled gracefully - likely resulting in an
 	 * %EAGAIN error for userspace.
 	 */
-	bool (*oa_buffer_is_empty)(struct drm_i915_private *dev_priv);
+	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2450,9 +2450,6 @@ struct drm_i915_private {
 
 			bool periodic;
 			int period_exponent;
-			int timestamp_frequency;
-
-			int tail_margin;
 
 			int metrics_set;
 
@@ -2468,6 +2465,59 @@ struct drm_i915_private {
 				int format_size;
 
 				/**
+				 * Locks reads and writes to all head/tail state
+				 *
+				 * Consider: the head and tail pointer state
+				 * needs to be read consistently from a hrtimer
+				 * callback (atomic context) and read() fop
+				 * (user context) with tail pointer updates
+				 * happening in atomic context and head updates
+				 * in user context and the (unlikely)
+				 * possibility of read() errors needing to
+				 * reset all head/tail state.
+				 *
+				 * Note: Contention or performance aren't
+				 * currently a significant concern here
+				 * considering the relatively low frequency of
+				 * hrtimer callbacks (5ms period) and that
+				 * reads typically only happen in response to a
+				 * hrtimer event and likely complete before the
+				 * next callback.
+				 *
+				 * Note: This lock is not held *while* reading
+				 * and copying data to userspace so the value
+				 * of head observed in htrimer callbacks won't
+				 * represent any partial consumption of data.
+				 */
+				spinlock_t ptr_lock;
+
+				/**
+				 * One 'aging' tail pointer and one 'aged'
+				 * tail pointer ready to used for reading.
+				 *
+				 * Initial values of 0xffffffff are invalid
+				 * and imply that an update is required
+				 * (and should be ignored by an attempted
+				 * read)
+				 */
+				struct {
+					u32 offset;
+				} tails[2];
+
+				/**
+				 * Index for the aged tail ready to read()
+				 * data up to.
+				 */
+				unsigned int aged_tail_idx;
+
+				/**
+				 * A monotonic timestamp for when the current
+				 * aging tail pointer was read; used to
+				 * determine when it is old enough to trust.
+				 */
+				u64 aging_timestamp;
+
+				/**
 				 * Although we can always read back the head
 				 * pointer register, we prefer to avoid
 				 * trusting the HW state, just to avoid any
drivers/gpu/drm/i915/i915_perf.c +186 -91

388 lines not on this page · raw message

Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw

PATCH v4 6/15 drm/i915/perf: improve invalid OA format debug message

In reply to Robert Bragg

A minor improvement to debugging output

---
 drivers/gpu/drm/i915/i915_perf.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +4 -2
index 18734e1926b9..08cc2b0dd734 100644
@@ -1904,11 +1904,13 @@ static int read_properties_unlocked(struct drm_i915_private *dev_priv,
 			break;
 		case DRM_I915_PERF_PROP_OA_FORMAT:
 			if (value == 0 || value >= I915_OA_FORMAT_MAX) {
-				DRM_DEBUG("Invalid OA report format\n");
+				DRM_DEBUG("Out-of-range OA report format %llu\n",
+					  value);
 				return -EINVAL;
 			}
 			if (!dev_priv->perf.oa.oa_formats[value].size) {
-				DRM_DEBUG("Invalid OA report format\n");
+				DRM_DEBUG("Unsupported OA report format %llu\n",
+					  value);
 				return -EINVAL;
 			}
 			props->oa_format = value;
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw

PATCH v4 7/15 drm/i915/perf: better pipeline aged/aging tail updates

In reply to Robert Bragg

This updates the tail pointer race workaround handling to updating the 'aged' pointer before looking to start aging a new one. There's the possibility that there is already new data available and so we can immediately start aging a new pointer without having to first wait for a later hrtimer callback (and then another to age).

---
 drivers/gpu/drm/i915/i915_perf.c | 41 ++++++++++++++++++++++------------------
 1 file changed, 23 insertions(+), 18 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +23 -18
index 08cc2b0dd734..5738b99caa5b 100644
@@ -391,6 +391,29 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 
 	now = ktime_get_mono_fast_ns();
 
+	/* Update the aged tail
+	 *
+	 * Flip the tail pointer available for read()s once the aging tail is
+	 * old enough to trust that the corresponding data will be visible to
+	 * the CPU...
+	 *
+	 * Do this before updating the aging pointer in case we may be able to
+	 * immediately start aging a new pointer too (if new data has become
+	 * available) without needing to wait for a later hrtimer callback.
+	 */
+	if (aging_tail != INVALID_TAIL_PTR &&
+	    ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
+	     OA_TAIL_MARGIN_NSEC)) {
+		aged_idx ^= 1;
+		dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
+
+		aged_tail = aging_tail;
+
+		/* Mark that we need a new pointer to start aging... */
+		dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset = INVALID_TAIL_PTR;
+		aging_tail = INVALID_TAIL_PTR;
+	}
+
 	/* Update the aging tail
 	 *
 	 * We throttle aging tail updates until we have a new tail that
@@ -420,24 +443,6 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 		}
 	}
 
-	/* Update the aged tail
-	 *
-	 * Flip the tail pointer available for read()s once the aging tail is
-	 * old enough to trust that the corresponding data will be visible to
-	 * the CPU...
-	 */
-	if (aging_tail != INVALID_TAIL_PTR &&
-	    ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
-	     OA_TAIL_MARGIN_NSEC)) {
-		aged_idx ^= 1;
-		dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
-
-		aged_tail = aging_tail;
-
-		/* Mark that we need a new pointer to start aging... */
-		dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset = INVALID_TAIL_PTR;
-	}
-
 	spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
 
 	return aged_tail == INVALID_TAIL_PTR ?
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#8 lore raw

PATCH v4 8/15 drm/i915/perf: rate limit spurious oa report notice

In reply to Robert Bragg

This change is pre-emptively aiming to avoid a potential cause of kernel logging noise in case some condition were to result in us seeing invalid OA reports.

The workaround for the OA unit's tail pointer race condition is what avoids the primary known cause of invalid reports being seen and with that in place we aren't expecting to see this notice but it can't be entirely ruled out.

Just in case some condition does lead to the notice then it's likely that it will be triggered repeatedly while attempting to append a sequence of reports and depending on the configured OA sampling frequency that might be a large number of repeat notices.

v2: (Chris) avoid inconsistent warning on throttle with
printk_ratelimit()
v3: (Matt) init and summarise with stream init/close not driver init/fini

---
 drivers/gpu/drm/i915/i915_drv.h  |  6 ++++++
 drivers/gpu/drm/i915/i915_perf.c | 28 +++++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_drv.h +6 -0
index 088c4c60bd38..a0e34934a11f 100644
@@ -2448,6 +2448,12 @@ struct drm_i915_private {
 			wait_queue_head_t poll_wq;
 			bool pollin;
 
+			/**
+			 * For rate limiting any notifications of spurious
+			 * invalid OA reports
+			 */
+			struct ratelimit_state spurious_report_rs;
+
 			bool periodic;
 			int period_exponent;
 
drivers/gpu/drm/i915/i915_perf.c +27 -1
index 5738b99caa5b..3277a52ce98e 100644
@@ -632,7 +632,8 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 		 * copying it to userspace...
 		 */
 		if (report32[0] == 0) {
-			DRM_NOTE("Skipping spurious, invalid OA report\n");
+			if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+				DRM_NOTE("Skipping spurious, invalid OA report\n");
 			continue;
 		}
 
@@ -913,6 +914,11 @@ static void i915_oa_stream_destroy(struct i915_perf_stream *stream)
 		oa_put_render_ctx_id(stream);
 
 	dev_priv->perf.oa.exclusive_stream = NULL;
+
+	if (dev_priv->perf.oa.spurious_report_rs.missed) {
+		DRM_NOTE("%d spurious OA report notices suppressed due to ratelimiting\n",
+			 dev_priv->perf.oa.spurious_report_rs.missed);
+	}
 }
 
 static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
@@ -1268,6 +1274,26 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream,
 		return -EINVAL;
 	}
 
+	/* We set up some ratelimit state to potentially throttle any _NOTES
+	 * about spurious, invalid OA reports which we don't forward to
+	 * userspace.
+	 *
+	 * The initialization is associated with opening the stream (not driver
+	 * init) considering we print a _NOTE about any throttling when closing
+	 * the stream instead of waiting until driver _fini which no one would
+	 * ever see.
+	 *
+	 * Using the same limiting factors as printk_ratelimit()
+	 */
+	ratelimit_state_init(&dev_priv->perf.oa.spurious_report_rs,
+			     5 * HZ, 10);
+	/* Since we use a DRM_NOTE for spurious reports it would be
+	 * inconsistent to let __ratelimit() automatically print a warning for
+	 * throttling.
+	 */
+	ratelimit_set_flags(&dev_priv->perf.oa.spurious_report_rs,
+			    RATELIMIT_MSG_ON_RELEASE);
+
 	stream->sample_size = sizeof(struct drm_i915_perf_record_header);
 
 	format_size = dev_priv->perf.oa.oa_formats[props->oa_format].size;
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#9 lore raw

PATCH v4 9/15 drm/i915: expose _SLICE_MASK GETPARM

In reply to Robert Bragg

Enables userspace to determine the number of slices enabled and also know what specific slices are enabled. This information is required, for example, to be able to analyse some OA counter reports where the counter configuration depends on the HW slice configuration.

---
 drivers/gpu/drm/i915/i915_drv.c | 5 +++++
 include/uapi/drm/i915_drm.h     | 3 +++
 2 files changed, 8 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +5 -0
index bd85e3826b72..76a724b7cc22 100644
@@ -357,6 +357,11 @@ static int i915_getparam(struct drm_device *dev, void *data,
 		 */
 		value = 1;
 		break;
+	case I915_PARAM_SLICE_MASK:
+		value = INTEL_INFO(dev_priv)->sseu.slice_mask;
+		if (!value)
+			return -ENODEV;
+		break;
 	default:
 		DRM_DEBUG("Unknown parameter %d\n", param->param);
 		return -EINVAL;
include/uapi/drm/i915_drm.h +3 -0
index 9ee06ec8a2d6..99bfc3648454 100644
@@ -412,6 +412,9 @@ typedef struct drm_i915_irq_wait {
  */
 #define I915_PARAM_HAS_EXEC_FENCE	 44
 
+/* Query the mask of slices available for this system */
+#define I915_PARAM_SLICE_MASK		 45
+
 typedef struct drm_i915_getparam {
 	__s32 param;
 	/*
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#10 lore raw

PATCH v4 10/15 drm/i915: expose _SUBSLICE_MASK GETPARM

In reply to Robert Bragg

Assuming a uniform mask across all slices, this enables userspace to determine the specific sub slices enabled. This information is required, for example, to be able to analyse some OA counter reports where the counter configuration depends on the HW sub slice configuration.

---
 drivers/gpu/drm/i915/i915_drv.c | 5 +++++
 include/uapi/drm/i915_drm.h     | 5 +++++
 2 files changed, 10 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +5 -0
index 76a724b7cc22..133ab46bf2f2 100644
@@ -362,6 +362,11 @@ static int i915_getparam(struct drm_device *dev, void *data,
 		if (!value)
 			return -ENODEV;
 		break;
+	case I915_PARAM_SUBSLICE_MASK:
+		value = INTEL_INFO(dev_priv)->sseu.subslice_mask;
+		if (!value)
+			return -ENODEV;
+		break;
 	default:
 		DRM_DEBUG("Unknown parameter %d\n", param->param);
 		return -EINVAL;
include/uapi/drm/i915_drm.h +5 -0
index 99bfc3648454..689fd7a418a7 100644
@@ -415,6 +415,11 @@ typedef struct drm_i915_irq_wait {
 /* Query the mask of slices available for this system */
 #define I915_PARAM_SLICE_MASK		 45
 
+/* Assuming it's uniform for each slice, this queries the mask of subslices
+ * per-slice for this system.
+ */
+#define I915_PARAM_SUBSLICE_MASK	 46
+
 typedef struct drm_i915_getparam {
 	__s32 param;
 	/*
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#11 lore raw

PATCH v4 11/15 drm/i915/perf: Add 'render basic' Gen8+ OA unit configs

In reply to Robert Bragg

Adds a static OA unit, MUX, B Counter + Flex EU configurations for basic render metrics on Broadwell, Cherryview, Skylake and Broxton. These are auto generated from an XML description of metric sets, currently maintained in gputop, ref:

https://github.com/rib/gputop

Quoted 2 lines

gputop-data/oa-*.xml
scripts/i915-perf-kernelgen.py

 $ make -C gputop-data -f Makefile.xml WHITELIST=RenderBasic

v2: add newlines to debug messages + fix comment (Matthew Auld)

---
 drivers/gpu/drm/i915/Makefile         |   8 +-
 drivers/gpu/drm/i915/i915_drv.h       |   2 +
 drivers/gpu/drm/i915/i915_oa_bdw.c    | 380 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bdw.h    |  38 ++++
 drivers/gpu/drm/i915/i915_oa_bxt.c    | 238 +++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bxt.h    |  38 ++++
 drivers/gpu/drm/i915/i915_oa_chv.c    | 225 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_chv.h    |  38 ++++
 drivers/gpu/drm/i915/i915_oa_sklgt2.c | 228 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt2.h |  38 ++++
 drivers/gpu/drm/i915/i915_oa_sklgt3.c | 236 +++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt3.h |  38 ++++
 drivers/gpu/drm/i915/i915_oa_sklgt4.c | 247 ++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt4.h |  38 ++++
 14 files changed, 1791 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.h
drivers/gpu/drm/i915/Makefile +7 -1
index 2cf04504e494..41400a138a1e 100644
@@ -127,7 +127,13 @@ i915-y += i915_vgpu.o
 
 # perf code
 i915-y += i915_perf.o \
-	  i915_oa_hsw.o
+	  i915_oa_hsw.o \
+	  i915_oa_bdw.o \
+	  i915_oa_chv.o \
+	  i915_oa_sklgt2.o \
+	  i915_oa_sklgt3.o \
+	  i915_oa_sklgt4.o \
+	  i915_oa_bxt.o
 
 ifeq ($(CONFIG_DRM_I915_GVT),y)
 i915-y += intel_gvt.o
drivers/gpu/drm/i915/i915_drv.h +2 -0
index a0e34934a11f..13b9125cacdd 100644
@@ -2463,6 +2463,8 @@ struct drm_i915_private {
 			int mux_regs_len;
 			const struct i915_oa_reg *b_counter_regs;
 			int b_counter_regs_len;
+			const struct i915_oa_reg *flex_regs;
+			int flex_regs_len;
 
 			struct {
 				struct i915_vma *vma;
drivers/gpu/drm/i915/i915_oa_bdw.c new file +380 -0

380 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_bdw.h new file +38 -0
new file mode 100644
index 000000000000..ea6969aaf91a
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_BDW_H__
+#define __I915_OA_BDW_H__
+
+extern int i915_oa_n_builtin_metric_sets_bdw;
+
+extern int i915_oa_select_metric_set_bdw(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_bdw(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_bdw(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_bxt.c new file +238 -0

238 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_bxt.h new file +38 -0
new file mode 100644
index 000000000000..9e86662c891d
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_BXT_H__
+#define __I915_OA_BXT_H__
+
+extern int i915_oa_n_builtin_metric_sets_bxt;
+
+extern int i915_oa_select_metric_set_bxt(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_bxt(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_bxt(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_chv.c new file +225 -0

225 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_chv.h new file +38 -0
new file mode 100644
index 000000000000..e494200ff04f
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_CHV_H__
+#define __I915_OA_CHV_H__
+
+extern int i915_oa_n_builtin_metric_sets_chv;
+
+extern int i915_oa_select_metric_set_chv(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_chv(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_chv(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_sklgt2.c new file +228 -0

228 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt2.h new file +38 -0
new file mode 100644
index 000000000000..e99b03f1bf9e
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_SKLGT2_H__
+#define __I915_OA_SKLGT2_H__
+
+extern int i915_oa_n_builtin_metric_sets_sklgt2;
+
+extern int i915_oa_select_metric_set_sklgt2(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_sklgt2(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_sklgt2(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_sklgt3.c new file +236 -0

236 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt3.h new file +38 -0
new file mode 100644
index 000000000000..ff7ec3206bf2
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_SKLGT3_H__
+#define __I915_OA_SKLGT3_H__
+
+extern int i915_oa_n_builtin_metric_sets_sklgt3;
+
+extern int i915_oa_select_metric_set_sklgt3(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_sklgt3(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_sklgt3(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_sklgt4.c new file +247 -0

247 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt4.h new file +38 -0
new file mode 100644
index 000000000000..4faa8c4579cd
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_SKLGT4_H__
+#define __I915_OA_SKLGT4_H__
+
+extern int i915_oa_n_builtin_metric_sets_sklgt4;
+
+extern int i915_oa_select_metric_set_sklgt4(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_sklgt4(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_sklgt4(struct drm_i915_private *dev_priv);
+
+#endif
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#12 lore raw

PATCH v4 12/15 drm/i915/perf: Add OA unit support for Gen 8+

In reply to Robert Bragg

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

---
 drivers/gpu/drm/i915/i915_drv.h         |  45 +-
 drivers/gpu/drm/i915/i915_gem_context.h |   1 +
 drivers/gpu/drm/i915/i915_perf.c        | 949 +++++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_reg.h         |  22 +
 drivers/gpu/drm/i915/intel_lrc.c        |   5 +
 include/uapi/drm/i915_drm.h             |  19 +-
 6 files changed, 948 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index 13b9125cacdd..b8dcf281db53 100644
@@ -2061,9 +2061,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2094,20 +2102,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2469,6 +2470,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2538,6 +2540,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2848,6 +2858,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3612,6 +3624,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_update_reg_state(struct intel_engine_cs *engine,
+			      struct i915_gem_context *ctx,
+			      uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_gem_context.h +1 -0
index 4af2ab94558b..3f4ce73bea43 100644
@@ -151,6 +151,7 @@ struct i915_gem_context {
 		u64 lrc_desc;
 		int pin_count;
 		bool initialised;
+		atomic_t oa_state_dirty;
 	} engine[I915_NUM_ENGINES];
 
 	/** ring_size: size for allocating the per-engine ring buffer */
drivers/gpu/drm/i915/i915_perf.c +878 -71

1,138 lines not on this page · raw message

drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +5 -0
index 7df278fe492e..e787656b630f 100644
@@ -337,6 +337,8 @@ static u64 execlists_update_context(struct drm_i915_gem_request *rq)
 	if (ppgtt && !i915_vm_is_48bit(&ppgtt->base))
 		execlists_update_context_pdps(ppgtt, reg_state);
 
+	i915_oa_update_reg_state(rq->engine, rq->ctx, reg_state);
+
 	return ce->lrc_desc;
 }
 
@@ -1879,6 +1881,9 @@ static void execlists_init_reg_state(u32 *regs,
 		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
 		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
 			make_rpcs(dev_priv));
+
+		atomic_set(&ctx->engine[RCS].oa_state_dirty, 1);
+		i915_oa_update_reg_state(engine, ctx, regs);
 	}
 }
 
include/uapi/drm/i915_drm.h +12 -7
index 689fd7a418a7..20d65a317492 100644
@@ -1303,13 +1303,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
 	I915_OA_FORMAT_MAX	    /* non-ABI */
 };
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v4 12/15
#13 lore raw

Re: [PATCH v4 12/15] drm/i915/perf: Add OA unit support for Gen 8+

Quoted 47 lines · Robert Bragg

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

Quoted 2 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#14 lore raw

PATCH v4 13/15 drm/i915/perf: Add more OA configs for BDW, CHV, SKL + BXT

In reply to Robert Bragg

These are auto generated from an XML description of metric sets, currently maintained in gputop, ref:

https://github.com/rib/gputop

Quoted 2 lines

gputop-data/oa-*.xml
scripts/i915-perf-kernelgen.py

 $ make -C gputop-data -f Makefile.xml
---
 drivers/gpu/drm/i915/i915_oa_bdw.c    | 4856 ++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_oa_bxt.c    | 2323 +++++++++++++++-
 drivers/gpu/drm/i915/i915_oa_chv.c    | 2525 ++++++++++++++++-
 drivers/gpu/drm/i915/i915_oa_hsw.c    |   58 +-
 drivers/gpu/drm/i915/i915_oa_sklgt2.c | 3156 ++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_oa_sklgt3.c | 2702 +++++++++++++++++-
 drivers/gpu/drm/i915/i915_oa_sklgt4.c | 2745 ++++++++++++++++++-
 7 files changed, 18176 insertions(+), 189 deletions(-)
drivers/gpu/drm/i915/i915_oa_bdw.c +4815 -41
index b0b1b75fb431..eff198ea639b 100644
@@ -31,9 +31,30 @@
 
 enum metric_set_id {
 	METRIC_SET_ID_RENDER_BASIC = 1,
+	METRIC_SET_ID_COMPUTE_BASIC,
+	METRIC_SET_ID_RENDER_PIPE_PROFILE,
+	METRIC_SET_ID_MEMORY_READS,
+	METRIC_SET_ID_MEMORY_WRITES,
+	METRIC_SET_ID_COMPUTE_EXTENDED,
+	METRIC_SET_ID_COMPUTE_L3_CACHE,
+	METRIC_SET_ID_DATA_PORT_READS_COALESCING,
+	METRIC_SET_ID_DATA_PORT_WRITES_COALESCING,
+	METRIC_SET_ID_HDC_AND_SF,
+	METRIC_SET_ID_L3_1,
+	METRIC_SET_ID_L3_2,
+	METRIC_SET_ID_L3_3,
+	METRIC_SET_ID_L3_4,
+	METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND,
+	METRIC_SET_ID_SAMPLER_1,
+	METRIC_SET_ID_SAMPLER_2,
+	METRIC_SET_ID_TDL_1,
+	METRIC_SET_ID_TDL_2,
+	METRIC_SET_ID_COMPUTE_EXTRA,
+	METRIC_SET_ID_VME_PIPE,
+	METRIC_SET_ID_TEST_OA,
 };
 
-int i915_oa_n_builtin_metric_sets_bdw = 1;
+int i915_oa_n_builtin_metric_sets_bdw = 22;
 
 static const struct i915_oa_reg b_counter_config_render_basic[] = {
 	{ _MMIO(0x2710), 0x00000000 },
@@ -290,66 +311,4609 @@ get_render_basic_mux_config(struct drm_i915_private *dev_priv,
 	return NULL;
 }
 
+static const struct i915_oa_reg b_counter_config_compute_basic[] = {
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0x00800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x00800000 },
+	{ _MMIO(0x2740), 0x00000000 },
+};
+
+static const struct i915_oa_reg flex_eu_config_compute_basic[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00000003 },
+	{ _MMIO(0xe658), 0x00002001 },
+	{ _MMIO(0xe758), 0x00778008 },
+	{ _MMIO(0xe45c), 0x00088078 },
+	{ _MMIO(0xe55c), 0x00808708 },
+	{ _MMIO(0xe65c), 0x00a08908 },
+};
+
+static const struct i915_oa_reg mux_config_compute_basic_0_slices_0x01[] = {
+	{ _MMIO(0x9888), 0x105c00e0 },
+	{ _MMIO(0x9888), 0x105800e0 },
+	{ _MMIO(0x9888), 0x103800e0 },
+	{ _MMIO(0x9888), 0x3580001a },
+	{ _MMIO(0x9888), 0x3b800060 },
+	{ _MMIO(0x9888), 0x3d800005 },
+	{ _MMIO(0x9888), 0x065c2100 },
+	{ _MMIO(0x9888), 0x0a5c0041 },
+	{ _MMIO(0x9888), 0x0c5c6600 },
+	{ _MMIO(0x9888), 0x005c6580 },
+	{ _MMIO(0x9888), 0x085c8000 },
+	{ _MMIO(0x9888), 0x0e5c8000 },
+	{ _MMIO(0x9888), 0x00580042 },
+	{ _MMIO(0x9888), 0x08582080 },
+	{ _MMIO(0x9888), 0x0c58004c },
+	{ _MMIO(0x9888), 0x0e582580 },
+	{ _MMIO(0x9888), 0x005b4000 },
+	{ _MMIO(0x9888), 0x185b1000 },
+	{ _MMIO(0x9888), 0x1a5b0104 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1faa00 },
+	{ _MMIO(0x9888), 0x101f02aa },
+	{ _MMIO(0x9888), 0x08380042 },
+	{ _MMIO(0x9888), 0x0a382080 },
+	{ _MMIO(0x9888), 0x0e38404c },
+	{ _MMIO(0x9888), 0x0238404b },
+	{ _MMIO(0x9888), 0x00384000 },
+	{ _MMIO(0x9888), 0x16380000 },
+	{ _MMIO(0x9888), 0x18381145 },
+	{ _MMIO(0x9888), 0x04380000 },
+	{ _MMIO(0x9888), 0x0039a000 },
+	{ _MMIO(0x9888), 0x06398000 },
+	{ _MMIO(0x9888), 0x0839a000 },
+	{ _MMIO(0x9888), 0x0a39a000 },
+	{ _MMIO(0x9888), 0x0c39a000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x02392000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8aaaa0 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x238b02a0 },
+	{ _MMIO(0x9888), 0x258b5550 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x1f850a80 },
+	{ _MMIO(0x9888), 0x2185aaa0 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x03844000 },
+	{ _MMIO(0x9888), 0x17808137 },
+	{ _MMIO(0x9888), 0x1980c147 },
+	{ _MMIO(0x9888), 0x1b80c0e5 },
+	{ _MMIO(0x9888), 0x1d80c0e3 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x13804000 },
+	{ _MMIO(0x9888), 0x15800000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d801000 },
+	{ _MMIO(0x9888), 0x4f800111 },
+	{ _MMIO(0x9888), 0x43800062 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800062 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800062 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f801062 },
+	{ _MMIO(0x9888), 0x41801084 },
+};
+
+static const struct i915_oa_reg mux_config_compute_basic_2_slices_0x02[] = {
+	{ _MMIO(0x9888), 0x10dc00e0 },
+	{ _MMIO(0x9888), 0x10d800e0 },
+	{ _MMIO(0x9888), 0x10b800e0 },
+	{ _MMIO(0x9888), 0x3580001a },
+	{ _MMIO(0x9888), 0x3b800060 },
+	{ _MMIO(0x9888), 0x3d800005 },
+	{ _MMIO(0x9888), 0x06dc2100 },
+	{ _MMIO(0x9888), 0x0adc0041 },
+	{ _MMIO(0x9888), 0x0cdc6600 },
+	{ _MMIO(0x9888), 0x00dc6580 },
+	{ _MMIO(0x9888), 0x08dc8000 },
+	{ _MMIO(0x9888), 0x0edc8000 },
+	{ _MMIO(0x9888), 0x00d80042 },
+	{ _MMIO(0x9888), 0x08d82080 },
+	{ _MMIO(0x9888), 0x0cd8004c },
+	{ _MMIO(0x9888), 0x0ed82580 },
+	{ _MMIO(0x9888), 0x00db4000 },
+	{ _MMIO(0x9888), 0x18db1000 },
+	{ _MMIO(0x9888), 0x1adb0104 },
+	{ _MMIO(0x9888), 0x0c9fa800 },
+	{ _MMIO(0x9888), 0x0e9faa00 },
+	{ _MMIO(0x9888), 0x109f02aa },
+	{ _MMIO(0x9888), 0x08b80042 },
+	{ _MMIO(0x9888), 0x0ab82080 },
+	{ _MMIO(0x9888), 0x0eb8404c },
+	{ _MMIO(0x9888), 0x02b8404b },
+	{ _MMIO(0x9888), 0x00b84000 },
+	{ _MMIO(0x9888), 0x16b80000 },
+	{ _MMIO(0x9888), 0x18b81145 },
+	{ _MMIO(0x9888), 0x04b80000 },
+	{ _MMIO(0x9888), 0x00b9a000 },
+	{ _MMIO(0x9888), 0x06b98000 },
+	{ _MMIO(0x9888), 0x08b9a000 },
+	{ _MMIO(0x9888), 0x0ab9a000 },
+	{ _MMIO(0x9888), 0x0cb9a000 },
+	{ _MMIO(0x9888), 0x0eb9a000 },
+	{ _MMIO(0x9888), 0x02b92000 },
+	{ _MMIO(0x9888), 0x01888000 },
+	{ _MMIO(0x9888), 0x0d88f800 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x238b0540 },
+	{ _MMIO(0x9888), 0x258baaa0 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x018c4000 },
+	{ _MMIO(0x9888), 0x0f8c4000 },
+	{ _MMIO(0x9888), 0x178c2000 },
+	{ _MMIO(0x9888), 0x198c5500 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x018da000 },
+	{ _MMIO(0x9888), 0x078d8000 },
+	{ _MMIO(0x9888), 0x098da000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x038d2000 },
+	{ _MMIO(0x9888), 0x1f850a80 },
+	{ _MMIO(0x9888), 0x2185aaa0 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x03844000 },
+	{ _MMIO(0x9888), 0x17808137 },
+	{ _MMIO(0x9888), 0x1980c147 },
+	{ _MMIO(0x9888), 0x1b80c0e5 },
+	{ _MMIO(0x9888), 0x1d80c0e3 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x13804000 },
+	{ _MMIO(0x9888), 0x15800000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d805000 },
+	{ _MMIO(0x9888), 0x4f800555 },
+	{ _MMIO(0x9888), 0x43800062 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800062 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800062 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800062 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg *
+get_compute_basic_mux_config(struct drm_i915_private *dev_priv,
+			     int *len)
+{
+	if (INTEL_INFO(dev_priv)->sseu.slice_mask & 0x01) {
+		*len = ARRAY_SIZE(mux_config_compute_basic_0_slices_0x01);
+		return mux_config_compute_basic_0_slices_0x01;
+	} else if (INTEL_INFO(dev_priv)->sseu.slice_mask & 0x02) {
+		*len = ARRAY_SIZE(mux_config_compute_basic_2_slices_0x02);
+		return mux_config_compute_basic_2_slices_0x02;
+	}
+
+	*len = 0;
+	return NULL;
+}
+
+static const struct i915_oa_reg b_counter_config_render_pipe_profile[] = {
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007ffea },
+	{ _MMIO(0x2774), 0x00007ffc },
+	{ _MMIO(0x2778), 0x0007affa },
+	{ _MMIO(0x277c), 0x0000f5fd },
+	{ _MMIO(0x2780), 0x00079ffa },
+	{ _MMIO(0x2784), 0x0000f3fb },
+	{ _MMIO(0x2788), 0x0007bf7a },
+	{ _MMIO(0x278c), 0x0000f7e7 },
+	{ _MMIO(0x2790), 0x0007fefa },
+	{ _MMIO(0x2794), 0x0000f7cf },
+	{ _MMIO(0x2798), 0x00077ffa },
+	{ _MMIO(0x279c), 0x0000efdf },
+	{ _MMIO(0x27a0), 0x0006fffa },
+	{ _MMIO(0x27a4), 0x0000cfbf },
+	{ _MMIO(0x27a8), 0x0003fffa },
+	{ _MMIO(0x27ac), 0x00005f7f },
+};
+
+static const struct i915_oa_reg flex_eu_config_render_pipe_profile[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00015014 },
+	{ _MMIO(0xe658), 0x00025024 },
+	{ _MMIO(0xe758), 0x00035034 },
+	{ _MMIO(0xe45c), 0x00045044 },
+	{ _MMIO(0xe55c), 0x00055054 },
+	{ _MMIO(0xe65c), 0x00065064 },
+};
+
+static const struct i915_oa_reg mux_config_render_pipe_profile[] = {
+	{ _MMIO(0x9888), 0x0a1e0000 },
+	{ _MMIO(0x9888), 0x0c1f000f },
+	{ _MMIO(0x9888), 0x10176800 },
+	{ _MMIO(0x9888), 0x1191001f },
+	{ _MMIO(0x9888), 0x0b880320 },
+	{ _MMIO(0x9888), 0x01890c40 },
+	{ _MMIO(0x9888), 0x118a1c00 },
+	{ _MMIO(0x9888), 0x118d7c00 },
+	{ _MMIO(0x9888), 0x118e0020 },
+	{ _MMIO(0x9888), 0x118f4c00 },
+	{ _MMIO(0x9888), 0x11900000 },
+	{ _MMIO(0x9888), 0x13900001 },
+	{ _MMIO(0x9888), 0x065c4000 },
+	{ _MMIO(0x9888), 0x0c3d8000 },
+	{ _MMIO(0x9888), 0x06584000 },
+	{ _MMIO(0x9888), 0x0c5b4000 },
+	{ _MMIO(0x9888), 0x081e0040 },
+	{ _MMIO(0x9888), 0x0e1e0000 },
+	{ _MMIO(0x9888), 0x021f5400 },
+	{ _MMIO(0x9888), 0x001f0000 },
+	{ _MMIO(0x9888), 0x101f0010 },
+	{ _MMIO(0x9888), 0x0e1f0080 },
+	{ _MMIO(0x9888), 0x0c384000 },
+	{ _MMIO(0x9888), 0x06392000 },
+	{ _MMIO(0x9888), 0x0c13c000 },
+	{ _MMIO(0x9888), 0x06164000 },
+	{ _MMIO(0x9888), 0x06170012 },
+	{ _MMIO(0x9888), 0x00170000 },
+	{ _MMIO(0x9888), 0x01910005 },
+	{ _MMIO(0x9888), 0x07880002 },
+	{ _MMIO(0x9888), 0x01880c00 },
+	{ _MMIO(0x9888), 0x0f880000 },
+	{ _MMIO(0x9888), 0x0d880000 },
+	{ _MMIO(0x9888), 0x05880000 },
+	{ _MMIO(0x9888), 0x09890032 },
+	{ _MMIO(0x9888), 0x078a0800 },
+	{ _MMIO(0x9888), 0x0f8a0a00 },
+	{ _MMIO(0x9888), 0x198a4000 },
+	{ _MMIO(0x9888), 0x1b8a2000 },
+	{ _MMIO(0x9888), 0x1d8a0000 },
+	{ _MMIO(0x9888), 0x038a4000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x238b54c0 },
+	{ _MMIO(0x9888), 0x258baa55 },
+	{ _MMIO(0x9888), 0x278b0019 },
+	{ _MMIO(0x9888), 0x198c0100 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x0f8d0015 },
+	{ _MMIO(0x9888), 0x018d1000 },
+	{ _MMIO(0x9888), 0x098d8000 },
+	{ _MMIO(0x9888), 0x0b8df000 },
+	{ _MMIO(0x9888), 0x0d8d3000 },
+	{ _MMIO(0x9888), 0x038de000 },
+	{ _MMIO(0x9888), 0x058d3000 },
+	{ _MMIO(0x9888), 0x0d8e0004 },
+	{ _MMIO(0x9888), 0x058e000c },
+	{ _MMIO(0x9888), 0x098e0000 },
+	{ _MMIO(0x9888), 0x078e0000 },
+	{ _MMIO(0x9888), 0x038e0000 },
+	{ _MMIO(0x9888), 0x0b8f0020 },
+	{ _MMIO(0x9888), 0x198f0c00 },
+	{ _MMIO(0x9888), 0x078f8000 },
+	{ _MMIO(0x9888), 0x098f4000 },
+	{ _MMIO(0x9888), 0x0b900980 },
+	{ _MMIO(0x9888), 0x03900d80 },
+	{ _MMIO(0x9888), 0x01900000 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaaa },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x0784c000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x1780c000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d801111 },
+	{ _MMIO(0x9888), 0x3d800800 },
+	{ _MMIO(0x9888), 0x4f801011 },
+	{ _MMIO(0x9888), 0x43800443 },
+	{ _MMIO(0x9888), 0x51801111 },
+	{ _MMIO(0x9888), 0x45800422 },
+	{ _MMIO(0x9888), 0x53801111 },
+	{ _MMIO(0x9888), 0x47800c60 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800422 },
+	{ _MMIO(0x9888), 0x41800021 },
+};
+
+static const struct i915_oa_reg *
+get_render_pipe_profile_mux_config(struct drm_i915_private *dev_priv,
+				   int *len)
+{
+	*len = ARRAY_SIZE(mux_config_render_pipe_profile);
+	return mux_config_render_pipe_profile;
+}
+
+static const struct i915_oa_reg b_counter_config_memory_reads[] = {
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x274c), 0x86543210 },
+	{ _MMIO(0x2748), 0x86543210 },
+	{ _MMIO(0x2744), 0x00006667 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x275c), 0x86543210 },
+	{ _MMIO(0x2758), 0x86543210 },
+	{ _MMIO(0x2754), 0x00006465 },
+	{ _MMIO(0x2750), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007f81a },
+	{ _MMIO(0x2774), 0x0000fe00 },
+	{ _MMIO(0x2778), 0x0007f82a },
+	{ _MMIO(0x277c), 0x0000fe00 },
+	{ _MMIO(0x2780), 0x0007f872 },
+	{ _MMIO(0x2784), 0x0000fe00 },
+	{ _MMIO(0x2788), 0x0007f8ba },
+	{ _MMIO(0x278c), 0x0000fe00 },
+	{ _MMIO(0x2790), 0x0007f87a },
+	{ _MMIO(0x2794), 0x0000fe00 },
+	{ _MMIO(0x2798), 0x0007f8ea },
+	{ _MMIO(0x279c), 0x0000fe00 },
+	{ _MMIO(0x27a0), 0x0007f8e2 },
+	{ _MMIO(0x27a4), 0x0000fe00 },
+	{ _MMIO(0x27a8), 0x0007f8f2 },
+	{ _MMIO(0x27ac), 0x0000fe00 },
+};
+
+static const struct i915_oa_reg flex_eu_config_memory_reads[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00015014 },
+	{ _MMIO(0xe658), 0x00025024 },
+	{ _MMIO(0xe758), 0x00035034 },
+	{ _MMIO(0xe45c), 0x00045044 },
+	{ _MMIO(0xe55c), 0x00055054 },
+	{ _MMIO(0xe65c), 0x00065064 },
+};
+
+static const struct i915_oa_reg mux_config_memory_reads[] = {
+	{ _MMIO(0x9888), 0x198b0343 },
+	{ _MMIO(0x9888), 0x13845800 },
+	{ _MMIO(0x9888), 0x15840018 },
+	{ _MMIO(0x9888), 0x3580001a },
+	{ _MMIO(0x9888), 0x038b6300 },
+	{ _MMIO(0x9888), 0x058b6b62 },
+	{ _MMIO(0x9888), 0x078b006a },
+	{ _MMIO(0x9888), 0x118b0000 },
+	{ _MMIO(0x9888), 0x238b0000 },
+	{ _MMIO(0x9888), 0x258b0000 },
+	{ _MMIO(0x9888), 0x1f85a080 },
+	{ _MMIO(0x9888), 0x2185aaaa },
+	{ _MMIO(0x9888), 0x2385000a },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x01840018 },
+	{ _MMIO(0x9888), 0x07844c80 },
+	{ _MMIO(0x9888), 0x09840d9a },
+	{ _MMIO(0x9888), 0x0b840e9c },
+	{ _MMIO(0x9888), 0x0d840f9e },
+	{ _MMIO(0x9888), 0x0f840010 },
+	{ _MMIO(0x9888), 0x11840000 },
+	{ _MMIO(0x9888), 0x03848000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x2f8000e5 },
+	{ _MMIO(0x9888), 0x138080e3 },
+	{ _MMIO(0x9888), 0x1580c0e1 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x11804000 },
+	{ _MMIO(0x9888), 0x1780c000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f804000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800800 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800842 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800842 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47801042 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800084 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg *
+get_memory_reads_mux_config(struct drm_i915_private *dev_priv,
+			    int *len)
+{
+	*len = ARRAY_SIZE(mux_config_memory_reads);
+	return mux_config_memory_reads;
+}
+
+static const struct i915_oa_reg b_counter_config_memory_writes[] = {
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x274c), 0x86543210 },
+	{ _MMIO(0x2748), 0x86543210 },
+	{ _MMIO(0x2744), 0x00006667 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x275c), 0x86543210 },
+	{ _MMIO(0x2758), 0x86543210 },
+	{ _MMIO(0x2754), 0x00006465 },
+	{ _MMIO(0x2750), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007f81a },
+	{ _MMIO(0x2774), 0x0000fe00 },
+	{ _MMIO(0x2778), 0x0007f82a },
+	{ _MMIO(0x277c), 0x0000fe00 },
+	{ _MMIO(0x2780), 0x0007f822 },
+	{ _MMIO(0x2784), 0x0000fe00 },
+	{ _MMIO(0x2788), 0x0007f8ba },
+	{ _MMIO(0x278c), 0x0000fe00 },
+	{ _MMIO(0x2790), 0x0007f87a },
+	{ _MMIO(0x2794), 0x0000fe00 },
+	{ _MMIO(0x2798), 0x0007f8ea },
+	{ _MMIO(0x279c), 0x0000fe00 },
+	{ _MMIO(0x27a0), 0x0007f8e2 },
+	{ _MMIO(0x27a4), 0x0000fe00 },
+	{ _MMIO(0x27a8), 0x0007f8f2 },
+	{ _MMIO(0x27ac), 0x0000fe00 },
+};
+
+static const struct i915_oa_reg flex_eu_config_memory_writes[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00015014 },
+	{ _MMIO(0xe658), 0x00025024 },
+	{ _MMIO(0xe758), 0x00035034 },
+	{ _MMIO(0xe45c), 0x00045044 },
+	{ _MMIO(0xe55c), 0x00055054 },
+	{ _MMIO(0xe65c), 0x00065064 },
+};
+
+static const struct i915_oa_reg mux_config_memory_writes[] = {
+	{ _MMIO(0x9888), 0x198b0343 },
+	{ _MMIO(0x9888), 0x13845400 },
+	{ _MMIO(0x9888), 0x3580001a },
+	{ _MMIO(0x9888), 0x3d800805 },
+	{ _MMIO(0x9888), 0x038b6300 },
+	{ _MMIO(0x9888), 0x058b6b62 },
+	{ _MMIO(0x9888), 0x078b006a },
+	{ _MMIO(0x9888), 0x118b0000 },
+	{ _MMIO(0x9888), 0x238b0000 },
+	{ _MMIO(0x9888), 0x258b0000 },
+	{ _MMIO(0x9888), 0x1f85a080 },
+	{ _MMIO(0x9888), 0x2185aaaa },
+	{ _MMIO(0x9888), 0x23850002 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x01840010 },
+	{ _MMIO(0x9888), 0x07844880 },
+	{ _MMIO(0x9888), 0x09840992 },
+	{ _MMIO(0x9888), 0x0b840a94 },
+	{ _MMIO(0x9888), 0x0d840b96 },
+	{ _MMIO(0x9888), 0x11840000 },
+	{ _MMIO(0x9888), 0x03848000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x2d800147 },
+	{ _MMIO(0x9888), 0x2f8000e5 },
+	{ _MMIO(0x9888), 0x138080e3 },
+	{ _MMIO(0x9888), 0x1580c0e1 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x11804000 },
+	{ _MMIO(0x9888), 0x1780c000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f800000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800842 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800842 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47801082 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800084 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg *
+get_memory_writes_mux_config(struct drm_i915_private *dev_priv,
+			     int *len)
+{
+	*len = ARRAY_SIZE(mux_config_memory_writes);
+	return mux_config_memory_writes;
+}
+
+static const struct i915_oa_reg b_counter_config_compute_extended[] = {
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007fc2a },
+	{ _MMIO(0x2774), 0x0000bf00 },
+	{ _MMIO(0x2778), 0x0007fc6a },
+	{ _MMIO(0x277c), 0x0000bf00 },
+	{ _MMIO(0x2780), 0x0007fc92 },
+	{ _MMIO(0x2784), 0x0000bf00 },
+	{ _MMIO(0x2788), 0x0007fca2 },
+	{ _MMIO(0x278c), 0x0000bf00 },
+	{ _MMIO(0x2790), 0x0007fc32 },
+	{ _MMIO(0x2794), 0x0000bf00 },
+	{ _MMIO(0x2798), 0x0007fc9a },
+	{ _MMIO(0x279c), 0x0000bf00 },
+	{ _MMIO(0x27a0), 0x0007fe6a },
+	{ _MMIO(0x27a4), 0x0000bf00 },
+	{ _MMIO(0x27a8), 0x0007fe7a },
+	{ _MMIO(0x27ac), 0x0000bf00 },
+};
+
+static const struct i915_oa_reg flex_eu_config_compute_extended[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00000003 },
+	{ _MMIO(0xe658), 0x00002001 },
+	{ _MMIO(0xe758), 0x00778008 },
+	{ _MMIO(0xe45c), 0x00088078 },
+	{ _MMIO(0xe55c), 0x00808708 },
+	{ _MMIO(0xe65c), 0x00a08908 },
+};
+
+static const struct i915_oa_reg mux_config_compute_extended_0_subslices_0x01[] = {
+	{ _MMIO(0x9888), 0x143d0160 },
+	{ _MMIO(0x9888), 0x163d2800 },
+	{ _MMIO(0x9888), 0x183d0120 },
+	{ _MMIO(0x9888), 0x105800e0 },
+	{ _MMIO(0x9888), 0x005cc000 },
+	{ _MMIO(0x9888), 0x065c8000 },
+	{ _MMIO(0x9888), 0x085cc000 },
+	{ _MMIO(0x9888), 0x0a5cc000 },
+	{ _MMIO(0x9888), 0x0c5cc000 },
+	{ _MMIO(0x9888), 0x0e5cc000 },
+	{ _MMIO(0x9888), 0x025cc000 },
+	{ _MMIO(0x9888), 0x045cc000 },
+	{ _MMIO(0x9888), 0x003d0011 },
+	{ _MMIO(0x9888), 0x063d0900 },
+	{ _MMIO(0x9888), 0x083d0a13 },
+	{ _MMIO(0x9888), 0x0a3d0b15 },
+	{ _MMIO(0x9888), 0x0c3d2317 },
+	{ _MMIO(0x9888), 0x043d21b7 },
+	{ _MMIO(0x9888), 0x103d0000 },
+	{ _MMIO(0x9888), 0x0e3d0000 },
+	{ _MMIO(0x9888), 0x1a3d0000 },
+	{ _MMIO(0x9888), 0x0e5825c1 },
+	{ _MMIO(0x9888), 0x00586100 },
+	{ _MMIO(0x9888), 0x0258204c },
+	{ _MMIO(0x9888), 0x06588000 },
+	{ _MMIO(0x9888), 0x0858c000 },
+	{ _MMIO(0x9888), 0x0a58c000 },
+	{ _MMIO(0x9888), 0x0c58c000 },
+	{ _MMIO(0x9888), 0x0458c000 },
+	{ _MMIO(0x9888), 0x005b4000 },
+	{ _MMIO(0x9888), 0x0e5b4000 },
+	{ _MMIO(0x9888), 0x185b5400 },
+	{ _MMIO(0x9888), 0x1a5b0155 },
+	{ _MMIO(0x9888), 0x025b4000 },
+	{ _MMIO(0x9888), 0x045b4000 },
+	{ _MMIO(0x9888), 0x065b4000 },
+	{ _MMIO(0x9888), 0x085b4000 },
+	{ _MMIO(0x9888), 0x0a5b4000 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1faa2a },
+	{ _MMIO(0x9888), 0x101f02aa },
+	{ _MMIO(0x9888), 0x00384000 },
+	{ _MMIO(0x9888), 0x0e384000 },
+	{ _MMIO(0x9888), 0x16384000 },
+	{ _MMIO(0x9888), 0x18381555 },
+	{ _MMIO(0x9888), 0x02384000 },
+	{ _MMIO(0x9888), 0x04384000 },
+	{ _MMIO(0x9888), 0x06384000 },
+	{ _MMIO(0x9888), 0x08384000 },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x0039a000 },
+	{ _MMIO(0x9888), 0x06398000 },
+	{ _MMIO(0x9888), 0x0839a000 },
+	{ _MMIO(0x9888), 0x0a39a000 },
+	{ _MMIO(0x9888), 0x0c39a000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x0239a000 },
+	{ _MMIO(0x9888), 0x0439a000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8aaaa0 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x238b2aa0 },
+	{ _MMIO(0x9888), 0x258b5551 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa2 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg mux_config_compute_extended_2_subslices_0x02[] = {
+	{ _MMIO(0x9888), 0x105c00e0 },
+	{ _MMIO(0x9888), 0x145b0160 },
+	{ _MMIO(0x9888), 0x165b2800 },
+	{ _MMIO(0x9888), 0x185b0120 },
+	{ _MMIO(0x9888), 0x0e5c25c1 },
+	{ _MMIO(0x9888), 0x005c6100 },
+	{ _MMIO(0x9888), 0x025c204c },
+	{ _MMIO(0x9888), 0x065c8000 },
+	{ _MMIO(0x9888), 0x085cc000 },
+	{ _MMIO(0x9888), 0x0a5cc000 },
+	{ _MMIO(0x9888), 0x0c5cc000 },
+	{ _MMIO(0x9888), 0x045cc000 },
+	{ _MMIO(0x9888), 0x005b0011 },
+	{ _MMIO(0x9888), 0x065b0900 },
+	{ _MMIO(0x9888), 0x085b0a13 },
+	{ _MMIO(0x9888), 0x0a5b0b15 },
+	{ _MMIO(0x9888), 0x0c5b2317 },
+	{ _MMIO(0x9888), 0x045b21b7 },
+	{ _MMIO(0x9888), 0x105b0000 },
+	{ _MMIO(0x9888), 0x0e5b0000 },
+	{ _MMIO(0x9888), 0x1a5b0000 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1faa2a },
+	{ _MMIO(0x9888), 0x101f02aa },
+	{ _MMIO(0x9888), 0x00384000 },
+	{ _MMIO(0x9888), 0x0e384000 },
+	{ _MMIO(0x9888), 0x16384000 },
+	{ _MMIO(0x9888), 0x18381555 },
+	{ _MMIO(0x9888), 0x02384000 },
+	{ _MMIO(0x9888), 0x04384000 },
+	{ _MMIO(0x9888), 0x06384000 },
+	{ _MMIO(0x9888), 0x08384000 },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x0039a000 },
+	{ _MMIO(0x9888), 0x06398000 },
+	{ _MMIO(0x9888), 0x0839a000 },
+	{ _MMIO(0x9888), 0x0a39a000 },
+	{ _MMIO(0x9888), 0x0c39a000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x0239a000 },
+	{ _MMIO(0x9888), 0x0439a000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8aaaa0 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x238b2aa0 },
+	{ _MMIO(0x9888), 0x258b5551 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa2 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg mux_config_compute_extended_4_subslices_0x04[] = {
+	{ _MMIO(0x9888), 0x103800e0 },
+	{ _MMIO(0x9888), 0x143a0160 },
+	{ _MMIO(0x9888), 0x163a2800 },
+	{ _MMIO(0x9888), 0x183a0120 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1faa2a },
+	{ _MMIO(0x9888), 0x101f02aa },
+	{ _MMIO(0x9888), 0x0e38a5c1 },
+	{ _MMIO(0x9888), 0x0038a100 },
+	{ _MMIO(0x9888), 0x0238204c },
+	{ _MMIO(0x9888), 0x16388000 },
+	{ _MMIO(0x9888), 0x183802aa },
+	{ _MMIO(0x9888), 0x04380000 },
+	{ _MMIO(0x9888), 0x06380000 },
+	{ _MMIO(0x9888), 0x08388000 },
+	{ _MMIO(0x9888), 0x0a388000 },
+	{ _MMIO(0x9888), 0x0039a000 },
+	{ _MMIO(0x9888), 0x06398000 },
+	{ _MMIO(0x9888), 0x0839a000 },
+	{ _MMIO(0x9888), 0x0a39a000 },
+	{ _MMIO(0x9888), 0x0c39a000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x0239a000 },
+	{ _MMIO(0x9888), 0x0439a000 },
+	{ _MMIO(0x9888), 0x003a0011 },
+	{ _MMIO(0x9888), 0x063a0900 },
+	{ _MMIO(0x9888), 0x083a0a13 },
+	{ _MMIO(0x9888), 0x0a3a0b15 },
+	{ _MMIO(0x9888), 0x0c3a2317 },
+	{ _MMIO(0x9888), 0x043a21b7 },
+	{ _MMIO(0x9888), 0x103a0000 },
+	{ _MMIO(0x9888), 0x0e3a0000 },
+	{ _MMIO(0x9888), 0x1a3a0000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8aaaa0 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x238b2aa0 },
+	{ _MMIO(0x9888), 0x258b5551 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa2 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg mux_config_compute_extended_1_subslices_0x08[] = {
+	{ _MMIO(0x9888), 0x14bd0160 },
+	{ _MMIO(0x9888), 0x16bd2800 },
+	{ _MMIO(0x9888), 0x18bd0120 },
+	{ _MMIO(0x9888), 0x10d800e0 },
+	{ _MMIO(0x9888), 0x00dcc000 },
+	{ _MMIO(0x9888), 0x06dc8000 },
+	{ _MMIO(0x9888), 0x08dcc000 },
+	{ _MMIO(0x9888), 0x0adcc000 },
+	{ _MMIO(0x9888), 0x0cdcc000 },
+	{ _MMIO(0x9888), 0x0edcc000 },
+	{ _MMIO(0x9888), 0x02dcc000 },
+	{ _MMIO(0x9888), 0x04dcc000 },
+	{ _MMIO(0x9888), 0x00bd0011 },
+	{ _MMIO(0x9888), 0x06bd0900 },
+	{ _MMIO(0x9888), 0x08bd0a13 },
+	{ _MMIO(0x9888), 0x0abd0b15 },
+	{ _MMIO(0x9888), 0x0cbd2317 },
+	{ _MMIO(0x9888), 0x04bd21b7 },
+	{ _MMIO(0x9888), 0x10bd0000 },
+	{ _MMIO(0x9888), 0x0ebd0000 },
+	{ _MMIO(0x9888), 0x1abd0000 },
+	{ _MMIO(0x9888), 0x0ed825c1 },
+	{ _MMIO(0x9888), 0x00d86100 },
+	{ _MMIO(0x9888), 0x02d8204c },
+	{ _MMIO(0x9888), 0x06d88000 },
+	{ _MMIO(0x9888), 0x08d8c000 },
+	{ _MMIO(0x9888), 0x0ad8c000 },
+	{ _MMIO(0x9888), 0x0cd8c000 },
+	{ _MMIO(0x9888), 0x04d8c000 },
+	{ _MMIO(0x9888), 0x00db4000 },
+	{ _MMIO(0x9888), 0x0edb4000 },
+	{ _MMIO(0x9888), 0x18db5400 },
+	{ _MMIO(0x9888), 0x1adb0155 },
+	{ _MMIO(0x9888), 0x02db4000 },
+	{ _MMIO(0x9888), 0x04db4000 },
+	{ _MMIO(0x9888), 0x06db4000 },
+	{ _MMIO(0x9888), 0x08db4000 },
+	{ _MMIO(0x9888), 0x0adb4000 },
+	{ _MMIO(0x9888), 0x0c9fa800 },
+	{ _MMIO(0x9888), 0x0e9faa2a },
+	{ _MMIO(0x9888), 0x109f02aa },
+	{ _MMIO(0x9888), 0x00b84000 },
+	{ _MMIO(0x9888), 0x0eb84000 },
+	{ _MMIO(0x9888), 0x16b84000 },
+	{ _MMIO(0x9888), 0x18b81555 },
+	{ _MMIO(0x9888), 0x02b84000 },
+	{ _MMIO(0x9888), 0x04b84000 },
+	{ _MMIO(0x9888), 0x06b84000 },
+	{ _MMIO(0x9888), 0x08b84000 },
+	{ _MMIO(0x9888), 0x0ab84000 },
+	{ _MMIO(0x9888), 0x00b9a000 },
+	{ _MMIO(0x9888), 0x06b98000 },
+	{ _MMIO(0x9888), 0x08b9a000 },
+	{ _MMIO(0x9888), 0x0ab9a000 },
+	{ _MMIO(0x9888), 0x0cb9a000 },
+	{ _MMIO(0x9888), 0x0eb9a000 },
+	{ _MMIO(0x9888), 0x02b9a000 },
+	{ _MMIO(0x9888), 0x04b9a000 },
+	{ _MMIO(0x9888), 0x01888000 },
+	{ _MMIO(0x9888), 0x0d88f800 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x07888000 },
+	{ _MMIO(0x9888), 0x09888000 },
+	{ _MMIO(0x9888), 0x0b888000 },
+	{ _MMIO(0x9888), 0x238b5540 },
+	{ _MMIO(0x9888), 0x258baaa2 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x018c4000 },
+	{ _MMIO(0x9888), 0x0f8c4000 },
+	{ _MMIO(0x9888), 0x178c2000 },
+	{ _MMIO(0x9888), 0x198c5500 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x078c4000 },
+	{ _MMIO(0x9888), 0x098c4000 },
+	{ _MMIO(0x9888), 0x0b8c4000 },
+	{ _MMIO(0x9888), 0x018da000 },
+	{ _MMIO(0x9888), 0x078d8000 },
+	{ _MMIO(0x9888), 0x098da000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x038da000 },
+	{ _MMIO(0x9888), 0x058da000 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa2 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg mux_config_compute_extended_3_subslices_0x10[] = {
+	{ _MMIO(0x9888), 0x10dc00e0 },
+	{ _MMIO(0x9888), 0x14db0160 },
+	{ _MMIO(0x9888), 0x16db2800 },
+	{ _MMIO(0x9888), 0x18db0120 },
+	{ _MMIO(0x9888), 0x0edc25c1 },
+	{ _MMIO(0x9888), 0x00dc6100 },
+	{ _MMIO(0x9888), 0x02dc204c },
+	{ _MMIO(0x9888), 0x06dc8000 },
+	{ _MMIO(0x9888), 0x08dcc000 },
+	{ _MMIO(0x9888), 0x0adcc000 },
+	{ _MMIO(0x9888), 0x0cdcc000 },
+	{ _MMIO(0x9888), 0x04dcc000 },
+	{ _MMIO(0x9888), 0x00db0011 },
+	{ _MMIO(0x9888), 0x06db0900 },
+	{ _MMIO(0x9888), 0x08db0a13 },
+	{ _MMIO(0x9888), 0x0adb0b15 },
+	{ _MMIO(0x9888), 0x0cdb2317 },
+	{ _MMIO(0x9888), 0x04db21b7 },
+	{ _MMIO(0x9888), 0x10db0000 },
+	{ _MMIO(0x9888), 0x0edb0000 },
+	{ _MMIO(0x9888), 0x1adb0000 },
+	{ _MMIO(0x9888), 0x0c9fa800 },
+	{ _MMIO(0x9888), 0x0e9faa2a },
+	{ _MMIO(0x9888), 0x109f02aa },
+	{ _MMIO(0x9888), 0x00b84000 },
+	{ _MMIO(0x9888), 0x0eb84000 },
+	{ _MMIO(0x9888), 0x16b84000 },
+	{ _MMIO(0x9888), 0x18b81555 },
+	{ _MMIO(0x9888), 0x02b84000 },
+	{ _MMIO(0x9888), 0x04b84000 },
+	{ _MMIO(0x9888), 0x06b84000 },
+	{ _MMIO(0x9888), 0x08b84000 },
+	{ _MMIO(0x9888), 0x0ab84000 },
+	{ _MMIO(0x9888), 0x00b9a000 },
+	{ _MMIO(0x9888), 0x06b98000 },
+	{ _MMIO(0x9888), 0x08b9a000 },
+	{ _MMIO(0x9888), 0x0ab9a000 },
+	{ _MMIO(0x9888), 0x0cb9a000 },
+	{ _MMIO(0x9888), 0x0eb9a000 },
+	{ _MMIO(0x9888), 0x02b9a000 },
+	{ _MMIO(0x9888), 0x04b9a000 },
+	{ _MMIO(0x9888), 0x01888000 },
+	{ _MMIO(0x9888), 0x0d88f800 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x07888000 },
+	{ _MMIO(0x9888), 0x09888000 },
+	{ _MMIO(0x9888), 0x0b888000 },
+	{ _MMIO(0x9888), 0x238b5540 },
+	{ _MMIO(0x9888), 0x258baaa2 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x018c4000 },
+	{ _MMIO(0x9888), 0x0f8c4000 },
+	{ _MMIO(0x9888), 0x178c2000 },
+	{ _MMIO(0x9888), 0x198c5500 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x078c4000 },
+	{ _MMIO(0x9888), 0x098c4000 },
+	{ _MMIO(0x9888), 0x0b8c4000 },
+	{ _MMIO(0x9888), 0x018da000 },
+	{ _MMIO(0x9888), 0x078d8000 },
+	{ _MMIO(0x9888), 0x098da000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x038da000 },
+	{ _MMIO(0x9888), 0x058da000 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa2 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg mux_config_compute_extended_5_subslices_0x20[] = {
+	{ _MMIO(0x9888), 0x10b800e0 },
+	{ _MMIO(0x9888), 0x14ba0160 },
+	{ _MMIO(0x9888), 0x16ba2800 },
+	{ _MMIO(0x9888), 0x18ba0120 },
+	{ _MMIO(0x9888), 0x0c9fa800 },
+	{ _MMIO(0x9888), 0x0e9faa2a },
+	{ _MMIO(0x9888), 0x109f02aa },
+	{ _MMIO(0x9888), 0x0eb8a5c1 },
+	{ _MMIO(0x9888), 0x00b8a100 },
+	{ _MMIO(0x9888), 0x02b8204c },
+	{ _MMIO(0x9888), 0x16b88000 },
+	{ _MMIO(0x9888), 0x18b802aa },
+	{ _MMIO(0x9888), 0x04b80000 },
+	{ _MMIO(0x9888), 0x06b80000 },
+	{ _MMIO(0x9888), 0x08b88000 },
+	{ _MMIO(0x9888), 0x0ab88000 },
+	{ _MMIO(0x9888), 0x00b9a000 },
+	{ _MMIO(0x9888), 0x06b98000 },
+	{ _MMIO(0x9888), 0x08b9a000 },
+	{ _MMIO(0x9888), 0x0ab9a000 },
+	{ _MMIO(0x9888), 0x0cb9a000 },
+	{ _MMIO(0x9888), 0x0eb9a000 },
+	{ _MMIO(0x9888), 0x02b9a000 },
+	{ _MMIO(0x9888), 0x04b9a000 },
+	{ _MMIO(0x9888), 0x00ba0011 },
+	{ _MMIO(0x9888), 0x06ba0900 },
+	{ _MMIO(0x9888), 0x08ba0a13 },
+	{ _MMIO(0x9888), 0x0aba0b15 },
+	{ _MMIO(0x9888), 0x0cba2317 },
+	{ _MMIO(0x9888), 0x04ba21b7 },
+	{ _MMIO(0x9888), 0x10ba0000 },
+	{ _MMIO(0x9888), 0x0eba0000 },
+	{ _MMIO(0x9888), 0x1aba0000 },
+	{ _MMIO(0x9888), 0x01888000 },
+	{ _MMIO(0x9888), 0x0d88f800 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x07888000 },
+	{ _MMIO(0x9888), 0x09888000 },
+	{ _MMIO(0x9888), 0x0b888000 },
+	{ _MMIO(0x9888), 0x238b5540 },
+	{ _MMIO(0x9888), 0x258baaa2 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x018c4000 },
+	{ _MMIO(0x9888), 0x0f8c4000 },
+	{ _MMIO(0x9888), 0x178c2000 },
+	{ _MMIO(0x9888), 0x198c5500 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x078c4000 },
+	{ _MMIO(0x9888), 0x098c4000 },
+	{ _MMIO(0x9888), 0x0b8c4000 },
+	{ _MMIO(0x9888), 0x018da000 },
+	{ _MMIO(0x9888), 0x078d8000 },
+	{ _MMIO(0x9888), 0x098da000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x038da000 },
+	{ _MMIO(0x9888), 0x058da000 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa2 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800000 },
+};
+
+static const struct i915_oa_reg *
+get_compute_extended_mux_config(struct drm_i915_private *dev_priv,
+				int *len)
+{
+	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x01) {
+		*len = ARRAY_SIZE(mux_config_compute_extended_0_subslices_0x01);
+		return mux_config_compute_extended_0_subslices_0x01;
+	} else if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x02) {
+		*len = ARRAY_SIZE(mux_config_compute_extended_2_subslices_0x02);
+		return mux_config_compute_extended_2_subslices_0x02;
+	} else if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x04) {
+		*len = ARRAY_SIZE(mux_config_compute_extended_4_subslices_0x04);
+		return mux_config_compute_extended_4_subslices_0x04;
+	} else if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x08) {
+		*len = ARRAY_SIZE(mux_config_compute_extended_1_subslices_0x08);
+		return mux_config_compute_extended_1_subslices_0x08;
+	} else if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x10) {
+		*len = ARRAY_SIZE(mux_config_compute_extended_3_subslices_0x10);
+		return mux_config_compute_extended_3_subslices_0x10;
+	} else if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x20) {
+		*len = ARRAY_SIZE(mux_config_compute_extended_5_subslices_0x20);
+		return mux_config_compute_extended_5_subslices_0x20;
+	}
+
+	*len = 0;
+	return NULL;
+}
+
+static const struct i915_oa_reg b_counter_config_compute_l3_cache[] = {
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0x30800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x30800000 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007fffa },
+	{ _MMIO(0x2774), 0x0000fefe },
+	{ _MMIO(0x2778), 0x0007fffa },
+	{ _MMIO(0x277c), 0x0000fefd },
+	{ _MMIO(0x2790), 0x0007fffa },
+	{ _MMIO(0x2794), 0x0000fbef },
+	{ _MMIO(0x2798), 0x0007fffa },
+	{ _MMIO(0x279c), 0x0000fbdf },
+};
+
+static const struct i915_oa_reg flex_eu_config_compute_l3_cache[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00000003 },
+	{ _MMIO(0xe658), 0x00002001 },
+	{ _MMIO(0xe758), 0x00101100 },
+	{ _MMIO(0xe45c), 0x00201200 },
+	{ _MMIO(0xe55c), 0x00301300 },
+	{ _MMIO(0xe65c), 0x00401400 },
+};
+
+static const struct i915_oa_reg mux_config_compute_l3_cache[] = {
+	{ _MMIO(0x9888), 0x143f00b3 },
+	{ _MMIO(0x9888), 0x14bf00b3 },
+	{ _MMIO(0x9888), 0x138303c0 },
+	{ _MMIO(0x9888), 0x3b800060 },
+	{ _MMIO(0x9888), 0x3d800805 },
+	{ _MMIO(0x9888), 0x003f0029 },
+	{ _MMIO(0x9888), 0x063f1400 },
+	{ _MMIO(0x9888), 0x083f1225 },
+	{ _MMIO(0x9888), 0x0e3f1327 },
+	{ _MMIO(0x9888), 0x103f0000 },
+	{ _MMIO(0x9888), 0x005a4000 },
+	{ _MMIO(0x9888), 0x065a8000 },
+	{ _MMIO(0x9888), 0x085ac000 },
+	{ _MMIO(0x9888), 0x0e5ac000 },
+	{ _MMIO(0x9888), 0x001d4000 },
+	{ _MMIO(0x9888), 0x061d8000 },
+	{ _MMIO(0x9888), 0x081dc000 },
+	{ _MMIO(0x9888), 0x0e1dc000 },
+	{ _MMIO(0x9888), 0x0c1f0800 },
+	{ _MMIO(0x9888), 0x0e1f2a00 },
+	{ _MMIO(0x9888), 0x101f0280 },
+	{ _MMIO(0x9888), 0x00391000 },
+	{ _MMIO(0x9888), 0x06394000 },
+	{ _MMIO(0x9888), 0x08395000 },
+	{ _MMIO(0x9888), 0x0e395000 },
+	{ _MMIO(0x9888), 0x0abf1429 },
+	{ _MMIO(0x9888), 0x0cbf1225 },
+	{ _MMIO(0x9888), 0x00bf1380 },
+	{ _MMIO(0x9888), 0x02bf0026 },
+	{ _MMIO(0x9888), 0x10bf0000 },
+	{ _MMIO(0x9888), 0x0adac000 },
+	{ _MMIO(0x9888), 0x0cdac000 },
+	{ _MMIO(0x9888), 0x00da8000 },
+	{ _MMIO(0x9888), 0x02da4000 },
+	{ _MMIO(0x9888), 0x0a9dc000 },
+	{ _MMIO(0x9888), 0x0c9dc000 },
+	{ _MMIO(0x9888), 0x009d8000 },
+	{ _MMIO(0x9888), 0x029d4000 },
+	{ _MMIO(0x9888), 0x0e9f8000 },
+	{ _MMIO(0x9888), 0x109f002a },
+	{ _MMIO(0x9888), 0x0c9fa000 },
+	{ _MMIO(0x9888), 0x0ab95000 },
+	{ _MMIO(0x9888), 0x0cb95000 },
+	{ _MMIO(0x9888), 0x00b94000 },
+	{ _MMIO(0x9888), 0x02b91000 },
+	{ _MMIO(0x9888), 0x0d88c000 },
+	{ _MMIO(0x9888), 0x0f880003 },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8a8020 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x238b0520 },
+	{ _MMIO(0x9888), 0x258ba950 },
+	{ _MMIO(0x9888), 0x278b0016 },
+	{ _MMIO(0x9888), 0x198c5400 },
+	{ _MMIO(0x9888), 0x1b8c0001 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x018d8000 },
+	{ _MMIO(0x9888), 0x038d2000 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaa0 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x03835180 },
+	{ _MMIO(0x9888), 0x05834022 },
+	{ _MMIO(0x9888), 0x11830000 },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x07830000 },
+	{ _MMIO(0x9888), 0x09830000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x07848000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x05844000 },
+	{ _MMIO(0x9888), 0x1b80c137 },
+	{ _MMIO(0x9888), 0x1d80c147 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x17808000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x15804000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d801000 },
+	{ _MMIO(0x9888), 0x4f800111 },
+	{ _MMIO(0x9888), 0x43800842 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800840 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800800 },
+	{ _MMIO(0x9888), 0x418014a2 },
+};
+
+static const struct i915_oa_reg *
+get_compute_l3_cache_mux_config(struct drm_i915_private *dev_priv,
+				int *len)
+{
+	*len = ARRAY_SIZE(mux_config_compute_l3_cache);
+	return mux_config_compute_l3_cache;
+}
+
+static const struct i915_oa_reg b_counter_config_data_port_reads_coalescing[] = {
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x274c), 0xba98ba98 },
+	{ _MMIO(0x2748), 0xba98ba98 },
+	{ _MMIO(0x2744), 0x00003377 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007fff2 },
+	{ _MMIO(0x2774), 0x00007ff0 },
+	{ _MMIO(0x2778), 0x0007ffe2 },
+	{ _MMIO(0x277c), 0x00007ff0 },
+	{ _MMIO(0x2780), 0x0007ffc2 },
+	{ _MMIO(0x2784), 0x00007ff0 },
+	{ _MMIO(0x2788), 0x0007ff82 },
+	{ _MMIO(0x278c), 0x00007ff0 },
+	{ _MMIO(0x2790), 0x0007fffa },
+	{ _MMIO(0x2794), 0x0000bfef },
+	{ _MMIO(0x2798), 0x0007fffa },
+	{ _MMIO(0x279c), 0x0000bfdf },
+	{ _MMIO(0x27a0), 0x0007fffa },
+	{ _MMIO(0x27a4), 0x0000bfbf },
+	{ _MMIO(0x27a8), 0x0007fffa },
+	{ _MMIO(0x27ac), 0x0000bf7f },
+};
+
+static const struct i915_oa_reg flex_eu_config_data_port_reads_coalescing[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00000003 },
+	{ _MMIO(0xe658), 0x00002001 },
+	{ _MMIO(0xe758), 0x00778008 },
+	{ _MMIO(0xe45c), 0x00088078 },
+	{ _MMIO(0xe55c), 0x00808708 },
+	{ _MMIO(0xe65c), 0x00a08908 },
+};
+
+static const struct i915_oa_reg mux_config_data_port_reads_coalescing_0_subslices_0x01[] = {
+	{ _MMIO(0x9888), 0x103d0005 },
+	{ _MMIO(0x9888), 0x163d240b },
+	{ _MMIO(0x9888), 0x1058022f },
+	{ _MMIO(0x9888), 0x185b5520 },
+	{ _MMIO(0x9888), 0x198b0003 },
+	{ _MMIO(0x9888), 0x005cc000 },
+	{ _MMIO(0x9888), 0x065cc000 },
+	{ _MMIO(0x9888), 0x085cc000 },
+	{ _MMIO(0x9888), 0x0a5cc000 },
+	{ _MMIO(0x9888), 0x0c5cc000 },
+	{ _MMIO(0x9888), 0x0e5cc000 },
+	{ _MMIO(0x9888), 0x025c4000 },
+	{ _MMIO(0x9888), 0x045c8000 },
+	{ _MMIO(0x9888), 0x003d0000 },
+	{ _MMIO(0x9888), 0x063d00b0 },
+	{ _MMIO(0x9888), 0x083d0182 },
+	{ _MMIO(0x9888), 0x0a3d10a0 },
+	{ _MMIO(0x9888), 0x0c3d11a2 },
+	{ _MMIO(0x9888), 0x0e3d0000 },
+	{ _MMIO(0x9888), 0x183d0000 },
+	{ _MMIO(0x9888), 0x1a3d0000 },
+	{ _MMIO(0x9888), 0x0e582242 },
+	{ _MMIO(0x9888), 0x00586700 },
+	{ _MMIO(0x9888), 0x0258004f },
+	{ _MMIO(0x9888), 0x0658c000 },
+	{ _MMIO(0x9888), 0x0858c000 },
+	{ _MMIO(0x9888), 0x0a58c000 },
+	{ _MMIO(0x9888), 0x0c58c000 },
+	{ _MMIO(0x9888), 0x045b6300 },
+	{ _MMIO(0x9888), 0x105b0000 },
+	{ _MMIO(0x9888), 0x005b4000 },
+	{ _MMIO(0x9888), 0x0e5b4000 },
+	{ _MMIO(0x9888), 0x1a5b0155 },
+	{ _MMIO(0x9888), 0x025b4000 },
+	{ _MMIO(0x9888), 0x0a5b0000 },
+	{ _MMIO(0x9888), 0x0c5b4000 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1faaa0 },
+	{ _MMIO(0x9888), 0x101f02aa },
+	{ _MMIO(0x9888), 0x00384000 },
+	{ _MMIO(0x9888), 0x0e384000 },
+	{ _MMIO(0x9888), 0x16384000 },
+	{ _MMIO(0x9888), 0x18381555 },
+	{ _MMIO(0x9888), 0x02384000 },
+	{ _MMIO(0x9888), 0x04384000 },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x0c384000 },
+	{ _MMIO(0x9888), 0x0039a000 },
+	{ _MMIO(0x9888), 0x0639a000 },
+	{ _MMIO(0x9888), 0x0839a000 },
+	{ _MMIO(0x9888), 0x0a39a000 },
+	{ _MMIO(0x9888), 0x0c39a000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x02392000 },
+	{ _MMIO(0x9888), 0x04398000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8aaaa0 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x038b6300 },
+	{ _MMIO(0x9888), 0x058b0062 },
+	{ _MMIO(0x9888), 0x118b0000 },
+	{ _MMIO(0x9888), 0x238b02a0 },
+	{ _MMIO(0x9888), 0x258b5555 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaaa },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x0784c000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x1780c000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d801000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800001 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800041 },
+};
+
+static const struct i915_oa_reg *
+get_data_port_reads_coalescing_mux_config(struct drm_i915_private *dev_priv,
+					  int *len)
+{
+	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x01) {
+		*len = ARRAY_SIZE(mux_config_data_port_reads_coalescing_0_subslices_0x01);
+		return mux_config_data_port_reads_coalescing_0_subslices_0x01;
+	}
+
+	*len = 0;
+	return NULL;
+}
+
+static const struct i915_oa_reg b_counter_config_data_port_writes_coalescing[] = {
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x274c), 0xba98ba98 },
+	{ _MMIO(0x2748), 0xba98ba98 },
+	{ _MMIO(0x2744), 0x00003377 },
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2770), 0x0007ff72 },
+	{ _MMIO(0x2774), 0x0000bfd0 },
+	{ _MMIO(0x2778), 0x0007ff62 },
+	{ _MMIO(0x277c), 0x0000bfd0 },
+	{ _MMIO(0x2780), 0x0007ff42 },
+	{ _MMIO(0x2784), 0x0000bfd0 },
+	{ _MMIO(0x2788), 0x0007ff02 },
+	{ _MMIO(0x278c), 0x0000bfd0 },
+	{ _MMIO(0x2790), 0x0005fff2 },
+	{ _MMIO(0x2794), 0x0000bfd0 },
+	{ _MMIO(0x2798), 0x0005ffe2 },
+	{ _MMIO(0x279c), 0x0000bfd0 },
+	{ _MMIO(0x27a0), 0x0005ffc2 },
+	{ _MMIO(0x27a4), 0x0000bfd0 },
+	{ _MMIO(0x27a8), 0x0005ff82 },
+	{ _MMIO(0x27ac), 0x0000bfd0 },
+};
+
+static const struct i915_oa_reg flex_eu_config_data_port_writes_coalescing[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00000003 },
+	{ _MMIO(0xe658), 0x00002001 },
+	{ _MMIO(0xe758), 0x00778008 },
+	{ _MMIO(0xe45c), 0x00088078 },
+	{ _MMIO(0xe55c), 0x00808708 },
+	{ _MMIO(0xe65c), 0x00a08908 },
+};
+
+static const struct i915_oa_reg mux_config_data_port_writes_coalescing_0_subslices_0x01[] = {
+	{ _MMIO(0x9888), 0x103d0005 },
+	{ _MMIO(0x9888), 0x143d0120 },
+	{ _MMIO(0x9888), 0x163d2400 },
+	{ _MMIO(0x9888), 0x1058022f },
+	{ _MMIO(0x9888), 0x105b0000 },
+	{ _MMIO(0x9888), 0x198b0003 },
+	{ _MMIO(0x9888), 0x005cc000 },
+	{ _MMIO(0x9888), 0x065cc000 },
+	{ _MMIO(0x9888), 0x085cc000 },
+	{ _MMIO(0x9888), 0x0a5cc000 },
+	{ _MMIO(0x9888), 0x0e5cc000 },
+	{ _MMIO(0x9888), 0x025c4000 },
+	{ _MMIO(0x9888), 0x045c8000 },
+	{ _MMIO(0x9888), 0x003d0000 },
+	{ _MMIO(0x9888), 0x063d0094 },
+	{ _MMIO(0x9888), 0x083d0182 },
+	{ _MMIO(0x9888), 0x0a3d1814 },
+	{ _MMIO(0x9888), 0x0e3d0000 },
+	{ _MMIO(0x9888), 0x183d0000 },
+	{ _MMIO(0x9888), 0x1a3d0000 },
+	{ _MMIO(0x9888), 0x0c3d0000 },
+	{ _MMIO(0x9888), 0x0e582242 },
+	{ _MMIO(0x9888), 0x00586700 },
+	{ _MMIO(0x9888), 0x0258004f },
+	{ _MMIO(0x9888), 0x0658c000 },
+	{ _MMIO(0x9888), 0x0858c000 },
+	{ _MMIO(0x9888), 0x0a58c000 },
+	{ _MMIO(0x9888), 0x045b6a80 },
+	{ _MMIO(0x9888), 0x005b4000 },
+	{ _MMIO(0x9888), 0x0e5b4000 },
+	{ _MMIO(0x9888), 0x185b5400 },
+	{ _MMIO(0x9888), 0x1a5b0141 },
+	{ _MMIO(0x9888), 0x025b4000 },
+	{ _MMIO(0x9888), 0x0a5b0000 },
+	{ _MMIO(0x9888), 0x0c5b4000 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1faaa0 },
+	{ _MMIO(0x9888), 0x101f0282 },
+	{ _MMIO(0x9888), 0x00384000 },
+	{ _MMIO(0x9888), 0x0e384000 },
+	{ _MMIO(0x9888), 0x16384000 },
+	{ _MMIO(0x9888), 0x18381415 },
+	{ _MMIO(0x9888), 0x02384000 },
+	{ _MMIO(0x9888), 0x04384000 },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x0c384000 },
+	{ _MMIO(0x9888), 0x0039a000 },
+	{ _MMIO(0x9888), 0x0639a000 },
+	{ _MMIO(0x9888), 0x0839a000 },
+	{ _MMIO(0x9888), 0x0a39a000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x02392000 },
+	{ _MMIO(0x9888), 0x04398000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x198a8000 },
+	{ _MMIO(0x9888), 0x1b8a82a0 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x038b6300 },
+	{ _MMIO(0x9888), 0x058b0062 },
+	{ _MMIO(0x9888), 0x118b0000 },
+	{ _MMIO(0x9888), 0x238b02a0 },
+	{ _MMIO(0x9888), 0x258b1555 },
+	{ _MMIO(0x9888), 0x278b0014 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x21852aaa },
+	{ _MMIO(0x9888), 0x23850028 },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830141 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x0784c000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x1780c000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0xd24), 0x00000000 },
+	{ _MMIO(0x9888), 0x4d801000 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f800001 },
+	{ _MMIO(0x9888), 0x43800000 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800420 },
+	{ _MMIO(0x9888), 0x3f800421 },
+	{ _MMIO(0x9888), 0x41800041 },
+};
+
+static const struct i915_oa_reg *
+get_data_port_writes_coalescing_mux_config(struct drm_i915_private *dev_priv,
+					   int *len)
+{
+	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x01) {
+		*len = ARRAY_SIZE(mux_config_data_port_writes_coalescing_0_subslices_0x01);
+		return mux_config_data_port_writes_coalescing_0_subslices_0x01;
+	}
+
+	*len = 0;
+	return NULL;
+}
+
+static const struct i915_oa_reg b_counter_config_hdc_and_sf[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0x10800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x00800000 },
+	{ _MMIO(0x2770), 0x00000002 },
+	{ _MMIO(0x2774), 0x0000fff7 },
+};
+
+static const struct i915_oa_reg flex_eu_config_hdc_and_sf[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_hdc_and_sf[] = {
+	{ _MMIO(0x9888), 0x105c0232 },
+	{ _MMIO(0x9888), 0x10580232 },
+	{ _MMIO(0x9888), 0x10380232 },
+	{ _MMIO(0x9888), 0x10dc0232 },
+	{ _MMIO(0x9888), 0x10d80232 },
+	{ _MMIO(0x9888), 0x10b80232 },
+	{ _MMIO(0x9888), 0x118e4400 },
+	{ _MMIO(0x9888), 0x025c6080 },
+	{ _MMIO(0x9888), 0x045c004b },
+	{ _MMIO(0x9888), 0x005c8000 },
+	{ _MMIO(0x9888), 0x00582080 },
+	{ _MMIO(0x9888), 0x0258004b },
+	{ _MMIO(0x9888), 0x025b4000 },
+	{ _MMIO(0x9888), 0x045b4000 },
+	{ _MMIO(0x9888), 0x0c1fa000 },
+	{ _MMIO(0x9888), 0x0e1f00aa },
+	{ _MMIO(0x9888), 0x04386080 },
+	{ _MMIO(0x9888), 0x0638404b },
+	{ _MMIO(0x9888), 0x02384000 },
+	{ _MMIO(0x9888), 0x08384000 },
+	{ _MMIO(0x9888), 0x0a380000 },
+	{ _MMIO(0x9888), 0x0c380000 },
+	{ _MMIO(0x9888), 0x00398000 },
+	{ _MMIO(0x9888), 0x0239a000 },
+	{ _MMIO(0x9888), 0x0439a000 },
+	{ _MMIO(0x9888), 0x06392000 },
+	{ _MMIO(0x9888), 0x0cdc25c1 },
+	{ _MMIO(0x9888), 0x0adcc000 },
+	{ _MMIO(0x9888), 0x0ad825c1 },
+	{ _MMIO(0x9888), 0x18db4000 },
+	{ _MMIO(0x9888), 0x1adb0001 },
+	{ _MMIO(0x9888), 0x0e9f8000 },
+	{ _MMIO(0x9888), 0x109f02aa },
+	{ _MMIO(0x9888), 0x0eb825c1 },
+	{ _MMIO(0x9888), 0x18b80154 },
+	{ _MMIO(0x9888), 0x0ab9a000 },
+	{ _MMIO(0x9888), 0x0cb9a000 },
+	{ _MMIO(0x9888), 0x0eb9a000 },
+	{ _MMIO(0x9888), 0x0d88c000 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x258baa05 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x238b2a80 },
+	{ _MMIO(0x9888), 0x198c5400 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x098dc000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x098e05c0 },
+	{ _MMIO(0x9888), 0x058e0000 },
+	{ _MMIO(0x9888), 0x198f0020 },
+	{ _MMIO(0x9888), 0x2185aa0a },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x19835000 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x09848000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x19808000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x51800040 },
+	{ _MMIO(0x9888), 0x43800400 },
+	{ _MMIO(0x9888), 0x45800800 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47800c62 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f801042 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x418014a4 },
+};
+
+static const struct i915_oa_reg *
+get_hdc_and_sf_mux_config(struct drm_i915_private *dev_priv,
+			  int *len)
+{
+	*len = ARRAY_SIZE(mux_config_hdc_and_sf);
+	return mux_config_hdc_and_sf;
+}
+
+static const struct i915_oa_reg b_counter_config_l3_1[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2770), 0x00100070 },
+	{ _MMIO(0x2774), 0x0000fff1 },
+	{ _MMIO(0x2778), 0x00014002 },
+	{ _MMIO(0x277c), 0x0000c3ff },
+	{ _MMIO(0x2780), 0x00010002 },
+	{ _MMIO(0x2784), 0x0000c7ff },
+	{ _MMIO(0x2788), 0x00004002 },
+	{ _MMIO(0x278c), 0x0000d3ff },
+	{ _MMIO(0x2790), 0x00100700 },
+	{ _MMIO(0x2794), 0x0000ff1f },
+	{ _MMIO(0x2798), 0x00001402 },
+	{ _MMIO(0x279c), 0x0000fc3f },
+	{ _MMIO(0x27a0), 0x00001002 },
+	{ _MMIO(0x27a4), 0x0000fc7f },
+	{ _MMIO(0x27a8), 0x00000402 },
+	{ _MMIO(0x27ac), 0x0000fd3f },
+};
+
+static const struct i915_oa_reg flex_eu_config_l3_1[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_l3_1[] = {
+	{ _MMIO(0x9888), 0x10bf03da },
+	{ _MMIO(0x9888), 0x14bf0001 },
+	{ _MMIO(0x9888), 0x12980340 },
+	{ _MMIO(0x9888), 0x12990340 },
+	{ _MMIO(0x9888), 0x0cbf1187 },
+	{ _MMIO(0x9888), 0x0ebf1205 },
+	{ _MMIO(0x9888), 0x00bf0500 },
+	{ _MMIO(0x9888), 0x02bf042b },
+	{ _MMIO(0x9888), 0x04bf002c },
+	{ _MMIO(0x9888), 0x0cdac000 },
+	{ _MMIO(0x9888), 0x0edac000 },
+	{ _MMIO(0x9888), 0x00da8000 },
+	{ _MMIO(0x9888), 0x02dac000 },
+	{ _MMIO(0x9888), 0x04da4000 },
+	{ _MMIO(0x9888), 0x04983400 },
+	{ _MMIO(0x9888), 0x10980000 },
+	{ _MMIO(0x9888), 0x06990034 },
+	{ _MMIO(0x9888), 0x10990000 },
+	{ _MMIO(0x9888), 0x0c9dc000 },
+	{ _MMIO(0x9888), 0x0e9dc000 },
+	{ _MMIO(0x9888), 0x009d8000 },
+	{ _MMIO(0x9888), 0x029dc000 },
+	{ _MMIO(0x9888), 0x049d4000 },
+	{ _MMIO(0x9888), 0x109f02a8 },
+	{ _MMIO(0x9888), 0x0c9fa000 },
+	{ _MMIO(0x9888), 0x0e9f00ba },
+	{ _MMIO(0x9888), 0x0cb88000 },
+	{ _MMIO(0x9888), 0x0cb95000 },
+	{ _MMIO(0x9888), 0x0eb95000 },
+	{ _MMIO(0x9888), 0x00b94000 },
+	{ _MMIO(0x9888), 0x02b95000 },
+	{ _MMIO(0x9888), 0x04b91000 },
+	{ _MMIO(0x9888), 0x06b92000 },
+	{ _MMIO(0x9888), 0x0cba4000 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x07888000 },
+	{ _MMIO(0x9888), 0x09888000 },
+	{ _MMIO(0x9888), 0x0b888000 },
+	{ _MMIO(0x9888), 0x0d880400 },
+	{ _MMIO(0x9888), 0x258b800a },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x238b5500 },
+	{ _MMIO(0x9888), 0x198c4000 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x078c4000 },
+	{ _MMIO(0x9888), 0x098c4000 },
+	{ _MMIO(0x9888), 0x0b8c4000 },
+	{ _MMIO(0x9888), 0x0d8c4000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x018d8000 },
+	{ _MMIO(0x9888), 0x038da000 },
+	{ _MMIO(0x9888), 0x058da000 },
+	{ _MMIO(0x9888), 0x078d2000 },
+	{ _MMIO(0x9888), 0x2185800a },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x1b830154 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x47800000 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x41800060 },
+};
+
+static const struct i915_oa_reg *
+get_l3_1_mux_config(struct drm_i915_private *dev_priv,
+		    int *len)
+{
+	*len = ARRAY_SIZE(mux_config_l3_1);
+	return mux_config_l3_1;
+}
+
+static const struct i915_oa_reg b_counter_config_l3_2[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2770), 0x00100070 },
+	{ _MMIO(0x2774), 0x0000fff1 },
+	{ _MMIO(0x2778), 0x00014002 },
+	{ _MMIO(0x277c), 0x0000c3ff },
+	{ _MMIO(0x2780), 0x00010002 },
+	{ _MMIO(0x2784), 0x0000c7ff },
+	{ _MMIO(0x2788), 0x00004002 },
+	{ _MMIO(0x278c), 0x0000d3ff },
+	{ _MMIO(0x2790), 0x00100700 },
+	{ _MMIO(0x2794), 0x0000ff1f },
+	{ _MMIO(0x2798), 0x00001402 },
+	{ _MMIO(0x279c), 0x0000fc3f },
+	{ _MMIO(0x27a0), 0x00001002 },
+	{ _MMIO(0x27a4), 0x0000fc7f },
+	{ _MMIO(0x27a8), 0x00000402 },
+	{ _MMIO(0x27ac), 0x0000fd3f },
+};
+
+static const struct i915_oa_reg flex_eu_config_l3_2[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_l3_2[] = {
+	{ _MMIO(0x9888), 0x103f03da },
+	{ _MMIO(0x9888), 0x143f0001 },
+	{ _MMIO(0x9888), 0x12180340 },
+	{ _MMIO(0x9888), 0x12190340 },
+	{ _MMIO(0x9888), 0x0c3f1187 },
+	{ _MMIO(0x9888), 0x0e3f1205 },
+	{ _MMIO(0x9888), 0x003f0500 },
+	{ _MMIO(0x9888), 0x023f042b },
+	{ _MMIO(0x9888), 0x043f002c },
+	{ _MMIO(0x9888), 0x0c5ac000 },
+	{ _MMIO(0x9888), 0x0e5ac000 },
+	{ _MMIO(0x9888), 0x005a8000 },
+	{ _MMIO(0x9888), 0x025ac000 },
+	{ _MMIO(0x9888), 0x045a4000 },
+	{ _MMIO(0x9888), 0x04183400 },
+	{ _MMIO(0x9888), 0x10180000 },
+	{ _MMIO(0x9888), 0x06190034 },
+	{ _MMIO(0x9888), 0x10190000 },
+	{ _MMIO(0x9888), 0x0c1dc000 },
+	{ _MMIO(0x9888), 0x0e1dc000 },
+	{ _MMIO(0x9888), 0x001d8000 },
+	{ _MMIO(0x9888), 0x021dc000 },
+	{ _MMIO(0x9888), 0x041d4000 },
+	{ _MMIO(0x9888), 0x101f02a8 },
+	{ _MMIO(0x9888), 0x0c1fa000 },
+	{ _MMIO(0x9888), 0x0e1f00ba },
+	{ _MMIO(0x9888), 0x0c388000 },
+	{ _MMIO(0x9888), 0x0c395000 },
+	{ _MMIO(0x9888), 0x0e395000 },
+	{ _MMIO(0x9888), 0x00394000 },
+	{ _MMIO(0x9888), 0x02395000 },
+	{ _MMIO(0x9888), 0x04391000 },
+	{ _MMIO(0x9888), 0x06392000 },
+	{ _MMIO(0x9888), 0x0c3a4000 },
+	{ _MMIO(0x9888), 0x1b8aa800 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x258b4005 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x238b2a80 },
+	{ _MMIO(0x9888), 0x2185800a },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x1b830154 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x45800000 },
+	{ _MMIO(0x9888), 0x47800000 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f800000 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x41800060 },
+};
+
+static const struct i915_oa_reg *
+get_l3_2_mux_config(struct drm_i915_private *dev_priv,
+		    int *len)
+{
+	*len = ARRAY_SIZE(mux_config_l3_2);
+	return mux_config_l3_2;
+}
+
+static const struct i915_oa_reg b_counter_config_l3_3[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2770), 0x00100070 },
+	{ _MMIO(0x2774), 0x0000fff1 },
+	{ _MMIO(0x2778), 0x00014002 },
+	{ _MMIO(0x277c), 0x0000c3ff },
+	{ _MMIO(0x2780), 0x00010002 },
+	{ _MMIO(0x2784), 0x0000c7ff },
+	{ _MMIO(0x2788), 0x00004002 },
+	{ _MMIO(0x278c), 0x0000d3ff },
+	{ _MMIO(0x2790), 0x00100700 },
+	{ _MMIO(0x2794), 0x0000ff1f },
+	{ _MMIO(0x2798), 0x00001402 },
+	{ _MMIO(0x279c), 0x0000fc3f },
+	{ _MMIO(0x27a0), 0x00001002 },
+	{ _MMIO(0x27a4), 0x0000fc7f },
+	{ _MMIO(0x27a8), 0x00000402 },
+	{ _MMIO(0x27ac), 0x0000fd3f },
+};
+
+static const struct i915_oa_reg flex_eu_config_l3_3[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_l3_3[] = {
+	{ _MMIO(0x9888), 0x121b0340 },
+	{ _MMIO(0x9888), 0x103f0274 },
+	{ _MMIO(0x9888), 0x123f0000 },
+	{ _MMIO(0x9888), 0x129b0340 },
+	{ _MMIO(0x9888), 0x10bf0274 },
+	{ _MMIO(0x9888), 0x12bf0000 },
+	{ _MMIO(0x9888), 0x041b3400 },
+	{ _MMIO(0x9888), 0x101b0000 },
+	{ _MMIO(0x9888), 0x045c8000 },
+	{ _MMIO(0x9888), 0x0a3d4000 },
+	{ _MMIO(0x9888), 0x003f0080 },
+	{ _MMIO(0x9888), 0x023f0793 },
+	{ _MMIO(0x9888), 0x043f0014 },
+	{ _MMIO(0x9888), 0x04588000 },
+	{ _MMIO(0x9888), 0x005a8000 },
+	{ _MMIO(0x9888), 0x025ac000 },
+	{ _MMIO(0x9888), 0x045a4000 },
+	{ _MMIO(0x9888), 0x0a5b4000 },
+	{ _MMIO(0x9888), 0x001d8000 },
+	{ _MMIO(0x9888), 0x021dc000 },
+	{ _MMIO(0x9888), 0x041d4000 },
+	{ _MMIO(0x9888), 0x0c1fa000 },
+	{ _MMIO(0x9888), 0x0e1f002a },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x00394000 },
+	{ _MMIO(0x9888), 0x02395000 },
+	{ _MMIO(0x9888), 0x04399000 },
+	{ _MMIO(0x9888), 0x069b0034 },
+	{ _MMIO(0x9888), 0x109b0000 },
+	{ _MMIO(0x9888), 0x06dc4000 },
+	{ _MMIO(0x9888), 0x0cbd4000 },
+	{ _MMIO(0x9888), 0x0cbf0981 },
+	{ _MMIO(0x9888), 0x0ebf0a0f },
+	{ _MMIO(0x9888), 0x06d84000 },
+	{ _MMIO(0x9888), 0x0cdac000 },
+	{ _MMIO(0x9888), 0x0edac000 },
+	{ _MMIO(0x9888), 0x0cdb4000 },
+	{ _MMIO(0x9888), 0x0c9dc000 },
+	{ _MMIO(0x9888), 0x0e9dc000 },
+	{ _MMIO(0x9888), 0x109f02a8 },
+	{ _MMIO(0x9888), 0x0e9f0080 },
+	{ _MMIO(0x9888), 0x0cb84000 },
+	{ _MMIO(0x9888), 0x0cb95000 },
+	{ _MMIO(0x9888), 0x0eb95000 },
+	{ _MMIO(0x9888), 0x06b92000 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x0d880400 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x258b8009 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x238b2a80 },
+	{ _MMIO(0x9888), 0x198c4000 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x0d8c4000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x078d2000 },
+	{ _MMIO(0x9888), 0x2185800a },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x1b830154 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x45800c00 },
+	{ _MMIO(0x9888), 0x47800c63 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f8014a5 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x41800045 },
+};
+
+static const struct i915_oa_reg *
+get_l3_3_mux_config(struct drm_i915_private *dev_priv,
+		    int *len)
+{
+	*len = ARRAY_SIZE(mux_config_l3_3);
+	return mux_config_l3_3;
+}
+
+static const struct i915_oa_reg b_counter_config_l3_4[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0xf0800000 },
+	{ _MMIO(0x2770), 0x00100070 },
+	{ _MMIO(0x2774), 0x0000fff1 },
+	{ _MMIO(0x2778), 0x00014002 },
+	{ _MMIO(0x277c), 0x0000c3ff },
+	{ _MMIO(0x2780), 0x00010002 },
+	{ _MMIO(0x2784), 0x0000c7ff },
+	{ _MMIO(0x2788), 0x00004002 },
+	{ _MMIO(0x278c), 0x0000d3ff },
+	{ _MMIO(0x2790), 0x00100700 },
+	{ _MMIO(0x2794), 0x0000ff1f },
+	{ _MMIO(0x2798), 0x00001402 },
+	{ _MMIO(0x279c), 0x0000fc3f },
+	{ _MMIO(0x27a0), 0x00001002 },
+	{ _MMIO(0x27a4), 0x0000fc7f },
+	{ _MMIO(0x27a8), 0x00000402 },
+	{ _MMIO(0x27ac), 0x0000fd3f },
+};
+
+static const struct i915_oa_reg flex_eu_config_l3_4[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_l3_4[] = {
+	{ _MMIO(0x9888), 0x121a0340 },
+	{ _MMIO(0x9888), 0x103f0017 },
+	{ _MMIO(0x9888), 0x123f0020 },
+	{ _MMIO(0x9888), 0x129a0340 },
+	{ _MMIO(0x9888), 0x10bf0017 },
+	{ _MMIO(0x9888), 0x12bf0020 },
+	{ _MMIO(0x9888), 0x041a3400 },
+	{ _MMIO(0x9888), 0x101a0000 },
+	{ _MMIO(0x9888), 0x043b8000 },
+	{ _MMIO(0x9888), 0x0a3e0010 },
+	{ _MMIO(0x9888), 0x003f0200 },
+	{ _MMIO(0x9888), 0x023f0113 },
+	{ _MMIO(0x9888), 0x043f0014 },
+	{ _MMIO(0x9888), 0x02592000 },
+	{ _MMIO(0x9888), 0x005a8000 },
+	{ _MMIO(0x9888), 0x025ac000 },
+	{ _MMIO(0x9888), 0x045a4000 },
+	{ _MMIO(0x9888), 0x0a1c8000 },
+	{ _MMIO(0x9888), 0x001d8000 },
+	{ _MMIO(0x9888), 0x021dc000 },
+	{ _MMIO(0x9888), 0x041d4000 },
+	{ _MMIO(0x9888), 0x0a1e8000 },
+	{ _MMIO(0x9888), 0x0c1fa000 },
+	{ _MMIO(0x9888), 0x0e1f001a },
+	{ _MMIO(0x9888), 0x00394000 },
+	{ _MMIO(0x9888), 0x02395000 },
+	{ _MMIO(0x9888), 0x04391000 },
+	{ _MMIO(0x9888), 0x069a0034 },
+	{ _MMIO(0x9888), 0x109a0000 },
+	{ _MMIO(0x9888), 0x06bb4000 },
+	{ _MMIO(0x9888), 0x0abe0040 },
+	{ _MMIO(0x9888), 0x0cbf0984 },
+	{ _MMIO(0x9888), 0x0ebf0a02 },
+	{ _MMIO(0x9888), 0x02d94000 },
+	{ _MMIO(0x9888), 0x0cdac000 },
+	{ _MMIO(0x9888), 0x0edac000 },
+	{ _MMIO(0x9888), 0x0c9c0400 },
+	{ _MMIO(0x9888), 0x0c9dc000 },
+	{ _MMIO(0x9888), 0x0e9dc000 },
+	{ _MMIO(0x9888), 0x0c9e0400 },
+	{ _MMIO(0x9888), 0x109f02a8 },
+	{ _MMIO(0x9888), 0x0e9f0040 },
+	{ _MMIO(0x9888), 0x0cb95000 },
+	{ _MMIO(0x9888), 0x0eb95000 },
+	{ _MMIO(0x9888), 0x0f88000f },
+	{ _MMIO(0x9888), 0x0d880400 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x258b8009 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x238b2a80 },
+	{ _MMIO(0x9888), 0x198c4000 },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x0d8c4000 },
+	{ _MMIO(0x9888), 0x0d8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x078d2000 },
+	{ _MMIO(0x9888), 0x2185800a },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x1b830154 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x45800800 },
+	{ _MMIO(0x9888), 0x47800842 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f801084 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x41800044 },
+};
+
+static const struct i915_oa_reg *
+get_l3_4_mux_config(struct drm_i915_private *dev_priv,
+		    int *len)
+{
+	*len = ARRAY_SIZE(mux_config_l3_4);
+	return mux_config_l3_4;
+}
+
+static const struct i915_oa_reg b_counter_config_rasterizer_and_pixel_backend[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x30800000 },
+	{ _MMIO(0x2770), 0x00006000 },
+	{ _MMIO(0x2774), 0x0000f3ff },
+	{ _MMIO(0x2778), 0x00001800 },
+	{ _MMIO(0x277c), 0x0000fcff },
+	{ _MMIO(0x2780), 0x00000600 },
+	{ _MMIO(0x2784), 0x0000ff3f },
+	{ _MMIO(0x2788), 0x00000180 },
+	{ _MMIO(0x278c), 0x0000ffcf },
+	{ _MMIO(0x2790), 0x00000060 },
+	{ _MMIO(0x2794), 0x0000fff3 },
+	{ _MMIO(0x2798), 0x00000018 },
+	{ _MMIO(0x279c), 0x0000fffc },
+};
+
+static const struct i915_oa_reg flex_eu_config_rasterizer_and_pixel_backend[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_rasterizer_and_pixel_backend[] = {
+	{ _MMIO(0x9888), 0x143b000e },
+	{ _MMIO(0x9888), 0x043c55c0 },
+	{ _MMIO(0x9888), 0x0a1e0280 },
+	{ _MMIO(0x9888), 0x0c1e0408 },
+	{ _MMIO(0x9888), 0x10390000 },
+	{ _MMIO(0x9888), 0x12397a1f },
+	{ _MMIO(0x9888), 0x14bb000e },
+	{ _MMIO(0x9888), 0x04bc5000 },
+	{ _MMIO(0x9888), 0x0a9e0296 },
+	{ _MMIO(0x9888), 0x0c9e0008 },
+	{ _MMIO(0x9888), 0x10b90000 },
+	{ _MMIO(0x9888), 0x12b97a1f },
+	{ _MMIO(0x9888), 0x063b0042 },
+	{ _MMIO(0x9888), 0x103b0000 },
+	{ _MMIO(0x9888), 0x083c0000 },
+	{ _MMIO(0x9888), 0x0a3e0040 },
+	{ _MMIO(0x9888), 0x043f8000 },
+	{ _MMIO(0x9888), 0x02594000 },
+	{ _MMIO(0x9888), 0x045a8000 },
+	{ _MMIO(0x9888), 0x0c1c0400 },
+	{ _MMIO(0x9888), 0x041d8000 },
+	{ _MMIO(0x9888), 0x081e02c0 },
+	{ _MMIO(0x9888), 0x0e1e0000 },
+	{ _MMIO(0x9888), 0x0c1fa800 },
+	{ _MMIO(0x9888), 0x0e1f0260 },
+	{ _MMIO(0x9888), 0x101f0014 },
+	{ _MMIO(0x9888), 0x003905e0 },
+	{ _MMIO(0x9888), 0x06390bc0 },
+	{ _MMIO(0x9888), 0x02390018 },
+	{ _MMIO(0x9888), 0x04394000 },
+	{ _MMIO(0x9888), 0x04bb0042 },
+	{ _MMIO(0x9888), 0x10bb0000 },
+	{ _MMIO(0x9888), 0x02bc05c0 },
+	{ _MMIO(0x9888), 0x08bc0000 },
+	{ _MMIO(0x9888), 0x0abe0004 },
+	{ _MMIO(0x9888), 0x02bf8000 },
+	{ _MMIO(0x9888), 0x02d91000 },
+	{ _MMIO(0x9888), 0x02da8000 },
+	{ _MMIO(0x9888), 0x089c8000 },
+	{ _MMIO(0x9888), 0x029d8000 },
+	{ _MMIO(0x9888), 0x089e8000 },
+	{ _MMIO(0x9888), 0x0e9e0000 },
+	{ _MMIO(0x9888), 0x0e9fa806 },
+	{ _MMIO(0x9888), 0x109f0142 },
+	{ _MMIO(0x9888), 0x08b90617 },
+	{ _MMIO(0x9888), 0x0ab90be0 },
+	{ _MMIO(0x9888), 0x02b94000 },
+	{ _MMIO(0x9888), 0x0d88f000 },
+	{ _MMIO(0x9888), 0x0f88000c },
+	{ _MMIO(0x9888), 0x07888000 },
+	{ _MMIO(0x9888), 0x09888000 },
+	{ _MMIO(0x9888), 0x018a8000 },
+	{ _MMIO(0x9888), 0x0f8a8000 },
+	{ _MMIO(0x9888), 0x1b8a2800 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x238b52a0 },
+	{ _MMIO(0x9888), 0x258b6a95 },
+	{ _MMIO(0x9888), 0x278b0029 },
+	{ _MMIO(0x9888), 0x178c2000 },
+	{ _MMIO(0x9888), 0x198c1500 },
+	{ _MMIO(0x9888), 0x1b8c0014 },
+	{ _MMIO(0x9888), 0x078c4000 },
+	{ _MMIO(0x9888), 0x098c4000 },
+	{ _MMIO(0x9888), 0x098da000 },
+	{ _MMIO(0x9888), 0x0b8da000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x038d8000 },
+	{ _MMIO(0x9888), 0x058d2000 },
+	{ _MMIO(0x9888), 0x1f85aa80 },
+	{ _MMIO(0x9888), 0x2185aaaa },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x01834000 },
+	{ _MMIO(0x9888), 0x0f834000 },
+	{ _MMIO(0x9888), 0x19835400 },
+	{ _MMIO(0x9888), 0x1b830155 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0184c000 },
+	{ _MMIO(0x9888), 0x0784c000 },
+	{ _MMIO(0x9888), 0x0984c000 },
+	{ _MMIO(0x9888), 0x0b84c000 },
+	{ _MMIO(0x9888), 0x0d84c000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x1180c000 },
+	{ _MMIO(0x9888), 0x1780c000 },
+	{ _MMIO(0x9888), 0x1980c000 },
+	{ _MMIO(0x9888), 0x1b80c000 },
+	{ _MMIO(0x9888), 0x1d80c000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x4d800444 },
+	{ _MMIO(0x9888), 0x3d800000 },
+	{ _MMIO(0x9888), 0x4f804000 },
+	{ _MMIO(0x9888), 0x43801080 },
+	{ _MMIO(0x9888), 0x51800000 },
+	{ _MMIO(0x9888), 0x45800084 },
+	{ _MMIO(0x9888), 0x53800044 },
+	{ _MMIO(0x9888), 0x47801080 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x3f800000 },
+	{ _MMIO(0x9888), 0x41800840 },
+};
+
+static const struct i915_oa_reg *
+get_rasterizer_and_pixel_backend_mux_config(struct drm_i915_private *dev_priv,
+					    int *len)
+{
+	*len = ARRAY_SIZE(mux_config_rasterizer_and_pixel_backend);
+	return mux_config_rasterizer_and_pixel_backend;
+}
+
+static const struct i915_oa_reg b_counter_config_sampler_1[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0x70800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x00800000 },
+	{ _MMIO(0x2770), 0x0000c000 },
+	{ _MMIO(0x2774), 0x0000e7ff },
+	{ _MMIO(0x2778), 0x00003000 },
+	{ _MMIO(0x277c), 0x0000f9ff },
+	{ _MMIO(0x2780), 0x00000c00 },
+	{ _MMIO(0x2784), 0x0000fe7f },
+};
+
+static const struct i915_oa_reg flex_eu_config_sampler_1[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_sampler_1[] = {
+	{ _MMIO(0x9888), 0x18921400 },
+	{ _MMIO(0x9888), 0x149500ab },
+	{ _MMIO(0x9888), 0x18b21400 },
+	{ _MMIO(0x9888), 0x14b500ab },
+	{ _MMIO(0x9888), 0x18d21400 },
+	{ _MMIO(0x9888), 0x14d500ab },
+	{ _MMIO(0x9888), 0x0cdc8000 },
+	{ _MMIO(0x9888), 0x0edc4000 },
+	{ _MMIO(0x9888), 0x02dcc000 },
+	{ _MMIO(0x9888), 0x04dcc000 },
+	{ _MMIO(0x9888), 0x1abd00a0 },
+	{ _MMIO(0x9888), 0x0abd8000 },
+	{ _MMIO(0x9888), 0x0cd88000 },
+	{ _MMIO(0x9888), 0x0ed84000 },
+	{ _MMIO(0x9888), 0x04d88000 },
+	{ _MMIO(0x9888), 0x1adb0050 },
+	{ _MMIO(0x9888), 0x04db8000 },
+	{ _MMIO(0x9888), 0x06db8000 },
+	{ _MMIO(0x9888), 0x08db8000 },
+	{ _MMIO(0x9888), 0x0adb4000 },
+	{ _MMIO(0x9888), 0x109f02a0 },
+	{ _MMIO(0x9888), 0x0c9fa000 },
+	{ _MMIO(0x9888), 0x0e9f00aa },
+	{ _MMIO(0x9888), 0x18b82500 },
+	{ _MMIO(0x9888), 0x02b88000 },
+	{ _MMIO(0x9888), 0x04b84000 },
+	{ _MMIO(0x9888), 0x06b84000 },
+	{ _MMIO(0x9888), 0x08b84000 },
+	{ _MMIO(0x9888), 0x0ab84000 },
+	{ _MMIO(0x9888), 0x0cb88000 },
+	{ _MMIO(0x9888), 0x0cb98000 },
+	{ _MMIO(0x9888), 0x0eb9a000 },
+	{ _MMIO(0x9888), 0x00b98000 },
+	{ _MMIO(0x9888), 0x02b9a000 },
+	{ _MMIO(0x9888), 0x04b9a000 },
+	{ _MMIO(0x9888), 0x06b92000 },
+	{ _MMIO(0x9888), 0x1aba0200 },
+	{ _MMIO(0x9888), 0x02ba8000 },
+	{ _MMIO(0x9888), 0x0cba8000 },
+	{ _MMIO(0x9888), 0x04908000 },
+	{ _MMIO(0x9888), 0x04918000 },
+	{ _MMIO(0x9888), 0x04927300 },
+	{ _MMIO(0x9888), 0x10920000 },
+	{ _MMIO(0x9888), 0x1893000a },
+	{ _MMIO(0x9888), 0x0a934000 },
+	{ _MMIO(0x9888), 0x0a946000 },
+	{ _MMIO(0x9888), 0x0c959000 },
+	{ _MMIO(0x9888), 0x0e950098 },
+	{ _MMIO(0x9888), 0x10950000 },
+	{ _MMIO(0x9888), 0x04b04000 },
+	{ _MMIO(0x9888), 0x04b14000 },
+	{ _MMIO(0x9888), 0x04b20073 },
+	{ _MMIO(0x9888), 0x10b20000 },
+	{ _MMIO(0x9888), 0x04b38000 },
+	{ _MMIO(0x9888), 0x06b38000 },
+	{ _MMIO(0x9888), 0x08b34000 },
+	{ _MMIO(0x9888), 0x04b4c000 },
+	{ _MMIO(0x9888), 0x02b59890 },
+	{ _MMIO(0x9888), 0x10b50000 },
+	{ _MMIO(0x9888), 0x06d04000 },
+	{ _MMIO(0x9888), 0x06d14000 },
+	{ _MMIO(0x9888), 0x06d20073 },
+	{ _MMIO(0x9888), 0x10d20000 },
+	{ _MMIO(0x9888), 0x18d30020 },
+	{ _MMIO(0x9888), 0x02d38000 },
+	{ _MMIO(0x9888), 0x0cd34000 },
+	{ _MMIO(0x9888), 0x0ad48000 },
+	{ _MMIO(0x9888), 0x04d42000 },
+	{ _MMIO(0x9888), 0x0ed59000 },
+	{ _MMIO(0x9888), 0x00d59800 },
+	{ _MMIO(0x9888), 0x10d50000 },
+	{ _MMIO(0x9888), 0x0f88000e },
+	{ _MMIO(0x9888), 0x03888000 },
+	{ _MMIO(0x9888), 0x05888000 },
+	{ _MMIO(0x9888), 0x07888000 },
+	{ _MMIO(0x9888), 0x09888000 },
+	{ _MMIO(0x9888), 0x0b888000 },
+	{ _MMIO(0x9888), 0x0d880400 },
+	{ _MMIO(0x9888), 0x278b002a },
+	{ _MMIO(0x9888), 0x238b5500 },
+	{ _MMIO(0x9888), 0x258b000a },
+	{ _MMIO(0x9888), 0x1b8c0015 },
+	{ _MMIO(0x9888), 0x038c4000 },
+	{ _MMIO(0x9888), 0x058c4000 },
+	{ _MMIO(0x9888), 0x078c4000 },
+	{ _MMIO(0x9888), 0x098c4000 },
+	{ _MMIO(0x9888), 0x0b8c4000 },
+	{ _MMIO(0x9888), 0x0d8c4000 },
+	{ _MMIO(0x9888), 0x0d8d8000 },
+	{ _MMIO(0x9888), 0x0f8da000 },
+	{ _MMIO(0x9888), 0x018d8000 },
+	{ _MMIO(0x9888), 0x038da000 },
+	{ _MMIO(0x9888), 0x058da000 },
+	{ _MMIO(0x9888), 0x078d2000 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x2185000a },
+	{ _MMIO(0x9888), 0x1b830150 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0d848000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x1d808000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47801021 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f800c64 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x41800c02 },
+};
+
+static const struct i915_oa_reg *
+get_sampler_1_mux_config(struct drm_i915_private *dev_priv,
+			 int *len)
+{
+	*len = ARRAY_SIZE(mux_config_sampler_1);
+	return mux_config_sampler_1;
+}
+
+static const struct i915_oa_reg b_counter_config_sampler_2[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0x70800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x00800000 },
+	{ _MMIO(0x2770), 0x0000c000 },
+	{ _MMIO(0x2774), 0x0000e7ff },
+	{ _MMIO(0x2778), 0x00003000 },
+	{ _MMIO(0x277c), 0x0000f9ff },
+	{ _MMIO(0x2780), 0x00000c00 },
+	{ _MMIO(0x2784), 0x0000fe7f },
+};
+
+static const struct i915_oa_reg flex_eu_config_sampler_2[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_sampler_2[] = {
+	{ _MMIO(0x9888), 0x18121400 },
+	{ _MMIO(0x9888), 0x141500ab },
+	{ _MMIO(0x9888), 0x18321400 },
+	{ _MMIO(0x9888), 0x143500ab },
+	{ _MMIO(0x9888), 0x18521400 },
+	{ _MMIO(0x9888), 0x145500ab },
+	{ _MMIO(0x9888), 0x0c5c8000 },
+	{ _MMIO(0x9888), 0x0e5c4000 },
+	{ _MMIO(0x9888), 0x025cc000 },
+	{ _MMIO(0x9888), 0x045cc000 },
+	{ _MMIO(0x9888), 0x1a3d00a0 },
+	{ _MMIO(0x9888), 0x0a3d8000 },
+	{ _MMIO(0x9888), 0x0c588000 },
+	{ _MMIO(0x9888), 0x0e584000 },
+	{ _MMIO(0x9888), 0x04588000 },
+	{ _MMIO(0x9888), 0x1a5b0050 },
+	{ _MMIO(0x9888), 0x045b8000 },
+	{ _MMIO(0x9888), 0x065b8000 },
+	{ _MMIO(0x9888), 0x085b8000 },
+	{ _MMIO(0x9888), 0x0a5b4000 },
+	{ _MMIO(0x9888), 0x101f02a0 },
+	{ _MMIO(0x9888), 0x0c1fa000 },
+	{ _MMIO(0x9888), 0x0e1f00aa },
+	{ _MMIO(0x9888), 0x18382500 },
+	{ _MMIO(0x9888), 0x02388000 },
+	{ _MMIO(0x9888), 0x04384000 },
+	{ _MMIO(0x9888), 0x06384000 },
+	{ _MMIO(0x9888), 0x08384000 },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x0c388000 },
+	{ _MMIO(0x9888), 0x0c398000 },
+	{ _MMIO(0x9888), 0x0e39a000 },
+	{ _MMIO(0x9888), 0x00398000 },
+	{ _MMIO(0x9888), 0x0239a000 },
+	{ _MMIO(0x9888), 0x0439a000 },
+	{ _MMIO(0x9888), 0x06392000 },
+	{ _MMIO(0x9888), 0x1a3a0200 },
+	{ _MMIO(0x9888), 0x023a8000 },
+	{ _MMIO(0x9888), 0x0c3a8000 },
+	{ _MMIO(0x9888), 0x04108000 },
+	{ _MMIO(0x9888), 0x04118000 },
+	{ _MMIO(0x9888), 0x04127300 },
+	{ _MMIO(0x9888), 0x10120000 },
+	{ _MMIO(0x9888), 0x1813000a },
+	{ _MMIO(0x9888), 0x0a134000 },
+	{ _MMIO(0x9888), 0x0a146000 },
+	{ _MMIO(0x9888), 0x0c159000 },
+	{ _MMIO(0x9888), 0x0e150098 },
+	{ _MMIO(0x9888), 0x10150000 },
+	{ _MMIO(0x9888), 0x04304000 },
+	{ _MMIO(0x9888), 0x04314000 },
+	{ _MMIO(0x9888), 0x04320073 },
+	{ _MMIO(0x9888), 0x10320000 },
+	{ _MMIO(0x9888), 0x04338000 },
+	{ _MMIO(0x9888), 0x06338000 },
+	{ _MMIO(0x9888), 0x08334000 },
+	{ _MMIO(0x9888), 0x0434c000 },
+	{ _MMIO(0x9888), 0x02359890 },
+	{ _MMIO(0x9888), 0x10350000 },
+	{ _MMIO(0x9888), 0x06504000 },
+	{ _MMIO(0x9888), 0x06514000 },
+	{ _MMIO(0x9888), 0x06520073 },
+	{ _MMIO(0x9888), 0x10520000 },
+	{ _MMIO(0x9888), 0x18530020 },
+	{ _MMIO(0x9888), 0x02538000 },
+	{ _MMIO(0x9888), 0x0c534000 },
+	{ _MMIO(0x9888), 0x0a548000 },
+	{ _MMIO(0x9888), 0x04542000 },
+	{ _MMIO(0x9888), 0x0e559000 },
+	{ _MMIO(0x9888), 0x00559800 },
+	{ _MMIO(0x9888), 0x10550000 },
+	{ _MMIO(0x9888), 0x1b8aa000 },
+	{ _MMIO(0x9888), 0x1d8a0002 },
+	{ _MMIO(0x9888), 0x038a8000 },
+	{ _MMIO(0x9888), 0x058a8000 },
+	{ _MMIO(0x9888), 0x078a8000 },
+	{ _MMIO(0x9888), 0x098a8000 },
+	{ _MMIO(0x9888), 0x0b8a8000 },
+	{ _MMIO(0x9888), 0x0d8a8000 },
+	{ _MMIO(0x9888), 0x278b0015 },
+	{ _MMIO(0x9888), 0x238b2a80 },
+	{ _MMIO(0x9888), 0x258b0005 },
+	{ _MMIO(0x9888), 0x2385002a },
+	{ _MMIO(0x9888), 0x1f85aa00 },
+	{ _MMIO(0x9888), 0x2185000a },
+	{ _MMIO(0x9888), 0x1b830150 },
+	{ _MMIO(0x9888), 0x03834000 },
+	{ _MMIO(0x9888), 0x05834000 },
+	{ _MMIO(0x9888), 0x07834000 },
+	{ _MMIO(0x9888), 0x09834000 },
+	{ _MMIO(0x9888), 0x0b834000 },
+	{ _MMIO(0x9888), 0x0d834000 },
+	{ _MMIO(0x9888), 0x0d848000 },
+	{ _MMIO(0x9888), 0x0f84c000 },
+	{ _MMIO(0x9888), 0x01848000 },
+	{ _MMIO(0x9888), 0x0384c000 },
+	{ _MMIO(0x9888), 0x0584c000 },
+	{ _MMIO(0x9888), 0x07844000 },
+	{ _MMIO(0x9888), 0x1d808000 },
+	{ _MMIO(0x9888), 0x1f80c000 },
+	{ _MMIO(0x9888), 0x11808000 },
+	{ _MMIO(0x9888), 0x1380c000 },
+	{ _MMIO(0x9888), 0x1580c000 },
+	{ _MMIO(0x9888), 0x17804000 },
+	{ _MMIO(0x9888), 0x53800000 },
+	{ _MMIO(0x9888), 0x47801021 },
+	{ _MMIO(0x9888), 0x21800000 },
+	{ _MMIO(0x9888), 0x31800000 },
+	{ _MMIO(0x9888), 0x4d800000 },
+	{ _MMIO(0x9888), 0x3f800c64 },
+	{ _MMIO(0x9888), 0x4f800000 },
+	{ _MMIO(0x9888), 0x41800c02 },
+};
+
+static const struct i915_oa_reg *
+get_sampler_2_mux_config(struct drm_i915_private *dev_priv,
+			 int *len)
+{
+	*len = ARRAY_SIZE(mux_config_sampler_2);
+	return mux_config_sampler_2;
+}
+
+static const struct i915_oa_reg b_counter_config_tdl_1[] = {
+	{ _MMIO(0x2740), 0x00000000 },
+	{ _MMIO(0x2744), 0x00800000 },
+	{ _MMIO(0x2710), 0x00000000 },
+	{ _MMIO(0x2714), 0xf0800000 },
+	{ _MMIO(0x2720), 0x00000000 },
+	{ _MMIO(0x2724), 0x30800000 },
+	{ _MMIO(0x2770), 0x00000002 },
+	{ _MMIO(0x2774), 0x0000fdff },
+	{ _MMIO(0x2778), 0x00000000 },
+	{ _MMIO(0x277c), 0x0000fe7f },
+	{ _MMIO(0x2780), 0x00000002 },
+	{ _MMIO(0x2784), 0x0000ffbf },
+	{ _MMIO(0x2788), 0x00000000 },
+	{ _MMIO(0x278c), 0x0000ffcf },
+	{ _MMIO(0x2790), 0x00000002 },
+	{ _MMIO(0x2794), 0x0000fff7 },
+	{ _MMIO(0x2798), 0x00000000 },
+	{ _MMIO(0x279c), 0x0000fff9 },
+};
+
+static const struct i915_oa_reg flex_eu_config_tdl_1[] = {
+	{ _MMIO(0xe458), 0x00005004 },
+	{ _MMIO(0xe558), 0x00010003 },
+	{ _MMIO(0xe658), 0x00012011 },
+	{ _MMIO(0xe758), 0x00015014 },
+	{ _MMIO(0xe45c), 0x00051050 },
+	{ _MMIO(0xe55c), 0x00053052 },
+	{ _MMIO(0xe65c), 0x00055054 },
+};
+
+static const struct i915_oa_reg mux_config_tdl_1[] = {
+	{ _MMIO(0x9888), 0x16154d60 },
+	{ _MMIO(0x9888), 0x16352e60 },
+	{ _MMIO(0x9888), 0x16554d60 },
+	{ _MMIO(0x9888), 0x16950000 },
+	{ _MMIO(0x9888), 0x16b50000 },
+	{ _MMIO(0x9888), 0x16d50000 },
+	{ _MMIO(0x9888), 0x005c8000 },
+	{ _MMIO(0x9888), 0x045cc000 },
+	{ _MMIO(0x9888), 0x065c4000 },
+	{ _MMIO(0x9888), 0x083d8000 },
+	{ _MMIO(0x9888), 0x0a3d8000 },
+	{ _MMIO(0x9888), 0x0458c000 },
+	{ _MMIO(0x9888), 0x025b8000 },
+	{ _MMIO(0x9888), 0x085b4000 },
+	{ _MMIO(0x9888), 0x0a5b4000 },
+	{ _MMIO(0x9888), 0x0c5b8000 },
+	{ _MMIO(0x9888), 0x0c1fa000 },
+	{ _MMIO(0x9888), 0x0e1f00aa },
+	{ _MMIO(0x9888), 0x02384000 },
+	{ _MMIO(0x9888), 0x04388000 },
+	{ _MMIO(0x9888), 0x06388000 },
+	{ _MMIO(0x9888), 0x08384000 },
+	{ _MMIO(0x9888), 0x0a384000 },
+	{ _MMIO(0x9888), 0x0c384000 },
+	{ _MMIO(0x9888), 0x00398000 },
+	{ _MMIO(0x9888), 0x0239a000 },
+	{ _MMIO(0x9888), 0x0439a000 },
+	{ _MMIO(0x9888), 0x06392000 },
+	{ _MMIO(0x9888), 0x043a8000 },
+	{ _MMIO(0x9888), 0x063a8000 },
+	{ _MMIO(0x9888), 0x08138000 },
+	{ _MMIO(0x9888), 0x0a138000 },
+	{ _MMIO(0x9888), 0x06143000 },
+	{ _MMIO(0x9888), 0x0415cfc7 },
+	{ _MMIO(0x9888), 0x10150000 },
+	{ _MMIO(0x9888), 0x02338000 },
+	{ _MMIO(0x9888), 0x0c338000 },
+	{ _MMIO(0x9888), 0x04342000 },
+	{ _MMIO(0x9888), 0x06344000 },
+	{ _MMIO(0x9888), 0x0035c700 },
+	{ _MMIO(0x9888), 0x063500cf },
+	{ _MMIO(0x9888), 0x10350000 },
+	{ _MMIO(0x9888), 0x04538000 },
+	{ _MMIO(0x9888), 0x06538000 },
+	{ _MMIO(0x9888), 0x0454c000 },
+	{ _MMIO(0x9888), 0x0255cfc7 },
+	{ _MMIO(0x9888), 0x10550000 },
+	{ _MMIO(0x9888), 0x06dc8000 },
+	{ _MMIO(0x9888), 0x08dc4000 },
+	{ _MMIO(0x9888), 0x0cdcc000 },
+	{ _MMIO(0x9888), 0x0edcc000 },
+	{ _MMIO(0x9888), 0x1abd00a8 },
+	{ _MMIO(0x9888), 0x0cd8c000 },
+	{ _MMIO(0x9888), 0x0ed84000 },
+	{ _MMIO(0x9888), 0x0edb8000 },
+	{ _MMIO(0x9888), 0x18db0800 },
+	{ _MMIO(0x9888), 0x1adb0254 },
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

drivers/gpu/drm/i915/i915_oa_bxt.c +2313 -10

2,323 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_chv.c +2515 -10

2,525 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_hsw.c +53 -5

58 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt2.c +3115 -41

3,156 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt3.c +2661 -41

2,702 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt4.c +2704 -41

2,745 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Part of the diff is not shown here. View the raw message.

#15 lore raw

PATCH v4 14/15 drm/i915/perf: per-gen timebase for checking sample freq

In reply to Robert Bragg

An oa_exponent_to_ns() utility and per-gen timebase constants where recently removed when updating the tail pointer race condition WA, and this restores those so we can update the _PROP_OA_EXPONENT validation done in read_properties_unlocked() to not assume we have a 12.5MHz timebase as we did for Haswell.

Accordingly the oa_sample_rate_hard_limit value that's referenced by proc_dointvec_minmax defining the absolute limit for the OA sampling frequency is now initialized to (timestamp_frequency / 2) instead of the 6.25MHz constant for Haswell.

v2:
Specify frequency of 19.2MHz for BXT (Ville) Initialize oa_sample_rate_hard_limit per-gen too (Lionel)

---
 drivers/gpu/drm/i915/i915_drv.h  |  1 +
 drivers/gpu/drm/i915/i915_perf.c | 37 ++++++++++++++++++++++++++-----------
 2 files changed, 27 insertions(+), 11 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
index b8dcf281db53..59dcce3b40a9 100644
@@ -2457,6 +2457,7 @@ struct drm_i915_private {
 
 			bool periodic;
 			int period_exponent;
+			int timestamp_frequency;
 
 			int metrics_set;
 
drivers/gpu/drm/i915/i915_perf.c +26 -11
index 611f996bece7..5de8d57e0b77 100644
@@ -288,10 +288,12 @@ static u32 i915_perf_stream_paranoid = true;
 
 /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
  *
- * 160ns is the smallest sampling period we can theoretically program the OA
- * unit with on Haswell, corresponding to 6.25MHz.
+ * The highest sampling frequency we can theoretically program the OA unit
+ * with is always half the timestamp frequency: E.g. 6.25Mhz for Haswell.
+ *
+ * Initialized just before we register the sysctl parameter.
  */
-static int oa_sample_rate_hard_limit = 6250000;
+static int oa_sample_rate_hard_limit;
 
 /* Theoretically we can program the OA unit to sample every 160ns but don't
  * allow that by default unless root...
@@ -2560,6 +2562,12 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
 	return ret;
 }
 
+static u64 oa_exponent_to_ns(struct drm_i915_private *dev_priv, int exponent)
+{
+	return div_u64(1000000000ULL * (2ULL << exponent),
+		       dev_priv->perf.oa.timestamp_frequency);
+}
+
 /**
  * read_properties_unlocked - validate + copy userspace stream open properties
  * @dev_priv: i915 device instance
@@ -2656,16 +2664,13 @@ static int read_properties_unlocked(struct drm_i915_private *dev_priv,
 			}
 
 			/* Theoretically we can program the OA unit to sample
-			 * every 160ns but don't allow that by default unless
-			 * root.
-			 *
-			 * On Haswell the period is derived from the exponent
-			 * as:
-			 *
-			 *   period = 80ns * 2^(exponent + 1)
+			 * e.g. every 160ns for HSW, 167ns for BDW/SKL or 104ns
+			 * for BXT. We don't allow such high sampling
+			 * frequencies by default unless root.
 			 */
+
 			BUILD_BUG_ON(sizeof(oa_period) != 8);
-			oa_period = 80ull * (2ull << value);
+			oa_period = oa_exponent_to_ns(dev_priv, value);
 
 			/* This check is primarily to ensure that oa_period <=
 			 * UINT32_MAX (before passing to do_div which only
@@ -2921,6 +2926,8 @@ void i915_perf_init(struct drm_i915_private *dev_priv)
 		dev_priv->perf.oa.ops.oa_hw_tail_read =
 			gen7_oa_hw_tail_read;
 
+		dev_priv->perf.oa.timestamp_frequency = 12500000;
+
 		dev_priv->perf.oa.oa_formats = hsw_oa_formats;
 
 		dev_priv->perf.oa.n_builtin_sets =
@@ -2934,6 +2941,8 @@ void i915_perf_init(struct drm_i915_private *dev_priv)
 		 */
 
 		if (IS_GEN8(dev_priv)) {
+			dev_priv->perf.oa.timestamp_frequency = 12500000;
+
 			dev_priv->perf.oa.ctx_oactxctrl_offset = 0x120;
 			dev_priv->perf.oa.ctx_flexeu0_offset = 0x2ce;
 			dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<25);
@@ -2950,6 +2959,8 @@ void i915_perf_init(struct drm_i915_private *dev_priv)
 					i915_oa_select_metric_set_chv;
 			}
 		} else if (IS_GEN9(dev_priv)) {
+			dev_priv->perf.oa.timestamp_frequency = 12000000;
+
 			dev_priv->perf.oa.ctx_oactxctrl_offset = 0x128;
 			dev_priv->perf.oa.ctx_flexeu0_offset = 0x3de;
 			dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<16);
@@ -2970,6 +2981,8 @@ void i915_perf_init(struct drm_i915_private *dev_priv)
 				dev_priv->perf.oa.ops.select_metric_set =
 					i915_oa_select_metric_set_sklgt4;
 			} else if (IS_BROXTON(dev_priv)) {
+				dev_priv->perf.oa.timestamp_frequency = 19200000;
+
 				dev_priv->perf.oa.n_builtin_sets =
 					i915_oa_n_builtin_metric_sets_bxt;
 				dev_priv->perf.oa.ops.select_metric_set =
@@ -3004,6 +3017,8 @@ void i915_perf_init(struct drm_i915_private *dev_priv)
 		spin_lock_init(&dev_priv->perf.hook_lock);
 		spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
 
+		oa_sample_rate_hard_limit =
+			dev_priv->perf.oa.timestamp_frequency / 2;
 		dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
 
 		dev_priv->perf.initialized = true;
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#16 lore raw

PATCH v4 15/15 drm/i915/perf: remove perf.hook_lock

In reply to Robert Bragg

In earlier iterations of the i915-perf driver we had a number of callbacks/hooks from other parts of the i915 driver to e.g. notify us when a legacy context was pinned and these could run asynchronously with respect to the stream file operations and might also run in atomic context.

dev_priv->perf.hook_lock had been for serialising access to state needed within these callbacks, but as the code has evolved some of the hooks have gone away or are implemented to avoid needing to lock any state.

The remaining use of this lock was actually redundant considering how the gen7 oacontrol state used to be updated as part of a context pin hook.

---
 drivers/gpu/drm/i915/i915_drv.h  |  2 --
 drivers/gpu/drm/i915/i915_perf.c | 32 ++++++++++----------------------
 2 files changed, 10 insertions(+), 24 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +0 -2
index 59dcce3b40a9..94c1f5331daf 100644
@@ -2438,8 +2438,6 @@ struct drm_i915_private {
 		struct mutex lock;
 		struct list_head streams;
 
-		spinlock_t hook_lock;
-
 		struct {
 			struct i915_perf_stream *exclusive_stream;
 
drivers/gpu/drm/i915/i915_perf.c +10 -22
index 5de8d57e0b77..1f25a6690f61 100644
@@ -1690,9 +1690,17 @@ static void gen8_disable_metric_set(struct drm_i915_private *dev_priv)
 	/* NOP */
 }
 
-static void gen7_update_oacontrol_locked(struct drm_i915_private *dev_priv)
+static void gen7_oa_enable(struct drm_i915_private *dev_priv)
 {
-	lockdep_assert_held(&dev_priv->perf.hook_lock);
+	/* Reset buf pointers so we don't forward reports from before now.
+	 *
+	 * Think carefully if considering trying to avoid this, since it
+	 * also ensures status flags and the buffer itself are cleared
+	 * in error paths, and we have checks for invalid reports based
+	 * on the assumption that certain fields are written to zeroed
+	 * memory which this helps maintains.
+	 */
+	gen7_init_oa_buffer(dev_priv);
 
 	if (dev_priv->perf.oa.exclusive_stream->enabled) {
 		struct i915_gem_context *ctx =
@@ -1715,25 +1723,6 @@ static void gen7_update_oacontrol_locked(struct drm_i915_private *dev_priv)
 		I915_WRITE(GEN7_OACONTROL, 0);
 }
 
-static void gen7_oa_enable(struct drm_i915_private *dev_priv)
-{
-	unsigned long flags;
-
-	/* Reset buf pointers so we don't forward reports from before now.
-	 *
-	 * Think carefully if considering trying to avoid this, since it
-	 * also ensures status flags and the buffer itself are cleared
-	 * in error paths, and we have checks for invalid reports based
-	 * on the assumption that certain fields are written to zeroed
-	 * memory which this helps maintains.
-	 */
-	gen7_init_oa_buffer(dev_priv);
-
-	spin_lock_irqsave(&dev_priv->perf.hook_lock, flags);
-	gen7_update_oacontrol_locked(dev_priv);
-	spin_unlock_irqrestore(&dev_priv->perf.hook_lock, flags);
-}
-
 static void gen8_oa_enable(struct drm_i915_private *dev_priv)
 {
 	u32 report_format = dev_priv->perf.oa.oa_buffer.format;
@@ -3014,7 +3003,6 @@ void i915_perf_init(struct drm_i915_private *dev_priv)
 
 		INIT_LIST_HEAD(&dev_priv->perf.streams);
 		mutex_init(&dev_priv->perf.lock);
-		spin_lock_init(&dev_priv->perf.hook_lock);
 		spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
 
 		oa_sample_rate_hard_limit =
Signature · 6 lines
2.12.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#17 lore raw

✓ Fi.CI.BAT: success for Enable OA unit for Gen 8 and 9 in i915 perf (rev6)

In reply to Robert Bragg
== Series Details ==

== Series Details ==

Series: Enable OA unit for Gen 8 and 9 in i915 perf (rev6)
URL : https://patchwork.freedesktop.org/series/20084/
State : success

== Summary ==

Series 20084v6 Enable OA unit for Gen 8 and 9 in i915 perf
https://patchwork.freedesktop.org/api/1.0/series/20084/revisions/6/mbox/

Test gem_exec_flush:

        Subgroup basic-batch-kernel-default-uc:
                fail       -> PASS       (fi-snb-2600) fdo#100007
Test gem_exec_suspend:
        Subgroup basic-s4-devices:
                dmesg-warn -> PASS       (fi-kbl-7560u) fdo#100125

fdo#100007 https://bugs.freedesktop.org/show_bug.cgi?id=100007
fdo#100125 https://bugs.freedesktop.org/show_bug.cgi?id=100125

fi-bdw-5557u     total:278  pass:267  dwarn:0   dfail:0   fail:0   skip:11  time:428s
fi-bdw-gvtdvm    total:278  pass:256  dwarn:8   dfail:0   fail:0   skip:14  time:428s
fi-bsw-n3050     total:278  pass:242  dwarn:0   dfail:0   fail:0   skip:36  time:590s
fi-bxt-t5700     total:278  pass:258  dwarn:0   dfail:0   fail:0   skip:20  time:549s
fi-byt-n2820     total:278  pass:250  dwarn:0   dfail:0   fail:0   skip:28  time:479s
fi-hsw-4770      total:278  pass:262  dwarn:0   dfail:0   fail:0   skip:16  time:411s
fi-hsw-4770r     total:278  pass:262  dwarn:0   dfail:0   fail:0   skip:16  time:403s
fi-ilk-650       total:278  pass:228  dwarn:0   dfail:0   fail:0   skip:50  time:422s
fi-ivb-3520m     total:278  pass:260  dwarn:0   dfail:0   fail:0   skip:18  time:492s
fi-ivb-3770      total:278  pass:260  dwarn:0   dfail:0   fail:0   skip:18  time:466s
fi-kbl-7500u     total:278  pass:260  dwarn:0   dfail:0   fail:0   skip:18  time:454s
fi-kbl-7560u     total:278  pass:268  dwarn:0   dfail:0   fail:0   skip:10  time:564s
fi-skl-6260u     total:278  pass:268  dwarn:0   dfail:0   fail:0   skip:10  time:452s
fi-skl-6700hq    total:278  pass:261  dwarn:0   dfail:0   fail:0   skip:17  time:575s
fi-skl-6700k     total:278  pass:256  dwarn:4   dfail:0   fail:0   skip:18  time:456s
fi-skl-6770hq    total:278  pass:268  dwarn:0   dfail:0   fail:0   skip:10  time:494s
fi-skl-gvtdvm    total:278  pass:265  dwarn:0   dfail:0   fail:0   skip:13  time:429s
fi-snb-2520m     total:278  pass:250  dwarn:0   dfail:0   fail:0   skip:28  time:533s
fi-snb-2600      total:278  pass:249  dwarn:0   dfail:0   fail:0   skip:29  time:403s

fi-bxt-j4205 failed to collect. IGT log at Patchwork_4497/fi-bxt-j4205/igt.log fi-byt-j1900 failed to collect. IGT log at Patchwork_4497/fi-byt-j1900/igt.log

57188aaf3b96d08e4095755fda0e577b1e0d8c60 drm-tip: 2017y-04m-12d-16h-12m-10s UTC integration manifest 19c9282 drm/i915/perf: remove perf.hook_lock
2f78071 drm/i915/perf: per-gen timebase for checking sample freq
953347e drm/i915/perf: Add more OA configs for BDW, CHV, SKL + BXT
96f8110 drm/i915/perf: Add OA unit support for Gen 8+
c5ccb72 drm/i915/perf: Add 'render basic' Gen8+ OA unit configs
8241dda drm/i915: expose _SUBSLICE_MASK GETPARM
0bae603 drm/i915: expose _SLICE_MASK GETPARM
29590b2 drm/i915/perf: rate limit spurious oa report notice
5ab4f33 drm/i915/perf: better pipeline aged/aging tail updates
818cc70 drm/i915/perf: improve invalid OA format debug message
9fe2f7e drm/i915/perf: improve tail race workaround
69ac693 drm/i915/perf: no head/tail ref in gen7_oa_read
00e051e drm/i915/perf: avoid read back of head register
372021d drm/i915/perf: avoid poll, read, EAGAIN busy loops
227c908 drm/i915/perf: fix gen7_append_oa_reports comment

== Logs ==

For more details see: https://intel-gfx-ci.01.org/CI/Patchwork_4497/
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#18 lore raw

cover Enable OA unit for Gen 8 and 9 in i915 perf

In reply to Robert Bragg

Hi,

Taking over from Rob for this v5. This series only has the following changes from v4 :

- patch 9 & 10 : updated number for GETPARAM after rebase

- patch 12 : drain the GPU before reconfiguring the OA unit to work around a race condition where the CPU & GPU update the context image at the same time (see gen8_configure_all_contexts)

The second item had been causing instabilities for a while in the IGT tests. With that out of the way, tests are a lot more stable now.

Cheers,

Robert Bragg (15):
drm/i915/perf: fix gen7_append_oa_reports comment drm/i915/perf: avoid poll, read, EAGAIN busy loops drm/i915/perf: avoid read back of head register drm/i915/perf: no head/tail ref in gen7_oa_read drm/i915/perf: improve tail race workaround drm/i915/perf: improve invalid OA format debug message drm/i915/perf: better pipeline aged/aging tail updates drm/i915/perf: rate limit spurious oa report notice drm/i915: expose _SLICE_MASK GETPARM
drm/i915: expose _SUBSLICE_MASK GETPARM drm/i915/perf: Add 'render basic' Gen8+ OA unit configs drm/i915/perf: Add OA unit support for Gen 8+ drm/i915/perf: Add more OA configs for BDW, CHV, SKL + BXT drm/i915/perf: per-gen timebase for checking sample freq drm/i915/perf: remove perf.hook_lock

 drivers/gpu/drm/i915/Makefile           |    8 +-
 drivers/gpu/drm/i915/i915_drv.c         |   10 +
 drivers/gpu/drm/i915/i915_drv.h         |  121 +-
 drivers/gpu/drm/i915/i915_gem_context.h |    1 +
 drivers/gpu/drm/i915/i915_oa_bdw.c      | 5154 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bdw.h      |   38 +
 drivers/gpu/drm/i915/i915_oa_bxt.c      | 2541 +++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bxt.h      |   38 +
 drivers/gpu/drm/i915/i915_oa_chv.c      | 2730 ++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_chv.h      |   38 +
 drivers/gpu/drm/i915/i915_oa_hsw.c      |   58 +-
 drivers/gpu/drm/i915/i915_oa_sklgt2.c   | 3302 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt2.h   |   38 +
 drivers/gpu/drm/i915/i915_oa_sklgt3.c   | 2856 +++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt3.h   |   38 +
 drivers/gpu/drm/i915/i915_oa_sklgt4.c   | 2910 +++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt4.h   |   38 +
 drivers/gpu/drm/i915/i915_perf.c        | 1361 ++++++--
 drivers/gpu/drm/i915/i915_reg.h         |   22 +
 drivers/gpu/drm/i915/intel_lrc.c        |    5 +
 include/uapi/drm/i915_drm.h             |   27 +-
 21 files changed, 21096 insertions(+), 238 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.h

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#19 lore raw

PATCH v5 1/15 drm/i915/perf: fix gen7_append_oa_reports comment

From: Robert Bragg <[email protected]>

If I'm going to complain about a back-to-front convention then the least I can do is not muddle the comment up too.

---
 drivers/gpu/drm/i915/i915_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_perf.c +1 -1
index 060b171480d5..78fef53b45c9 100644
@@ -431,7 +431,7 @@ static int append_oa_sample(struct i915_perf_stream *stream,
  * userspace.
  *
  * Note: reports are consumed from the head, and appended to the
- * tail, so the head chases the tail?... If you think that's mad
+ * tail, so the tail chases the head?... If you think that's mad
  * and back-to-front you're not alone, but this follows the
  * Gen PRM naming convention.
  *

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#20 lore raw

PATCH v5 2/15 drm/i915/perf: avoid poll, read, EAGAIN busy loops

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

If the function for checking whether there is OA buffer data available (during a poll or blocking read) has false positives then we want to avoid a situation where the subsequent read() returns EAGAIN (after a more accurate check) followed by a poll() immediately reporting the same false positive POLLIN event and effectively maintaining a busy loop until there really is data.

This makes sure that we clear the .pollin event status whenever we return EAGAIN to userspace which will throttle subsequent POLLIN events and repeated attempts to read to the 5ms intervals of the hrtimer callback we have.

---
 drivers/gpu/drm/i915/i915_perf.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_perf.c +9 -1
index 78fef53b45c9..f59f6dd20922 100644
@@ -1352,7 +1352,15 @@ static ssize_t i915_perf_read(struct file *file,
 		mutex_unlock(&dev_priv->perf.lock);
 	}
 
-	if (ret >= 0) {
+	/* We allow the poll checking to sometimes report false positive POLLIN
+	 * events where we might actually report EAGAIN on read() if there's
+	 * not really any data available. In this situation though we don't
+	 * want to enter a busy loop between poll() reporting a POLLIN event
+	 * and read() returning -EAGAIN. Clearing the oa.pollin state here
+	 * effectively ensures we back off until the next hrtimer callback
+	 * before reporting another POLLIN event.
+	 */
+	if (ret >= 0 || ret == -EAGAIN) {
 		/* Maybe make ->pollin per-stream state if we support multiple
 		 * concurrent streams in the future.
 		 */

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#21 lore raw

PATCH v5 3/15 drm/i915/perf: avoid read back of head register

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

There's no need for the driver to keep reading back the head pointer from hardware since the hardware doesn't update it automatically. This way we can treat any invalid head pointer value as a software/driver bug instead of spurious hardware behaviour.

This change is also a small stepping stone towards re-working how the head and tail state is managed as part of an improved workaround for the tail register race condition.

---
 drivers/gpu/drm/i915/i915_drv.h  | 11 ++++++++++
 drivers/gpu/drm/i915/i915_perf.c | 46 ++++++++++++++++++----------------------
 2 files changed, 32 insertions(+), 25 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 357b6c6c2f04..b618a8bc54f6 100644
@@ -2469,6 +2469,17 @@ struct drm_i915_private {
 				u8 *vaddr;
 				int format;
 				int format_size;
+
+				/**
+				 * Although we can always read back the head
+				 * pointer register, we prefer to avoid
+				 * trusting the HW state, just to avoid any
+				 * risk that some hardware condition could
+				 * somehow bump the head pointer unpredictably
+				 * and cause us to forward the wrong OA buffer
+				 * data to userspace.
+				 */
+				u32 head;
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
drivers/gpu/drm/i915/i915_perf.c +21 -25

104 lines not on this page · raw message

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#22 lore raw

PATCH v5 4/15 drm/i915/perf: no head/tail ref in gen7_oa_read

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

This avoids redundantly passing an (inout) head and tail pointer to gen7_append_oa_reports() from gen7_oa_read which doesn't need to reference either itself.

Moving the head/tail reads and writes into gen7_append_oa_reports should have no functional effect except to avoid some redundant head pointer writes in cases where nothing was copied to userspace.

This is a stepping stone towards updating how the head and tail pointer state is managed to improve the workaround for the OA unit's tail pointer race. It reduces the number of places we need to read/write the head and tail pointers.

---
 drivers/gpu/drm/i915/i915_perf.c | 51 +++++++++++++++-------------------------
 1 file changed, 19 insertions(+), 32 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +19 -32
index f47d1cc2144b..83dc67a635fb 100644
@@ -420,8 +420,6 @@ static int append_oa_sample(struct i915_perf_stream *stream,
  * @buf: destination buffer given by userspace
  * @count: the number of bytes userspace wants to read
  * @offset: (inout): the current position for writing into @buf
- * @head_ptr: (inout): the current oa buffer cpu read position
- * @tail: the current oa buffer gpu write position
  *
  * Notably any error condition resulting in a short read (-%ENOSPC or
  * -%EFAULT) will be returned even though one or more records may
@@ -439,9 +437,7 @@ static int append_oa_sample(struct i915_perf_stream *stream,
 static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 				  char __user *buf,
 				  size_t count,
-				  size_t *offset,
-				  u32 *head_ptr,
-				  u32 tail)
+				  size_t *offset)
 {
 	struct drm_i915_private *dev_priv = stream->dev_priv;
 	int report_size = dev_priv->perf.oa.oa_buffer.format_size;
@@ -449,14 +445,15 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 	int tail_margin = dev_priv->perf.oa.tail_margin;
 	u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
 	u32 mask = (OA_BUFFER_SIZE - 1);
-	u32 head;
+	size_t start_offset = *offset;
+	u32 head, oastatus1, tail;
 	u32 taken;
 	int ret = 0;
 
 	if (WARN_ON(!stream->enabled))
 		return -EIO;
 
-	head = *head_ptr - gtt_offset;
+	head = dev_priv->perf.oa.oa_buffer.head - gtt_offset;
 
 	/* An out of bounds or misaligned head pointer implies a driver bug
 	 * since we are in full control of head pointer which should only
@@ -467,7 +464,8 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 		      "Inconsistent OA buffer head pointer = %u\n", head))
 		return -EIO;
 
-	tail -= gtt_offset;
+	oastatus1 = I915_READ(GEN7_OASTATUS1);
+	tail = (oastatus1 & GEN7_OASTATUS1_TAIL_MASK) - gtt_offset;
 
 	/* The OA unit is expected to wrap the tail pointer according to the OA
 	 * buffer size
@@ -477,8 +475,6 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 			  tail);
 		dev_priv->perf.oa.ops.oa_disable(dev_priv);
 		dev_priv->perf.oa.ops.oa_enable(dev_priv);
-		*head_ptr = I915_READ(GEN7_OASTATUS2) &
-			GEN7_OASTATUS2_HEAD_MASK;
 		return -EIO;
 	}
 
@@ -542,7 +538,18 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 		report32[0] = 0;
 	}
 
-	*head_ptr = gtt_offset + head;
+
+	if (start_offset != *offset) {
+		/* We removed the gtt_offset for the copy loop above, indexing
+		 * relative to oa_buf_base so put back here...
+		 */
+		head += gtt_offset;
+
+		I915_WRITE(GEN7_OASTATUS2,
+			   ((head & GEN7_OASTATUS2_HEAD_MASK) |
+			    OA_MEM_SELECT_GGTT));
+		dev_priv->perf.oa.oa_buffer.head = head;
+	}
 
 	return ret;
 }
@@ -570,8 +577,6 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 {
 	struct drm_i915_private *dev_priv = stream->dev_priv;
 	u32 oastatus1;
-	u32 head;
-	u32 tail;
 	int ret;
 
 	if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
@@ -579,9 +584,6 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 
 	oastatus1 = I915_READ(GEN7_OASTATUS1);
 
-	head = dev_priv->perf.oa.oa_buffer.head;
-	tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
-
 	/* XXX: On Haswell we don't have a safe way to clear oastatus1
 	 * bits while the OA unit is enabled (while the tail pointer
 	 * may be updated asynchronously) so we ignore status bits
@@ -621,9 +623,6 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 		dev_priv->perf.oa.ops.oa_enable(dev_priv);
 
 		oastatus1 = I915_READ(GEN7_OASTATUS1);
-
-		head = dev_priv->perf.oa.oa_buffer.head;
-		tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
 	}
 
 	if (unlikely(oastatus1 & GEN7_OASTATUS1_REPORT_LOST)) {
@@ -635,19 +634,7 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 			GEN7_OASTATUS1_REPORT_LOST;
 	}
 
-	ret = gen7_append_oa_reports(stream, buf, count, offset,
-				     &head, tail);
-
-	/* Note: we update the head pointer here even if an error
-	 * was returned since the error may represent a short read
-	 * where some some reports were successfully copied.
-	 */
-	I915_WRITE(GEN7_OASTATUS2,
-		   ((head & GEN7_OASTATUS2_HEAD_MASK) |
-		    OA_MEM_SELECT_GGTT));
-	dev_priv->perf.oa.oa_buffer.head = head;
-
-	return ret;
+	return gen7_append_oa_reports(stream, buf, count, offset);
 }
 
 /**

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#23 lore raw

PATCH v5 5/15 drm/i915/perf: improve tail race workaround

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

There's a HW race condition between OA unit tail pointer register updates and writes to memory whereby the tail pointer can sometimes get ahead of what's been written out to the OA buffer so far (in terms of what's visible to the CPU).

Although this can be observed explicitly while copying reports to userspace by checking for a zeroed report-id field in tail reports, we want to account for this earlier, as part of the _oa_buffer_check to avoid lots of redundant read() attempts.

Previously the driver used to define an effective tail pointer that lagged the real pointer by a 'tail margin' measured in bytes derived from OA_TAIL_MARGIN_NSEC and the configured sampling frequency. Unfortunately this was flawed considering that the OA unit may also automatically generate non-periodic reports (such as on context switch) or the OA unit may be enabled without any periodic sampling.

This improves how we define a tail pointer for reading that lags the real tail pointer by at least %OA_TAIL_MARGIN_NSEC nanoseconds, which gives enough time for the corresponding reports to become visible to the CPU.

The driver now maintains two tail pointers:
1) An 'aging' tail with an associated timestamp that is tracked until we can trust the corresponding data is visible to the CPU; at which point it is considered 'aged'.
2) An 'aged' tail that can be used for read()ing.

The two separate pointers let us decouple read()s from tail pointer aging.

The tail pointers are checked and updated at a limited rate within a hrtimer callback (the same callback that is used for delivering POLLIN events) and since we're now measuring the wall clock time elapsed since a given tail pointer was read the mechanism no longer cares about the OA unit's periodic sampling frequency.

The natural place to handle the tail pointer updates was in gen7_oa_buffer_is_empty() which is called as part of blocking reads and the hrtimer callback used for polling, and so this was renamed to oa_buffer_check() considering the added side effect while checking whether the buffer contains data.

---
 drivers/gpu/drm/i915/i915_drv.h  |  60 ++++++++-
 drivers/gpu/drm/i915/i915_perf.c | 277 ++++++++++++++++++++++++++-------------
 2 files changed, 241 insertions(+), 96 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +55 -5
index b618a8bc54f6..69a914870774 100644
@@ -2097,7 +2097,7 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_is_empty: Check if OA buffer empty (false positives OK)
+	 * @oa_buffer_check: Check for OA buffer data + update tail
 	 *
 	 * This is either called via fops or the poll check hrtimer (atomic
 	 * ctx) without any locks taken.
@@ -2110,7 +2110,7 @@ struct i915_oa_ops {
 	 * here, which will be handled gracefully - likely resulting in an
 	 * %EAGAIN error for userspace.
 	 */
-	bool (*oa_buffer_is_empty)(struct drm_i915_private *dev_priv);
+	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2453,9 +2453,6 @@ struct drm_i915_private {
 
 			bool periodic;
 			int period_exponent;
-			int timestamp_frequency;
-
-			int tail_margin;
 
 			int metrics_set;
 
@@ -2471,6 +2468,59 @@ struct drm_i915_private {
 				int format_size;
 
 				/**
+				 * Locks reads and writes to all head/tail state
+				 *
+				 * Consider: the head and tail pointer state
+				 * needs to be read consistently from a hrtimer
+				 * callback (atomic context) and read() fop
+				 * (user context) with tail pointer updates
+				 * happening in atomic context and head updates
+				 * in user context and the (unlikely)
+				 * possibility of read() errors needing to
+				 * reset all head/tail state.
+				 *
+				 * Note: Contention or performance aren't
+				 * currently a significant concern here
+				 * considering the relatively low frequency of
+				 * hrtimer callbacks (5ms period) and that
+				 * reads typically only happen in response to a
+				 * hrtimer event and likely complete before the
+				 * next callback.
+				 *
+				 * Note: This lock is not held *while* reading
+				 * and copying data to userspace so the value
+				 * of head observed in htrimer callbacks won't
+				 * represent any partial consumption of data.
+				 */
+				spinlock_t ptr_lock;
+
+				/**
+				 * One 'aging' tail pointer and one 'aged'
+				 * tail pointer ready to used for reading.
+				 *
+				 * Initial values of 0xffffffff are invalid
+				 * and imply that an update is required
+				 * (and should be ignored by an attempted
+				 * read)
+				 */
+				struct {
+					u32 offset;
+				} tails[2];
+
+				/**
+				 * Index for the aged tail ready to read()
+				 * data up to.
+				 */
+				unsigned int aged_tail_idx;
+
+				/**
+				 * A monotonic timestamp for when the current
+				 * aging tail pointer was read; used to
+				 * determine when it is old enough to trust.
+				 */
+				u64 aging_timestamp;
+
+				/**
 				 * Although we can always read back the head
 				 * pointer register, we prefer to avoid
 				 * trusting the HW state, just to avoid any
drivers/gpu/drm/i915/i915_perf.c +186 -91

388 lines not on this page · raw message

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#24 lore raw

PATCH v5 6/15 drm/i915/perf: improve invalid OA format debug message

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

A minor improvement to debugging output

---
 drivers/gpu/drm/i915/i915_perf.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +4 -2
index 18734e1926b9..08cc2b0dd734 100644
@@ -1904,11 +1904,13 @@ static int read_properties_unlocked(struct drm_i915_private *dev_priv,
 			break;
 		case DRM_I915_PERF_PROP_OA_FORMAT:
 			if (value == 0 || value >= I915_OA_FORMAT_MAX) {
-				DRM_DEBUG("Invalid OA report format\n");
+				DRM_DEBUG("Out-of-range OA report format %llu\n",
+					  value);
 				return -EINVAL;
 			}
 			if (!dev_priv->perf.oa.oa_formats[value].size) {
-				DRM_DEBUG("Invalid OA report format\n");
+				DRM_DEBUG("Unsupported OA report format %llu\n",
+					  value);
 				return -EINVAL;
 			}
 			props->oa_format = value;

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#25 lore raw

PATCH v5 7/15 drm/i915/perf: better pipeline aged/aging tail updates

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

This updates the tail pointer race workaround handling to updating the 'aged' pointer before looking to start aging a new one. There's the possibility that there is already new data available and so we can immediately start aging a new pointer without having to first wait for a later hrtimer callback (and then another to age).

---
 drivers/gpu/drm/i915/i915_perf.c | 41 ++++++++++++++++++++++------------------
 1 file changed, 23 insertions(+), 18 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +23 -18
index 08cc2b0dd734..5738b99caa5b 100644
@@ -391,6 +391,29 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 
 	now = ktime_get_mono_fast_ns();
 
+	/* Update the aged tail
+	 *
+	 * Flip the tail pointer available for read()s once the aging tail is
+	 * old enough to trust that the corresponding data will be visible to
+	 * the CPU...
+	 *
+	 * Do this before updating the aging pointer in case we may be able to
+	 * immediately start aging a new pointer too (if new data has become
+	 * available) without needing to wait for a later hrtimer callback.
+	 */
+	if (aging_tail != INVALID_TAIL_PTR &&
+	    ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
+	     OA_TAIL_MARGIN_NSEC)) {
+		aged_idx ^= 1;
+		dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
+
+		aged_tail = aging_tail;
+
+		/* Mark that we need a new pointer to start aging... */
+		dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset = INVALID_TAIL_PTR;
+		aging_tail = INVALID_TAIL_PTR;
+	}
+
 	/* Update the aging tail
 	 *
 	 * We throttle aging tail updates until we have a new tail that
@@ -420,24 +443,6 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 		}
 	}
 
-	/* Update the aged tail
-	 *
-	 * Flip the tail pointer available for read()s once the aging tail is
-	 * old enough to trust that the corresponding data will be visible to
-	 * the CPU...
-	 */
-	if (aging_tail != INVALID_TAIL_PTR &&
-	    ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
-	     OA_TAIL_MARGIN_NSEC)) {
-		aged_idx ^= 1;
-		dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
-
-		aged_tail = aging_tail;
-
-		/* Mark that we need a new pointer to start aging... */
-		dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset = INVALID_TAIL_PTR;
-	}
-
 	spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
 
 	return aged_tail == INVALID_TAIL_PTR ?

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#26 lore raw

PATCH v5 8/15 drm/i915/perf: rate limit spurious oa report notice

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

This change is pre-emptively aiming to avoid a potential cause of kernel logging noise in case some condition were to result in us seeing invalid OA reports.

The workaround for the OA unit's tail pointer race condition is what avoids the primary known cause of invalid reports being seen and with that in place we aren't expecting to see this notice but it can't be entirely ruled out.

Just in case some condition does lead to the notice then it's likely that it will be triggered repeatedly while attempting to append a sequence of reports and depending on the configured OA sampling frequency that might be a large number of repeat notices.

v2: (Chris) avoid inconsistent warning on throttle with
printk_ratelimit()
v3: (Matt) init and summarise with stream init/close not driver init/fini

---
 drivers/gpu/drm/i915/i915_drv.h  |  6 ++++++
 drivers/gpu/drm/i915/i915_perf.c | 28 +++++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_drv.h +6 -0
index 69a914870774..fb2188c8d72f 100644
@@ -2451,6 +2451,12 @@ struct drm_i915_private {
 			wait_queue_head_t poll_wq;
 			bool pollin;
 
+			/**
+			 * For rate limiting any notifications of spurious
+			 * invalid OA reports
+			 */
+			struct ratelimit_state spurious_report_rs;
+
 			bool periodic;
 			int period_exponent;
 
drivers/gpu/drm/i915/i915_perf.c +27 -1
index 5738b99caa5b..3277a52ce98e 100644
@@ -632,7 +632,8 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 		 * copying it to userspace...
 		 */
 		if (report32[0] == 0) {
-			DRM_NOTE("Skipping spurious, invalid OA report\n");
+			if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+				DRM_NOTE("Skipping spurious, invalid OA report\n");
 			continue;
 		}
 
@@ -913,6 +914,11 @@ static void i915_oa_stream_destroy(struct i915_perf_stream *stream)
 		oa_put_render_ctx_id(stream);
 
 	dev_priv->perf.oa.exclusive_stream = NULL;
+
+	if (dev_priv->perf.oa.spurious_report_rs.missed) {
+		DRM_NOTE("%d spurious OA report notices suppressed due to ratelimiting\n",
+			 dev_priv->perf.oa.spurious_report_rs.missed);
+	}
 }
 
 static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
@@ -1268,6 +1274,26 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream,
 		return -EINVAL;
 	}
 
+	/* We set up some ratelimit state to potentially throttle any _NOTES
+	 * about spurious, invalid OA reports which we don't forward to
+	 * userspace.
+	 *
+	 * The initialization is associated with opening the stream (not driver
+	 * init) considering we print a _NOTE about any throttling when closing
+	 * the stream instead of waiting until driver _fini which no one would
+	 * ever see.
+	 *
+	 * Using the same limiting factors as printk_ratelimit()
+	 */
+	ratelimit_state_init(&dev_priv->perf.oa.spurious_report_rs,
+			     5 * HZ, 10);
+	/* Since we use a DRM_NOTE for spurious reports it would be
+	 * inconsistent to let __ratelimit() automatically print a warning for
+	 * throttling.
+	 */
+	ratelimit_set_flags(&dev_priv->perf.oa.spurious_report_rs,
+			    RATELIMIT_MSG_ON_RELEASE);
+
 	stream->sample_size = sizeof(struct drm_i915_perf_record_header);
 
 	format_size = dev_priv->perf.oa.oa_formats[props->oa_format].size;

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#27 lore raw

PATCH v5 9/15 drm/i915: expose _SLICE_MASK GETPARM

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables userspace to determine the number of slices enabled and also know what specific slices are enabled. This information is required, for example, to be able to analyse some OA counter reports where the counter configuration depends on the HW slice configuration.

---
 drivers/gpu/drm/i915/i915_drv.c | 5 +++++
 include/uapi/drm/i915_drm.h     | 3 +++
 2 files changed, 8 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +5 -0
index cc7393e65e99..f13b2c3ce6eb 100644
@@ -358,6 +358,11 @@ static int i915_getparam(struct drm_device *dev, void *data,
 		 */
 		value = 1;
 		break;
+	case I915_PARAM_SLICE_MASK:
+		value = INTEL_INFO(dev_priv)->sseu.slice_mask;
+		if (!value)
+			return -ENODEV;
+		break;
 	default:
 		DRM_DEBUG("Unknown parameter %d\n", param->param);
 		return -EINVAL;
include/uapi/drm/i915_drm.h +3 -0
index f24a80d2d42e..25695c3d9a76 100644
@@ -418,6 +418,9 @@ typedef struct drm_i915_irq_wait {
  */
 #define I915_PARAM_HAS_EXEC_CAPTURE	 45
 
+/* Query the mask of slices available for this system */
+#define I915_PARAM_SLICE_MASK		 46
+
 typedef struct drm_i915_getparam {
 	__s32 param;
 	/*

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#28 lore raw

PATCH v5 10/15 drm/i915: expose _SUBSLICE_MASK GETPARM

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

Assuming a uniform mask across all slices, this enables userspace to determine the specific sub slices enabled. This information is required, for example, to be able to analyse some OA counter reports where the counter configuration depends on the HW sub slice configuration.

---
 drivers/gpu/drm/i915/i915_drv.c | 5 +++++
 include/uapi/drm/i915_drm.h     | 5 +++++
 2 files changed, 10 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +5 -0
index f13b2c3ce6eb..8769692b9408 100644
@@ -363,6 +363,11 @@ static int i915_getparam(struct drm_device *dev, void *data,
 		if (!value)
 			return -ENODEV;
 		break;
+	case I915_PARAM_SUBSLICE_MASK:
+		value = INTEL_INFO(dev_priv)->sseu.subslice_mask;
+		if (!value)
+			return -ENODEV;
+		break;
 	default:
 		DRM_DEBUG("Unknown parameter %d\n", param->param);
 		return -EINVAL;
include/uapi/drm/i915_drm.h +5 -0
index 25695c3d9a76..464547d08173 100644
@@ -421,6 +421,11 @@ typedef struct drm_i915_irq_wait {
 /* Query the mask of slices available for this system */
 #define I915_PARAM_SLICE_MASK		 46
 
+/* Assuming it's uniform for each slice, this queries the mask of subslices
+ * per-slice for this system.
+ */
+#define I915_PARAM_SUBSLICE_MASK	 47
+
 typedef struct drm_i915_getparam {
 	__s32 param;
 	/*

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#29 lore raw

PATCH v5 11/15 drm/i915/perf: Add 'render basic' Gen8+ OA unit configs

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

Adds a static OA unit, MUX, B Counter + Flex EU configurations for basic render metrics on Broadwell, Cherryview, Skylake and Broxton. These are auto generated from an XML description of metric sets, currently maintained in gputop, ref:

https://github.com/rib/gputop

Quoted 2 lines

gputop-data/oa-*.xml
scripts/i915-perf-kernelgen.py

 $ make -C gputop-data -f Makefile.xml WHITELIST=RenderBasic

v2: add newlines to debug messages + fix comment (Matthew Auld)

---
 drivers/gpu/drm/i915/Makefile         |   8 +-
 drivers/gpu/drm/i915/i915_drv.h       |   2 +
 drivers/gpu/drm/i915/i915_oa_bdw.c    | 380 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bdw.h    |  38 ++++
 drivers/gpu/drm/i915/i915_oa_bxt.c    | 238 +++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_bxt.h    |  38 ++++
 drivers/gpu/drm/i915/i915_oa_chv.c    | 225 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_chv.h    |  38 ++++
 drivers/gpu/drm/i915/i915_oa_sklgt2.c | 228 ++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt2.h |  38 ++++
 drivers/gpu/drm/i915/i915_oa_sklgt3.c | 236 +++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt3.h |  38 ++++
 drivers/gpu/drm/i915/i915_oa_sklgt4.c | 247 ++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_oa_sklgt4.h |  38 ++++
 14 files changed, 1791 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bdw.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_bxt.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_chv.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt2.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt3.h
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.c
 create mode 100644 drivers/gpu/drm/i915/i915_oa_sklgt4.h
drivers/gpu/drm/i915/Makefile +7 -1
index 2cf04504e494..41400a138a1e 100644
@@ -127,7 +127,13 @@ i915-y += i915_vgpu.o
 
 # perf code
 i915-y += i915_perf.o \
-	  i915_oa_hsw.o
+	  i915_oa_hsw.o \
+	  i915_oa_bdw.o \
+	  i915_oa_chv.o \
+	  i915_oa_sklgt2.o \
+	  i915_oa_sklgt3.o \
+	  i915_oa_sklgt4.o \
+	  i915_oa_bxt.o
 
 ifeq ($(CONFIG_DRM_I915_GVT),y)
 i915-y += intel_gvt.o
drivers/gpu/drm/i915/i915_drv.h +2 -0
index fb2188c8d72f..ffa1fc5eddfd 100644
@@ -2466,6 +2466,8 @@ struct drm_i915_private {
 			int mux_regs_len;
 			const struct i915_oa_reg *b_counter_regs;
 			int b_counter_regs_len;
+			const struct i915_oa_reg *flex_regs;
+			int flex_regs_len;
 
 			struct {
 				struct i915_vma *vma;
drivers/gpu/drm/i915/i915_oa_bdw.c new file +380 -0

380 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_bdw.h new file +38 -0
new file mode 100644
index 000000000000..ea6969aaf91a
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_BDW_H__
+#define __I915_OA_BDW_H__
+
+extern int i915_oa_n_builtin_metric_sets_bdw;
+
+extern int i915_oa_select_metric_set_bdw(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_bdw(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_bdw(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_bxt.c new file +238 -0

238 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_bxt.h new file +38 -0
new file mode 100644
index 000000000000..9e86662c891d
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_BXT_H__
+#define __I915_OA_BXT_H__
+
+extern int i915_oa_n_builtin_metric_sets_bxt;
+
+extern int i915_oa_select_metric_set_bxt(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_bxt(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_bxt(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_chv.c new file +225 -0

225 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_chv.h new file +38 -0
new file mode 100644
index 000000000000..e494200ff04f
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_CHV_H__
+#define __I915_OA_CHV_H__
+
+extern int i915_oa_n_builtin_metric_sets_chv;
+
+extern int i915_oa_select_metric_set_chv(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_chv(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_chv(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_sklgt2.c new file +228 -0

228 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt2.h new file +38 -0
new file mode 100644
index 000000000000..e99b03f1bf9e
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_SKLGT2_H__
+#define __I915_OA_SKLGT2_H__
+
+extern int i915_oa_n_builtin_metric_sets_sklgt2;
+
+extern int i915_oa_select_metric_set_sklgt2(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_sklgt2(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_sklgt2(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_sklgt3.c new file +236 -0

236 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt3.h new file +38 -0
new file mode 100644
index 000000000000..ff7ec3206bf2
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_SKLGT3_H__
+#define __I915_OA_SKLGT3_H__
+
+extern int i915_oa_n_builtin_metric_sets_sklgt3;
+
+extern int i915_oa_select_metric_set_sklgt3(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_sklgt3(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_sklgt3(struct drm_i915_private *dev_priv);
+
+#endif
drivers/gpu/drm/i915/i915_oa_sklgt4.c new file +247 -0

247 lines not on this page · raw message

drivers/gpu/drm/i915/i915_oa_sklgt4.h new file +38 -0
new file mode 100644
index 000000000000..4faa8c4579cd
@@ -0,0 +1,38 @@
+/*
+ * Autogenerated file, DO NOT EDIT manually!
+ *
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_OA_SKLGT4_H__
+#define __I915_OA_SKLGT4_H__
+
+extern int i915_oa_n_builtin_metric_sets_sklgt4;
+
+extern int i915_oa_select_metric_set_sklgt4(struct drm_i915_private *dev_priv);
+
+extern int i915_perf_register_sysfs_sklgt4(struct drm_i915_private *dev_priv);
+
+extern void i915_perf_unregister_sysfs_sklgt4(struct drm_i915_private *dev_priv);
+
+#endif

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#30 lore raw

PATCH v5 12/15 drm/i915/perf: Add OA unit support for Gen 8+

In reply to Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

---
 drivers/gpu/drm/i915/i915_drv.h         |  45 +-
 drivers/gpu/drm/i915/i915_gem_context.h |   1 +
 drivers/gpu/drm/i915/i915_perf.c        | 969 +++++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_reg.h         |  22 +
 drivers/gpu/drm/i915/intel_lrc.c        |   5 +
 include/uapi/drm/i915_drm.h             |  19 +-
 6 files changed, 968 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index ffa1fc5eddfd..41a97ade39ea 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_update_reg_state(struct intel_engine_cs *engine,
+			      struct i915_gem_context *ctx,
+			      uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_gem_context.h +1 -0
index 4af2ab94558b..3f4ce73bea43 100644
@@ -151,6 +151,7 @@ struct i915_gem_context {
 		u64 lrc_desc;
 		int pin_count;
 		bool initialised;
+		atomic_t oa_state_dirty;
 	} engine[I915_NUM_ENGINES];
 
 	/** ring_size: size for allocating the per-engine ring buffer */
drivers/gpu/drm/i915/i915_perf.c +898 -71

1,158 lines not on this page · raw message

drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +5 -0
index 7df278fe492e..e787656b630f 100644
@@ -337,6 +337,8 @@ static u64 execlists_update_context(struct drm_i915_gem_request *rq)
 	if (ppgtt && !i915_vm_is_48bit(&ppgtt->base))
 		execlists_update_context_pdps(ppgtt, reg_state);
 
+	i915_oa_update_reg_state(rq->engine, rq->ctx, reg_state);
+
 	return ce->lrc_desc;
 }
 
@@ -1879,6 +1881,9 @@ static void execlists_init_reg_state(u32 *regs,
 		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
 		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
 			make_rpcs(dev_priv));
+
+		atomic_set(&ctx->engine[RCS].oa_state_dirty, 1);
+		i915_oa_update_reg_state(engine, ctx, regs);
 	}
 }
 
include/uapi/drm/i915_drm.h +12 -7
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
 	I915_OA_FORMAT_MAX	    /* non-ABI */
 };

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#31 lore raw

Re: [PATCH v5 12/15] drm/i915/perf: Add OA unit support for Gen 8+

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1584

Quoted 8 lines · Lionel Landwerlin
drivers/gpu/drm/i915/i915_perf.c +8 -0
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{
+	struct i915_gem_context *ctx;
+	int ret;
+
+	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+	if (ret)
+		return ret;

Missed a switch to kernel context, to ensure that the last context is also saved to memory.

Quoted 24 lines
drivers/gpu/drm/i915/i915_perf.c +24 -0
+
+	/* The OA register config is setup through the context image. This image
+	 * might be written to by the GPU on context switch (in particular on
+	 * lite-restore). This means we can't safely update a context's image,
+	 * if this context is scheduled/submitted to run on the GPU.
+	 *
+	 * We could emit the OA register config through the batch buffer but
+	 * this might leave small interval of time where the OA unit is
+	 * configured at an invalid sampling period.
+	 *
+	 * So far the best way to work around this issue seems to be draining
+	 * the GPU from any submitted work.
+	 */
+        ret = i915_gem_wait_for_idle(dev_priv,
+                                     I915_WAIT_INTERRUPTIBLE |
+                                     I915_WAIT_LOCKED);
+        if (ret) {
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+                return ret;
+	}
+
+	/* Since execlist submission may be happening asynchronously here then
+	 * we first mark existing contexts dirty before we update the current
+	 * context so if any switches happen in the middle we can expect

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1618

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+	 * that the act of scheduling will have itself ensured a consistent
+	 * OA state update.
+	 */
+	list_for_each_entry(ctx, &dev_priv->context_list, link) {
+		/* The actual update of the register state context will happen
+		 * the next time this logical ring is submitted. (See
+		 * i915_oa_update_reg_state() which hooks into
+		 * execlists_update_context())
+		 */
+		atomic_set(&ctx->engine[RCS].oa_state_dirty, 1);

Considering that everything is now available to be modified, you can change the contexts right here and now and remove the checks from the much more common path.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#32 lore raw

PATCH v6 12/15 drm/i915/perf: Add OA unit support for Gen 8+

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

---
 drivers/gpu/drm/i915/i915_drv.h  |  45 +-
 drivers/gpu/drm/i915/i915_perf.c | 957 ++++++++++++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_reg.h  |  22 +
 drivers/gpu/drm/i915/intel_lrc.c |   2 +
 include/uapi/drm/i915_drm.h      |  19 +-
 5 files changed, 952 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index ffa1fc5eddfd..676b1227067c 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +886 -71

1,146 lines not on this page · raw message

drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 7df278fe492e..351e231a02f4 100644
@@ -1879,6 +1879,8 @@ static void execlists_init_reg_state(u32 *regs,
 		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
 		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
 			make_rpcs(dev_priv));
+
+		i915_oa_init_reg_state(engine, ctx, regs);
 	}
 }
 
include/uapi/drm/i915_drm.h +12 -7
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
 	I915_OA_FORMAT_MAX	    /* non-ABI */
 };

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#33 lore raw

Re: [PATCH v6 12/15] drm/i915/perf: Add OA unit support for Gen 8+

Hey Matt,

This commit had your reviewed-by on v4, are you still okay with it?

Thanks!

-
Lionel

Quoted 1,443 lines · Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

---
  drivers/gpu/drm/i915/i915_drv.h  |  45 +-
  drivers/gpu/drm/i915/i915_perf.c | 957 ++++++++++++++++++++++++++++++++++++---
  drivers/gpu/drm/i915/i915_reg.h  |  22 +
  drivers/gpu/drm/i915/intel_lrc.c |   2 +
  include/uapi/drm/i915_drm.h      |  19 +-
  5 files changed, 952 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index ffa1fc5eddfd..676b1227067c 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
  	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
  	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
  	 * disabling EU clock gating as required.
  	 */
  	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
  		    size_t *offset);
 
  	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
  	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
  	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
  };
 
  struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
  			struct {
  				struct i915_vma *vma;
  				u8 *vaddr;
+				u32 last_ctx_id;
  				int format;
  				int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
  			} oa_buffer;
 
  			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
  			struct i915_oa_ops ops;
  			const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
  #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
  				 INTEL_DEVID(dev_priv) == 0x5915 || \
  				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
  #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
  				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
  #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
  int i915_perf_open_ioctl(struct drm_device *dev, void *data,
  			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 
  /* i915_gem_evict.c */
  int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +886 -71
index 3277a52ce98e..f2688dee5a03 100644
@@ -196,6 +196,12 @@
 
  #include "i915_drv.h"
  #include "i915_oa_hsw.h"
+#include "i915_oa_bdw.h"
+#include "i915_oa_chv.h"
+#include "i915_oa_sklgt2.h"
+#include "i915_oa_sklgt3.h"
+#include "i915_oa_sklgt4.h"
+#include "i915_oa_bxt.h"
 
  /* HW requires this to be a power of two, between 128k and 16M, though driver
   * is currently generally designed assuming the largest 16M size is used such
@@ -215,7 +221,7 @@
   *
   * Although this can be observed explicitly while copying reports to userspace
   * by checking for a zeroed report-id field in tail reports, we want to account
- * for this earlier, as part of the _oa_buffer_check to avoid lots of redundant
+ * for this earlier, as part of the oa_buffer_check to avoid lots of redundant
   * read() attempts.
   *
   * In effect we define a tail pointer for reading that lags the real tail
@@ -237,7 +243,7 @@
   * indicates that an updated tail pointer is needed.
   *
   * Most of the implementation details for this workaround are in
- * gen7_oa_buffer_check_unlocked() and gen7_appand_oa_reports()
+ * oa_buffer_check_unlocked() and _append_oa_reports()
   *
   * Note for posterity: previously the driver used to define an effective tail
   * pointer that lagged the real pointer by a 'tail margin' measured in bytes
@@ -272,6 +278,13 @@ static u32 i915_perf_stream_paranoid = true;
 
  #define INVALID_CTX_ID 0xffffffff
 
+/* On Gen8+ automatically triggered OA reports include a 'reason' field... */
+#define OAREPORT_REASON_MASK           0x3f
+#define OAREPORT_REASON_SHIFT          19
+#define OAREPORT_REASON_TIMER          (1<<0)
+#define OAREPORT_REASON_CTX_SWITCH     (1<<3)
+#define OAREPORT_REASON_CLK_RATIO      (1<<5)
+
 
  /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
   *
@@ -303,6 +316,13 @@ static struct i915_oa_format hsw_oa_formats[I915_OA_FORMAT_MAX] = {
  	[I915_OA_FORMAT_C4_B8]	    = { 7, 64 },
  };
 
+static struct i915_oa_format gen8_plus_oa_formats[I915_OA_FORMAT_MAX] = {
+	[I915_OA_FORMAT_A12]		    = { 0, 64 },
+	[I915_OA_FORMAT_A12_B8_C8]	    = { 2, 128 },
+	[I915_OA_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 },
+	[I915_OA_FORMAT_C4_B8]		    = { 7, 64 },
+};
+
  #define SAMPLE_OA_REPORT      (1<<0)
 
  /**
@@ -332,8 +352,20 @@ struct perf_open_properties {
  	int oa_period_exponent;
  };
 
+static u32 gen8_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+	return I915_READ(GEN8_OATAILPTR) & GEN8_OATAILPTR_MASK;
+}
+
+static u32 gen7_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+	u32 oastatus1 = I915_READ(GEN7_OASTATUS1);
+
+	return oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+}
+
  /**
- * gen7_oa_buffer_check_unlocked - check for data and update tail ptr state
+ * oa_buffer_check_unlocked - check for data and update tail ptr state
   * @dev_priv: i915 device instance
   *
   * This is either called via fops (for blocking reads in user ctx) or the poll
@@ -356,12 +388,11 @@ struct perf_open_properties {
   *
   * Returns: %true if the OA buffer contains data, else %false
   */
-static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
+static bool oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
  {
  	int report_size = dev_priv->perf.oa.oa_buffer.format_size;
  	unsigned long flags;
  	unsigned int aged_idx;
-	u32 oastatus1;
  	u32 head, hw_tail, aged_tail, aging_tail;
  	u64 now;
 
@@ -381,8 +412,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
  	aged_tail = dev_priv->perf.oa.oa_buffer.tails[aged_idx].offset;
  	aging_tail = dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset;
 
-	oastatus1 = I915_READ(GEN7_OASTATUS1);
-	hw_tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+	hw_tail = dev_priv->perf.oa.ops.oa_hw_tail_read(dev_priv);
 
  	/* The tail pointer increases in 64 byte increments,
  	 * not in report_size steps...
@@ -404,6 +434,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
  	if (aging_tail != INVALID_TAIL_PTR &&
  	    ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
  	     OA_TAIL_MARGIN_NSEC)) {
+
  		aged_idx ^= 1;
  		dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
 
@@ -553,6 +584,287 @@ static int append_oa_sample(struct i915_perf_stream *stream,
   *
   * Returns: 0 on success, negative error code on failure.
   */
+static int gen8_append_oa_reports(struct i915_perf_stream *stream,
+				  char __user *buf,
+				  size_t count,
+				  size_t *offset)
+{
+	struct drm_i915_private *dev_priv = stream->dev_priv;
+	int report_size = dev_priv->perf.oa.oa_buffer.format_size;
+	u8 *oa_buf_base = dev_priv->perf.oa.oa_buffer.vaddr;
+	u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+	u32 mask = (OA_BUFFER_SIZE - 1);
+	size_t start_offset = *offset;
+	unsigned long flags;
+	unsigned int aged_tail_idx;
+	u32 head, tail;
+	u32 taken;
+	int ret = 0;
+
+	if (WARN_ON(!stream->enabled))
+		return -EIO;
+
+	spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	head = dev_priv->perf.oa.oa_buffer.head;
+	aged_tail_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+	tail = dev_priv->perf.oa.oa_buffer.tails[aged_tail_idx].offset;
+
+	spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	/* An invalid tail pointer here means we're still waiting for the poll
+	 * hrtimer callback to give us a pointer
+	 */
+	if (tail == INVALID_TAIL_PTR)
+		return -EAGAIN;
+
+	/* NB: oa_buffer.head/tail include the gtt_offset which we don't want
+	 * while indexing relative to oa_buf_base.
+	 */
+	head -= gtt_offset;
+	tail -= gtt_offset;
+
+	/* An out of bounds or misaligned head or tail pointer implies a driver
+	 * bug since we validate + align the tail pointers we read from the
+	 * hardware and we are in full control of the head pointer which should
+	 * only be incremented by multiples of the report size (notably also
+	 * all a power of two).
+	 */
+	if (WARN_ONCE(head > OA_BUFFER_SIZE || head % report_size ||
+		      tail > OA_BUFFER_SIZE || tail % report_size,
+		      "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
+		      head, tail))
+		return -EIO;
+
+
+	for (/* none */;
+	     (taken = OA_TAKEN(tail, head));
+	     head = (head + report_size) & mask) {
+		u8 *report = oa_buf_base + head;
+		u32 *report32 = (void *)report;
+		u32 ctx_id;
+		u32 reason;
+
+		/* All the report sizes factor neatly into the buffer
+		 * size so we never expect to see a report split
+		 * between the beginning and end of the buffer.
+		 *
+		 * Given the initial alignment check a misalignment
+		 * here would imply a driver bug that would result
+		 * in an overrun.
+		 */
+		if (WARN_ON((OA_BUFFER_SIZE - head) < report_size)) {
+			DRM_ERROR("Spurious OA head ptr: non-integral report offset\n");
+			break;
+		}
+
+		/* The reason field includes flags identifying what
+		 * triggered this specific report (mostly timer
+		 * triggered or e.g. due to a context switch).
+		 *
+		 * This field is never expected to be zero so we can
+		 * check that the report isn't invalid before copying
+		 * it to userspace...
+		 */
+		reason = ((report32[0] >> OAREPORT_REASON_SHIFT) &
+			  OAREPORT_REASON_MASK);
+		if (reason == 0) {
+			if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+				DRM_NOTE("Skipping spurious, invalid OA report\n");
+			continue;
+		}
+
+		/* XXX: Just keep the lower 21 bits for now since I'm not
+		 * entirely sure if the HW touches any of the higher bits in
+		 * this field
+		 */
+		ctx_id = report32[2] & 0x1fffff;
+
+		/* Squash whatever is in the CTX_ID field if it's marked as
+		 * invalid to be sure we avoid false-positive, single-context
+		 * filtering below...
+		 *
+		 * Note: that we don't clear the valid_ctx_bit so userspace can
+		 * understand that the ID has been squashed by the kernel.
+		 */
+		if (!(report32[0] & dev_priv->perf.oa.gen8_valid_ctx_bit))
+			ctx_id = report32[2] = INVALID_CTX_ID;
+
+		/* NB: For Gen 8 the OA unit no longer supports clock gating
+		 * off for a specific context and the kernel can't securely
+		 * stop the counters from updating as system-wide / global
+		 * values.
+		 *
+		 * Automatic reports now include a context ID so reports can be
+		 * filtered on the cpu but it's not worth trying to
+		 * automatically subtract/hide counter progress for other
+		 * contexts while filtering since we can't stop userspace
+		 * issuing MI_REPORT_PERF_COUNT commands which would still
+		 * provide a side-band view of the real values.
+		 *
+		 * To allow userspace (such as Mesa/GL_INTEL_performance_query)
+		 * to normalize counters for a single filtered context then it
+		 * needs be forwarded bookend context-switch reports so that it
+		 * can track switches in between MI_REPORT_PERF_COUNT commands
+		 * and can itself subtract/ignore the progress of counters
+		 * associated with other contexts. Note that the hardware
+		 * automatically triggers reports when switching to a new
+		 * context which are tagged with the ID of the newly active
+		 * context. To avoid the complexity (and likely fragility) of
+		 * reading ahead while parsing reports to try and minimize
+		 * forwarding redundant context switch reports (i.e. between
+		 * other, unrelated contexts) we simply elect to forward them
+		 * all.
+		 *
+		 * We don't rely solely on the reason field to identify context
+		 * switches since it's not-uncommon for periodic samples to
+		 * identify a switch before any 'context switch' report.
+		 */
+		if (!dev_priv->perf.oa.exclusive_stream->ctx ||
+		    dev_priv->perf.oa.specific_ctx_id == ctx_id ||
+		    (dev_priv->perf.oa.oa_buffer.last_ctx_id ==
+		     dev_priv->perf.oa.specific_ctx_id) ||
+		    reason & OAREPORT_REASON_CTX_SWITCH) {
+
+			/* While filtering for a single context we avoid
+			 * leaking the IDs of other contexts.
+			 */
+			if (dev_priv->perf.oa.exclusive_stream->ctx &&
+			    dev_priv->perf.oa.specific_ctx_id != ctx_id) {
+				report32[2] = INVALID_CTX_ID;
+			}
+
+			ret = append_oa_sample(stream, buf, count, offset,
+					       report);
+			if (ret)
+				break;
+
+			dev_priv->perf.oa.oa_buffer.last_ctx_id = ctx_id;
+		}
+
+		/* The above reason field sanity check is based on
+		 * the assumption that the OA buffer is initially
+		 * zeroed and we reset the field after copying so the
+		 * check is still meaningful once old reports start
+		 * being overwritten.
+		 */
+		report32[0] = 0;
+	}
+
+	if (start_offset != *offset) {
+		spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+		/* We removed the gtt_offset for the copy loop above, indexing
+		 * relative to oa_buf_base so put back here...
+		 */
+		head += gtt_offset;
+
+		I915_WRITE(GEN8_OAHEADPTR, head & GEN8_OAHEADPTR_MASK);
+		dev_priv->perf.oa.oa_buffer.head = head;
+
+		spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+	}
+
+	return ret;
+}
+
+/**
+ * gen8_oa_read - copy status records then buffered OA reports
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Checks OA unit status registers and if necessary appends corresponding
+ * status records for userspace (such as for a buffer full condition) and then
+ * initiate appending any buffered OA reports.
+ *
+ * Updates @offset according to the number of bytes successfully copied into
+ * the userspace buffer.
+ *
+ * NB: some data may be successfully copied to the userspace buffer
+ * even if an error is returned, and this is reflected in the
+ * updated @offset.
+ *
+ * Returns: zero on success or a negative error code
+ */
+static int gen8_oa_read(struct i915_perf_stream *stream,
+			char __user *buf,
+			size_t count,
+			size_t *offset)
+{
+	struct drm_i915_private *dev_priv = stream->dev_priv;
+	u32 oastatus;
+	int ret;
+
+	if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
+		return -EIO;
+
+	oastatus = I915_READ(GEN8_OASTATUS);
+
+	/* We treat OABUFFER_OVERFLOW as a significant error:
+	 *
+	 * Although theoretically we could handle this more gracefully
+	 * sometimes, some Gens don't correctly suppress certain
+	 * automatically triggered reports in this condition and so we
+	 * have to assume that old reports are now being trampled
+	 * over.
+	 *
+	 * Considering how we don't currently give userspace control
+	 * over the OA buffer size and always configure a large 16MB
+	 * buffer, then a buffer overflow does anyway likely indicate
+	 * that something has gone quite badly wrong.
+	 */
+	if (oastatus & GEN8_OASTATUS_OABUFFER_OVERFLOW) {
+		ret = append_oa_status(stream, buf, count, offset,
+				       DRM_I915_PERF_RECORD_OA_BUFFER_LOST);
+		if (ret)
+			return ret;
+
+		DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+			  dev_priv->perf.oa.period_exponent);
+
+		dev_priv->perf.oa.ops.oa_disable(dev_priv);
+		dev_priv->perf.oa.ops.oa_enable(dev_priv);
+
+		/* Note: .oa_enable() is expected to re-init the oabuffer
+		 * and reset GEN8_OASTATUS for us
+		 */
+		oastatus = I915_READ(GEN8_OASTATUS);
+	}
+
+	if (oastatus & GEN8_OASTATUS_REPORT_LOST) {
+		ret = append_oa_status(stream, buf, count, offset,
+				       DRM_I915_PERF_RECORD_OA_REPORT_LOST);
+		if (ret)
+			return ret;
+		I915_WRITE(GEN8_OASTATUS,
+			   oastatus & ~GEN8_OASTATUS_REPORT_LOST);
+	}
+
+	return gen8_append_oa_reports(stream, buf, count, offset);
+}
+
+/**
+ * Copies all buffered OA reports into userspace read() buffer.
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Notably any error condition resulting in a short read (-%ENOSPC or
+ * -%EFAULT) will be returned even though one or more records may
+ * have been successfully copied. In this case it's up to the caller
+ * to decide if the error should be squashed before returning to
+ * userspace.
+ *
+ * Note: reports are consumed from the head, and appended to the
+ * tail, so the tail chases the head?... If you think that's mad
+ * and back-to-front you're not alone, but this follows the
+ * Gen PRM naming convention.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
  static int gen7_append_oa_reports(struct i915_perf_stream *stream,
  				  char __user *buf,
  				  size_t count,
@@ -733,7 +1045,8 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
  		if (ret)
  			return ret;
 
-		DRM_DEBUG("OA buffer overflow: force restart\n");
+		DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+			  dev_priv->perf.oa.period_exponent);
 
  		dev_priv->perf.oa.ops.oa_disable(dev_priv);
  		dev_priv->perf.oa.ops.oa_enable(dev_priv);
@@ -776,7 +1089,7 @@ static int i915_oa_wait_unlocked(struct i915_perf_stream *stream)
  		return -EIO;
 
  	return wait_event_interruptible(dev_priv->perf.oa.poll_wq,
-					dev_priv->perf.oa.ops.oa_buffer_check(dev_priv));
+					oa_buffer_check_unlocked(dev_priv));
  }
 
  /**
@@ -833,33 +1146,37 @@ static int i915_oa_read(struct i915_perf_stream *stream,
  static int oa_get_render_ctx_id(struct i915_perf_stream *stream)
  {
  	struct drm_i915_private *dev_priv = stream->dev_priv;
-	struct intel_engine_cs *engine = dev_priv->engine[RCS];
-	int ret;
 
-	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
-	if (ret)
-		return ret;
+	if (i915.enable_execlists)
+		dev_priv->perf.oa.specific_ctx_id = stream->ctx->hw_id;
+	else {
+		struct intel_engine_cs *engine = dev_priv->engine[RCS];
+		int ret;
 
-	/* As the ID is the gtt offset of the context's vma we pin
-	 * the vma to ensure the ID remains fixed.
-	 *
-	 * NB: implied RCS engine...
-	 */
-	ret = engine->context_pin(engine, stream->ctx);
-	if (ret)
-		goto unlock;
+		ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+		if (ret)
+			return ret;
 
-	/* Explicitly track the ID (instead of calling i915_ggtt_offset()
-	 * on the fly) considering the difference with gen8+ and
-	 * execlists
-	 */
-	dev_priv->perf.oa.specific_ctx_id =
-		i915_ggtt_offset(stream->ctx->engine[engine->id].state);
+		/* As the ID is the gtt offset of the context's vma we pin
+		 * the vma to ensure the ID remains fixed.
+		 */
+		ret = engine->context_pin(engine, stream->ctx);
+		if (ret) {
+			mutex_unlock(&dev_priv->drm.struct_mutex);
+			return ret;
+		}
 
-unlock:
-	mutex_unlock(&dev_priv->drm.struct_mutex);
+		/* Explicitly track the ID (instead of calling
+		 * i915_ggtt_offset() on the fly) considering the difference
+		 * with gen8+ and execlists
+		 */
+		dev_priv->perf.oa.specific_ctx_id =
+			i915_ggtt_offset(stream->ctx->engine[engine->id].state);
 
-	return ret;
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+	}
+
+	return 0;
  }
 
  /**
@@ -874,12 +1191,16 @@ static void oa_put_render_ctx_id(struct i915_perf_stream *stream)
  	struct drm_i915_private *dev_priv = stream->dev_priv;
  	struct intel_engine_cs *engine = dev_priv->engine[RCS];
 
-	mutex_lock(&dev_priv->drm.struct_mutex);
+	if (i915.enable_execlists) {
+		dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+	} else {
+		mutex_lock(&dev_priv->drm.struct_mutex);
 
-	dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
-	engine->context_unpin(engine, stream->ctx);
+		dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+		engine->context_unpin(engine, stream->ctx);
 
-	mutex_unlock(&dev_priv->drm.struct_mutex);
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+	}
  }
 
  static void
@@ -969,6 +1290,61 @@ static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
  	dev_priv->perf.oa.pollin = false;
  }
 
+static void gen8_init_oa_buffer(struct drm_i915_private *dev_priv)
+{
+	u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+	unsigned long flags;
+
+	spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	I915_WRITE(GEN8_OASTATUS, 0);
+	I915_WRITE(GEN8_OAHEADPTR, gtt_offset);
+	dev_priv->perf.oa.oa_buffer.head = gtt_offset;
+
+	I915_WRITE(GEN8_OABUFFER_UDW, 0);
+
+	/* PRM says:
+	 *
+	 *  "This MMIO must be set before the OATAILPTR
+	 *  register and after the OAHEADPTR register. This is
+	 *  to enable proper functionality of the overflow
+	 *  bit."
+	 */
+	I915_WRITE(GEN8_OABUFFER, gtt_offset |
+		   OABUFFER_SIZE_16M | OA_MEM_SELECT_GGTT);
+	I915_WRITE(GEN8_OATAILPTR, gtt_offset & GEN8_OATAILPTR_MASK);
+
+	/* Mark that we need updated tail pointers to read from... */
+	dev_priv->perf.oa.oa_buffer.tails[0].offset = INVALID_TAIL_PTR;
+	dev_priv->perf.oa.oa_buffer.tails[1].offset = INVALID_TAIL_PTR;
+
+	/* Reset state used to recognise context switches, affecting which
+	 * reports we will forward to userspace while filtering for a single
+	 * context.
+	 */
+	dev_priv->perf.oa.oa_buffer.last_ctx_id = INVALID_CTX_ID;
+
+	spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	/* NB: although the OA buffer will initially be allocated
+	 * zeroed via shmfs (and so this memset is redundant when
+	 * first allocating), we may re-init the OA buffer, either
+	 * when re-enabling a stream or in error/reset paths.
+	 *
+	 * The reason we clear the buffer for each re-init is for the
+	 * sanity check in gen8_append_oa_reports() that looks at the
+	 * reason field to make sure it's non-zero which relies on
+	 * the assumption that new reports are being written to zeroed
+	 * memory...
+	 */
+	memset(dev_priv->perf.oa.oa_buffer.vaddr, 0, OA_BUFFER_SIZE);
+
+	/* Maybe make ->pollin per-stream state if we support multiple
+	 * concurrent streams in the future.
+	 */
+	dev_priv->perf.oa.pollin = false;
+}
+
  static int alloc_oa_buffer(struct drm_i915_private *dev_priv)
  {
  	struct drm_i915_gem_object *bo;
@@ -1113,6 +1489,280 @@ static void hsw_disable_metric_set(struct drm_i915_private *dev_priv)
  				      ~GT_NOA_ENABLE));
  }
 
+/*
+ * From Broadwell PRM, 3D-Media-GPGPU -> Register State Context
+ *
+ * MMIO reads or writes to any of the registers listed in the
+ * “Register State Context image” subsections through HOST/IA
+ * MMIO interface for debug purposes must follow the steps below:
+ *
+ * - SW should set the Force Wakeup bit to prevent GT from entering C6.
+ * - Write 0x2050[31:0] = 0x00010001 (disable sequence).
+ * - Disable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010001).
+ * - BDW:  Poll/Wait for register bits of 0x22AC[6:0] turn to 0x30 value.
+ * - SKL+: Poll/Wait for register bits of 0x22A4[6:0] turn to 0x30 value.
+ * - Read/Write to desired MMIO registers.
+ * - Enable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010000).
+ * - Force Wakeup bit should be reset to enable C6 entry.
+ *
+ * XXX: don't nest or overlap calls to this API, it has no ref
+ * counting to track how many entities require the RCS to be
+ * blocked from being idle.
+ */
+static int gen8_begin_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+	i915_reg_t fsm_reg = INTEL_GEN(dev_priv) > 8 ?
+		GEN9_RCS_FE_FSM2 : GEN6_RCS_PWR_FSM;
+	int ret = 0;
+
+	/* There's only no active context while idle in execlist mode
+	 * (though we shouldn't be using this in any other case)
+	 */
+	if (WARN_ON(!i915.enable_execlists))
+		return -EINVAL;
+
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_RENDER);
+
+	I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+		   _MASKED_BIT_ENABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+
+	/* Note: we don't currently have a good handle on the maximum
+	 * latency for this wake up so while we only need to hold rcs
+	 * busy from process context we at least keep the waiting
+	 * interruptible...
+	 */
+	ret = intel_wait_for_register(dev_priv, fsm_reg, 0x3f, 0x30, 50);
+	if (ret == -ETIMEDOUT)
+		DRM_ERROR("Timeout waiting to be able to begin per-ctx mmio\n");
+
+	if (ret)
+		intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+
+	return ret;
+}
+
+static void gen8_end_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+	if (WARN_ON(!i915.enable_execlists))
+		return;
+
+	I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+		   _MASKED_BIT_DISABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+}
+
+/* NB: It must always remain pointer safe to run this even if the OA unit
+ * has been disabled.
+ *
+ * It's fine to put out-of-date values into these per-context registers
+ * in the case that the OA unit has been disabled.
+ */
+static void gen8_update_reg_state_unlocked(struct i915_gem_context *ctx,
+					   u32 *reg_state)
+{
+	struct drm_i915_private *dev_priv = ctx->i915;
+	const struct i915_oa_reg *flex_regs = dev_priv->perf.oa.flex_regs;
+	int n_flex_regs = dev_priv->perf.oa.flex_regs_len;
+	u32 ctx_oactxctrl = dev_priv->perf.oa.ctx_oactxctrl_offset;
+	u32 ctx_flexeu0 = dev_priv->perf.oa.ctx_flexeu0_offset;
+	/* The MMIO offsets for Flex EU registers aren't contiguous */
+	u32 flex_mmio[] = {
+		i915_mmio_reg_offset(EU_PERF_CNTL0),
+		i915_mmio_reg_offset(EU_PERF_CNTL1),
+		i915_mmio_reg_offset(EU_PERF_CNTL2),
+		i915_mmio_reg_offset(EU_PERF_CNTL3),
+		i915_mmio_reg_offset(EU_PERF_CNTL4),
+		i915_mmio_reg_offset(EU_PERF_CNTL5),
+		i915_mmio_reg_offset(EU_PERF_CNTL6),
+	};
+	int i;
+
+	reg_state[ctx_oactxctrl] = i915_mmio_reg_offset(GEN8_OACTXCONTROL);
+	reg_state[ctx_oactxctrl+1] = (dev_priv->perf.oa.period_exponent <<
+				      GEN8_OA_TIMER_PERIOD_SHIFT) |
+				     (dev_priv->perf.oa.periodic ?
+				      GEN8_OA_TIMER_ENABLE : 0) |
+				     GEN8_OA_COUNTER_RESUME;
+
+	for (i = 0; i < ARRAY_SIZE(flex_mmio); i++) {
+		u32 state_offset = ctx_flexeu0 + i * 2;
+		u32 mmio = flex_mmio[i];
+
+		/* This arbitrary default will select the 'EU FPU0 Pipeline
+		 * Active' event. In the future it's anticipated that there
+		 * will be an explicit 'No Event' we can select, but not yet...
+		 */
+		u32 value = 0;
+		int j;
+
+		for (j = 0; j < n_flex_regs; j++) {
+			if (i915_mmio_reg_offset(flex_regs[j].addr) == mmio) {
+				value = flex_regs[j].value;
+				break;
+			}
+		}
+
+		reg_state[state_offset] = mmio;
+		reg_state[state_offset+1] = value;
+	}
+}
+
+/* Manages updating the per-context aspects of the OA stream
+ * configuration across all contexts.
+ *
+ * The awkward consideration here is that OACTXCONTROL controls the
+ * exponent for periodic sampling which is primarily used for system
+ * wide profiling where we'd like a consistent sampling period even in
+ * the face of context switches.
+ *
+ * Our approach of updating the register state context (as opposed to
+ * say using a workaround batch buffer) ensures that the hardware
+ * won't automatically reload an out-of-date timer exponent even
+ * transiently before a WA BB could be parsed.
+ *
+ * This function needs to:
+ * - Ensure the currently running context's per-context OA state is
+ *   updated
+ * - Ensure that all existing contexts will have the correct per-context
+ *   OA state if they are scheduled for use.
+ * - Ensure any new contexts will be initialized with the correct
+ *   per-context OA state.
+ *
+ * Note: it's only the RCS/Render context that has any OA state.
+ */
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{
+	struct i915_gem_context *ctx;
+	int ret;
+
+	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+	if (ret)
+		return ret;
+
+	/* Switch away from any user context.  */
+	ret = i915_gem_switch_to_kernel_context(dev_priv);
+	if (ret)
+		return ret;
+
+	/* The OA register config is setup through the context image. This image
+	 * might be written to by the GPU on context switch (in particular on
+	 * lite-restore). This means we can't safely update a context's image,
+	 * if this context is scheduled/submitted to run on the GPU.
+	 *
+	 * We could emit the OA register config through the batch buffer but
+	 * this might leave small interval of time where the OA unit is
+	 * configured at an invalid sampling period.
+	 *
+	 * So far the best way to work around this issue seems to be draining
+	 * the GPU from any submitted work.
+	 */
+	ret = i915_gem_wait_for_idle(dev_priv,
+				     I915_WAIT_INTERRUPTIBLE |
+				     I915_WAIT_LOCKED);
+	if (ret) {
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+		return ret;
+	}
+
+	/* Update all contexts now that we've stalled the submission. */
+	list_for_each_entry(ctx, &dev_priv->context_list, link) {
+		if (!ctx->engine[RCS].initialised)
+			continue;
+
+		gen8_update_reg_state_unlocked(ctx,
+					       ctx->engine[RCS].lrc_reg_state);
+	}
+
+	mutex_unlock(&dev_priv->drm.struct_mutex);
+
+	/* Now update the current context.
+	 *
+	 * Note: Using MMIO to update per-context registers requires
+	 * some extra care...
+	 */
+	ret = gen8_begin_ctx_mmio(dev_priv);
+	if (ret) {
+		DRM_ERROR("Failed to bring RCS out of idle to update current ctx OA state\n");
+		return ret;
+	}
+
+	I915_WRITE(GEN8_OACTXCONTROL, ((dev_priv->perf.oa.period_exponent <<
+					GEN8_OA_TIMER_PERIOD_SHIFT) |
+				      (dev_priv->perf.oa.periodic ?
+				       GEN8_OA_TIMER_ENABLE : 0) |
+				      GEN8_OA_COUNTER_RESUME));
+
+	config_oa_regs(dev_priv, dev_priv->perf.oa.flex_regs,
+			dev_priv->perf.oa.flex_regs_len);
+
+	gen8_end_ctx_mmio(dev_priv);
+
+	return 0;
+}
+
+static int gen8_enable_metric_set(struct drm_i915_private *dev_priv)
+{
+	int ret = dev_priv->perf.oa.ops.select_metric_set(dev_priv);
+
+	if (ret)
+		return ret;
+
+	/* We disable slice/unslice clock ratio change reports on SKL since
+	 * they are too noisy. The HW generates a lot of redundant reports
+	 * where the ratio hasn't really changed causing a lot of redundant
+	 * work to processes and increasing the chances we'll hit buffer
+	 * overruns.
+	 *
+	 * Although we don't currently use the 'disable overrun' OABUFFER
+	 * feature it's worth noting that clock ratio reports have to be
+	 * disabled before considering to use that feature since the HW doesn't
+	 * correctly block these reports.
+	 *
+	 * Currently none of the high-level metrics we have depend on knowing
+	 * this ratio to normalize.
+	 *
+	 * Note: This register is not power context saved and restored, but
+	 * that's OK considering that we disable RC6 while the OA unit is
+	 * enabled.
+	 *
+	 * The _INCLUDE_CLK_RATIO bit allows the slice/unslice frequency to
+	 * be read back from automatically triggered reports, as part of the
+	 * RPT_ID field.
+	 */
+	if (IS_SKYLAKE(dev_priv) || IS_BROXTON(dev_priv)) {
+		I915_WRITE(GEN8_OA_DEBUG,
+			   _MASKED_BIT_ENABLE(GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS |
+					      GEN9_OA_DEBUG_INCLUDE_CLK_RATIO));
+	}
+
+	I915_WRITE(GDT_CHICKEN_BITS, 0xA0);
+	config_oa_regs(dev_priv, dev_priv->perf.oa.mux_regs,
+		       dev_priv->perf.oa.mux_regs_len);
+	I915_WRITE(GDT_CHICKEN_BITS, 0x80);
+
+	/* It takes a fairly long time for a new MUX configuration to
+	 * be be applied after these register writes. This delay
+	 * duration is take from Haswell (derived empirically based on
+	 * the render_basic config) but hopefully it covers the
+	 * maximum configuration latency for Gen8+ too...
+	 */
+	usleep_range(15000, 20000);
+
+	config_oa_regs(dev_priv, dev_priv->perf.oa.b_counter_regs,
+		       dev_priv->perf.oa.b_counter_regs_len);
+
+	ret = gen8_configure_all_contexts(dev_priv);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void gen8_disable_metric_set(struct drm_i915_private *dev_priv)
+{
+	/* NOP */
+}
+
  static void gen7_update_oacontrol_locked(struct drm_i915_private *dev_priv)
  {
  	lockdep_assert_held(&dev_priv->perf.hook_lock);
@@ -1157,6 +1807,29 @@ static void gen7_oa_enable(struct drm_i915_private *dev_priv)
  	spin_unlock_irqrestore(&dev_priv->perf.hook_lock, flags);
  }
 
+static void gen8_oa_enable(struct drm_i915_private *dev_priv)
+{
+	u32 report_format = dev_priv->perf.oa.oa_buffer.format;
+
+	/* Reset buf pointers so we don't forward reports from before now.
+	 *
+	 * Think carefully if considering trying to avoid this, since it
+	 * also ensures status flags and the buffer itself are cleared
+	 * in error paths, and we have checks for invalid reports based
+	 * on the assumption that certain fields are written to zeroed
+	 * memory which this helps maintains.
+	 */
+	gen8_init_oa_buffer(dev_priv);
+
+	/* Note: we don't rely on the hardware to perform single context
+	 * filtering and instead filter on the cpu based on the context-id
+	 * field of reports
+	 */
+	I915_WRITE(GEN8_OACONTROL, (report_format <<
+				    GEN8_OA_REPORT_FORMAT_SHIFT) |
+				   GEN8_OA_COUNTER_ENABLE);
+}
+
  /**
   * i915_oa_stream_enable - handle `I915_PERF_IOCTL_ENABLE` for OA stream
   * @stream: An i915 perf stream opened for OA metrics
@@ -1183,6 +1856,11 @@ static void gen7_oa_disable(struct drm_i915_private *dev_priv)
  	I915_WRITE(GEN7_OACONTROL, 0);
  }
 
+static void gen8_oa_disable(struct drm_i915_private *dev_priv)
+{
+	I915_WRITE(GEN8_OACONTROL, 0);
+}
+
  /**
   * i915_oa_stream_disable - handle `I915_PERF_IOCTL_DISABLE` for OA stream
   * @stream: An i915 perf stream opened for OA metrics
@@ -1361,6 +2039,21 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream,
  	return ret;
  }
 
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    u32 *reg_state)
+{
+	struct drm_i915_private *dev_priv = engine->i915;
+
+	if (engine->id != RCS)
+		return;
+
+	if (!dev_priv->perf.initialized)
+		return;
+
+	gen8_update_reg_state_unlocked(ctx, reg_state);
+}
+
  /**
   * i915_perf_read_locked - &i915_perf_stream_ops->read with error normalisation
   * @stream: An i915 perf stream
@@ -1486,7 +2179,7 @@ static enum hrtimer_restart oa_poll_check_timer_cb(struct hrtimer *hrtimer)
  		container_of(hrtimer, typeof(*dev_priv),
  			     perf.oa.poll_check_timer);
 
-	if (dev_priv->perf.oa.ops.oa_buffer_check(dev_priv)) {
+	if (oa_buffer_check_unlocked(dev_priv)) {
  		dev_priv->perf.oa.pollin = true;
  		wake_up(&dev_priv->perf.oa.poll_wq);
  	}
@@ -1775,6 +2468,7 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
  	struct i915_gem_context *specific_ctx = NULL;
  	struct i915_perf_stream *stream = NULL;
  	unsigned long f_flags = 0;
+	bool privileged_op = true;
  	int stream_fd;
  	int ret;
 
@@ -1792,12 +2486,28 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
  		}
  	}
 
+	/* On Haswell the OA unit supports clock gating off for a specific
+	 * context and in this mode there's no visibility of metrics for the
+	 * rest of the system, which we consider acceptable for a
+	 * non-privileged client.
+	 *
+	 * For Gen8+ the OA unit no longer supports clock gating off for a
+	 * specific context and the kernel can't securely stop the counters
+	 * from updating as system-wide / global values. Even though we can
+	 * filter reports based on the included context ID we can't block
+	 * clients from seeing the raw / global counter values via
+	 * MI_REPORT_PERF_COUNT commands and so consider it a privileged op to
+	 * enable the OA unit by default.
+	 */
+	if (IS_HASWELL(dev_priv) && specific_ctx)
+		privileged_op = false;
+
  	/* Similar to perf's kernel.perf_paranoid_cpu sysctl option
  	 * we check a dev.i915.perf_stream_paranoid sysctl option
  	 * to determine if it's ok to access system wide OA counters
  	 * without CAP_SYS_ADMIN privileges.
  	 */
-	if (!specific_ctx &&
+	if (privileged_op &&
  	    i915_perf_stream_paranoid && !capable(CAP_SYS_ADMIN)) {
  		DRM_DEBUG("Insufficient privileges to open system-wide i915 perf stream\n");
  		ret = -EACCES;
@@ -2069,9 +2779,6 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data,
   */
  void i915_perf_register(struct drm_i915_private *dev_priv)
  {
-	if (!IS_HASWELL(dev_priv))
-		return;
-
  	if (!dev_priv->perf.initialized)
  		return;
 
@@ -2087,11 +2794,38 @@ void i915_perf_register(struct drm_i915_private *dev_priv)
  	if (!dev_priv->perf.metrics_kobj)
  		goto exit;
 
-	if (i915_perf_register_sysfs_hsw(dev_priv)) {
-		kobject_put(dev_priv->perf.metrics_kobj);
-		dev_priv->perf.metrics_kobj = NULL;
+	if (IS_HASWELL(dev_priv)) {
+		if (i915_perf_register_sysfs_hsw(dev_priv))
+			goto sysfs_error;
+	} else if (IS_BROADWELL(dev_priv)) {
+		if (i915_perf_register_sysfs_bdw(dev_priv))
+			goto sysfs_error;
+	} else if (IS_CHERRYVIEW(dev_priv)) {
+		if (i915_perf_register_sysfs_chv(dev_priv))
+			goto sysfs_error;
+	} else if (IS_SKYLAKE(dev_priv)) {
+		if (IS_SKL_GT2(dev_priv)) {
+			if (i915_perf_register_sysfs_sklgt2(dev_priv))
+				goto sysfs_error;
+		} else if (IS_SKL_GT3(dev_priv)) {
+			if (i915_perf_register_sysfs_sklgt3(dev_priv))
+				goto sysfs_error;
+		} else if (IS_SKL_GT4(dev_priv)) {
+			if (i915_perf_register_sysfs_sklgt4(dev_priv))
+				goto sysfs_error;
+		} else
+			goto sysfs_error;
+	} else if (IS_BROXTON(dev_priv)) {
+		if (i915_perf_register_sysfs_bxt(dev_priv))
+			goto sysfs_error;
  	}
 
+	goto exit;
+
+sysfs_error:
+	kobject_put(dev_priv->perf.metrics_kobj);
+	dev_priv->perf.metrics_kobj = NULL;
+
  exit:
  	mutex_unlock(&dev_priv->perf.lock);
  }
@@ -2107,13 +2841,24 @@ void i915_perf_register(struct drm_i915_private *dev_priv)
   */
  void i915_perf_unregister(struct drm_i915_private *dev_priv)
  {
-	if (!IS_HASWELL(dev_priv))
-		return;
-
  	if (!dev_priv->perf.metrics_kobj)
  		return;
 
-	i915_perf_unregister_sysfs_hsw(dev_priv);
+        if (IS_HASWELL(dev_priv))
+                i915_perf_unregister_sysfs_hsw(dev_priv);
+        else if (IS_BROADWELL(dev_priv))
+                i915_perf_unregister_sysfs_bdw(dev_priv);
+        else if (IS_CHERRYVIEW(dev_priv))
+                i915_perf_unregister_sysfs_chv(dev_priv);
+        else if (IS_SKYLAKE(dev_priv)) {
+		if (IS_SKL_GT2(dev_priv))
+			i915_perf_unregister_sysfs_sklgt2(dev_priv);
+		else if (IS_SKL_GT3(dev_priv))
+			i915_perf_unregister_sysfs_sklgt3(dev_priv);
+		else if (IS_SKL_GT4(dev_priv))
+			i915_perf_unregister_sysfs_sklgt4(dev_priv);
+	} else if (IS_BROXTON(dev_priv))
+                i915_perf_unregister_sysfs_bxt(dev_priv);
 
  	kobject_put(dev_priv->perf.metrics_kobj);
  	dev_priv->perf.metrics_kobj = NULL;
@@ -2172,36 +2917,105 @@ static struct ctl_table dev_root[] = {
   */
  void i915_perf_init(struct drm_i915_private *dev_priv)
  {
-	if (!IS_HASWELL(dev_priv))
-		return;
-
-	hrtimer_init(&dev_priv->perf.oa.poll_check_timer,
-		     CLOCK_MONOTONIC, HRTIMER_MODE_REL);
-	dev_priv->perf.oa.poll_check_timer.function = oa_poll_check_timer_cb;
-	init_waitqueue_head(&dev_priv->perf.oa.poll_wq);
+	dev_priv->perf.oa.n_builtin_sets = 0;
+
+	if (IS_HASWELL(dev_priv)) {
+		dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
+		dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
+		dev_priv->perf.oa.ops.disable_metric_set = hsw_disable_metric_set;
+		dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
+		dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
+		dev_priv->perf.oa.ops.read = gen7_oa_read;
+		dev_priv->perf.oa.ops.oa_hw_tail_read =
+			gen7_oa_hw_tail_read;
+
+		dev_priv->perf.oa.oa_formats = hsw_oa_formats;
+
+		dev_priv->perf.oa.n_builtin_sets =
+			i915_oa_n_builtin_metric_sets_hsw;
+	} else if (i915.enable_execlists) {
+		/* Note: that although we could theoretically also support the
+		 * legacy ringbuffer mode on BDW (and earlier iterations of
+		 * this driver, before upstreaming did this) it didn't seem
+		 * worth the complexity to maintain now that BDW+ enable
+		 * execlist mode by default.
+		 */
 
-	INIT_LIST_HEAD(&dev_priv->perf.streams);
-	mutex_init(&dev_priv->perf.lock);
-	spin_lock_init(&dev_priv->perf.hook_lock);
-	spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
+		if (IS_GEN8(dev_priv)) {
+			dev_priv->perf.oa.ctx_oactxctrl_offset = 0x120;
+			dev_priv->perf.oa.ctx_flexeu0_offset = 0x2ce;
+			dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<25);
+
+			if (IS_BROADWELL(dev_priv)) {
+				dev_priv->perf.oa.n_builtin_sets =
+					i915_oa_n_builtin_metric_sets_bdw;
+				dev_priv->perf.oa.ops.select_metric_set =
+					i915_oa_select_metric_set_bdw;
+			} else if (IS_CHERRYVIEW(dev_priv)) {
+				dev_priv->perf.oa.n_builtin_sets =
+					i915_oa_n_builtin_metric_sets_chv;
+				dev_priv->perf.oa.ops.select_metric_set =
+					i915_oa_select_metric_set_chv;
+			}
+		} else if (IS_GEN9(dev_priv)) {
+			dev_priv->perf.oa.ctx_oactxctrl_offset = 0x128;
+			dev_priv->perf.oa.ctx_flexeu0_offset = 0x3de;
+			dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<16);
+
+			if (IS_SKL_GT2(dev_priv)) {
+				dev_priv->perf.oa.n_builtin_sets =
+					i915_oa_n_builtin_metric_sets_sklgt2;
+				dev_priv->perf.oa.ops.select_metric_set =
+					i915_oa_select_metric_set_sklgt2;
+			} else if (IS_SKL_GT3(dev_priv)) {
+				dev_priv->perf.oa.n_builtin_sets =
+					i915_oa_n_builtin_metric_sets_sklgt3;
+				dev_priv->perf.oa.ops.select_metric_set =
+					i915_oa_select_metric_set_sklgt3;
+			} else if (IS_SKL_GT4(dev_priv)) {
+				dev_priv->perf.oa.n_builtin_sets =
+					i915_oa_n_builtin_metric_sets_sklgt4;
+				dev_priv->perf.oa.ops.select_metric_set =
+					i915_oa_select_metric_set_sklgt4;
+			} else if (IS_BROXTON(dev_priv)) {
+				dev_priv->perf.oa.n_builtin_sets =
+					i915_oa_n_builtin_metric_sets_bxt;
+				dev_priv->perf.oa.ops.select_metric_set =
+					i915_oa_select_metric_set_bxt;
+			}
+		}
 
-	dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
-	dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
-	dev_priv->perf.oa.ops.disable_metric_set = hsw_disable_metric_set;
-	dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
-	dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
-	dev_priv->perf.oa.ops.read = gen7_oa_read;
-	dev_priv->perf.oa.ops.oa_buffer_check =
-		gen7_oa_buffer_check_unlocked;
+		if (dev_priv->perf.oa.n_builtin_sets) {
+			dev_priv->perf.oa.ops.init_oa_buffer = gen8_init_oa_buffer;
+			dev_priv->perf.oa.ops.enable_metric_set =
+				gen8_enable_metric_set;
+			dev_priv->perf.oa.ops.disable_metric_set =
+				gen8_disable_metric_set;
+			dev_priv->perf.oa.ops.oa_enable = gen8_oa_enable;
+			dev_priv->perf.oa.ops.oa_disable = gen8_oa_disable;
+			dev_priv->perf.oa.ops.read = gen8_oa_read;
+			dev_priv->perf.oa.ops.oa_hw_tail_read =
+				gen8_oa_hw_tail_read;
+
+			dev_priv->perf.oa.oa_formats = gen8_plus_oa_formats;
+		}
+	}
 
-	dev_priv->perf.oa.oa_formats = hsw_oa_formats;
+	if (dev_priv->perf.oa.n_builtin_sets) {
+		hrtimer_init(&dev_priv->perf.oa.poll_check_timer,
+				CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+		dev_priv->perf.oa.poll_check_timer.function = oa_poll_check_timer_cb;
+		init_waitqueue_head(&dev_priv->perf.oa.poll_wq);
 
-	dev_priv->perf.oa.n_builtin_sets =
-		i915_oa_n_builtin_metric_sets_hsw;
+		INIT_LIST_HEAD(&dev_priv->perf.streams);
+		mutex_init(&dev_priv->perf.lock);
+		spin_lock_init(&dev_priv->perf.hook_lock);
+		spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
 
-	dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
+		dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
 
-	dev_priv->perf.initialized = true;
+		dev_priv->perf.initialized = true;
+	}
  }
 
  /**
@@ -2216,5 +3030,6 @@ void i915_perf_fini(struct drm_i915_private *dev_priv)
  	unregister_sysctl_table(dev_priv->perf.sysctl_header);
 
  	memset(&dev_priv->perf.oa.ops, 0, sizeof(dev_priv->perf.oa.ops));
+
  	dev_priv->perf.initialized = false;
  }
drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
  #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
  #define GEN8_OACONTROL _MMIO(0x2B00)
  #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
  #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
  #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
  #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
  #define GEN8_OABUFFER _MMIO(0x2b14)
 
  #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
  #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
  #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
  #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
  #define OABUFFER_SIZE_128K  (0<<3)
  #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
  #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
  #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
  #define GDT_CHICKEN_BITS    _MMIO(0x9840)
  #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
  #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
  #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
  /* Fuse readout registers for GT */
  #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
  #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 7df278fe492e..351e231a02f4 100644
@@ -1879,6 +1879,8 @@ static void execlists_init_reg_state(u32 *regs,
  		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
  		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
  			make_rpcs(dev_priv));
+
+		i915_oa_init_reg_state(engine, ctx, regs);
  	}
  }
 
include/uapi/drm/i915_drm.h +12 -8
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
  };
 
  enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
  	I915_OA_FORMAT_MAX	    /* non-ABI */
  };
--
Quoted 2 lines

2.11.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#34 lore raw

Re: [PATCH v6 12/15] drm/i915/perf: Add OA unit support for Gen 8+

In reply to Lionel Landwerlin
Quoted 873 lines · Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

---
 drivers/gpu/drm/i915/i915_drv.h  |  45 +-
 drivers/gpu/drm/i915/i915_perf.c | 957 ++++++++++++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_reg.h  |  22 +
 drivers/gpu/drm/i915/intel_lrc.c |   2 +
 include/uapi/drm/i915_drm.h      |  19 +-
 5 files changed, 952 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index ffa1fc5eddfd..676b1227067c 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
        void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
        /**
-        * @enable_metric_set: Applies any MUX configuration to set up the
-        * Boolean and Custom (B/C) counters that are part of the counter
-        * reports being sampled. May apply system constraints such as
+        * @select_metric_set: The auto generated code that checks whether a
+        * requested OA config is applicable to the system and if so sets up
+        * the mux, oa and flex eu register config pointers according to the
+        * current dev_priv->perf.oa.metrics_set.
+        */
+       int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+       /**
+        * @enable_metric_set: Selects and applies any MUX configuration to set
+        * up the Boolean and Custom (B/C) counters that are part of the
+        * counter reports being sampled. May apply system constraints such as
         * disabling EU clock gating as required.
         */
        int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
                    size_t *offset);
 
        /**
-        * @oa_buffer_check: Check for OA buffer data + update tail
-        *
-        * This is either called via fops or the poll check hrtimer (atomic
-        * ctx) without any locks taken.
+        * @oa_hw_tail_read: read the OA tail pointer register
         *
-        * It's safe to read OA config state here unlocked, assuming that this
-        * is only called while the stream is enabled, while the global OA
-        * configuration can't be modified.
-        *
-        * Efficiency is more important than avoiding some false positives
-        * here, which will be handled gracefully - likely resulting in an
-        * %EAGAIN error for userspace.
+        * In particular this enables us to share all the fiddly code for
+        * handling the OA unit tail pointer race that affects multiple
+        * generations.
         */
-       bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+       u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
                        struct {
                                struct i915_vma *vma;
                                u8 *vaddr;
+                               u32 last_ctx_id;
                                int format;
                                int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
                        } oa_buffer;
 
                        u32 gen7_latched_oastatus1;
+                       u32 ctx_oactxctrl_offset;
+                       u32 ctx_flexeu0_offset;
+
+                       /* The RPT_ID/reason field for Gen8+ includes a bit
+                        * to determine if the CTX ID in the report is valid
+                        * but the specific bit differs between Gen 8 and 9
+                        */
+                       u32 gen8_valid_ctx_bit;
 
                        struct i915_oa_ops ops;
                        const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)   (INTEL_DEVID(dev_priv) == 0x590E || \
                                 INTEL_DEVID(dev_priv) == 0x5915 || \
                                 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)   (IS_SKYLAKE(dev_priv) && \
+                                (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)   (IS_SKYLAKE(dev_priv) && \
                                 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)   (IS_SKYLAKE(dev_priv) && \
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
                         struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+                           struct i915_gem_context *ctx,
+                           uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +554 -35
index 3277a52ce98e..f2688dee5a03 100644
@@ -196,6 +196,12 @@
 
 #include "i915_drv.h"
 #include "i915_oa_hsw.h"
+#include "i915_oa_bdw.h"
+#include "i915_oa_chv.h"
+#include "i915_oa_sklgt2.h"
+#include "i915_oa_sklgt3.h"
+#include "i915_oa_sklgt4.h"
+#include "i915_oa_bxt.h"
 
 /* HW requires this to be a power of two, between 128k and 16M, though driver
  * is currently generally designed assuming the largest 16M size is used such
@@ -215,7 +221,7 @@
  *
  * Although this can be observed explicitly while copying reports to userspace
  * by checking for a zeroed report-id field in tail reports, we want to account
- * for this earlier, as part of the _oa_buffer_check to avoid lots of redundant
+ * for this earlier, as part of the oa_buffer_check to avoid lots of redundant
  * read() attempts.
  *
  * In effect we define a tail pointer for reading that lags the real tail
@@ -237,7 +243,7 @@
  * indicates that an updated tail pointer is needed.
  *
  * Most of the implementation details for this workaround are in
- * gen7_oa_buffer_check_unlocked() and gen7_appand_oa_reports()
+ * oa_buffer_check_unlocked() and _append_oa_reports()
  *
  * Note for posterity: previously the driver used to define an effective tail
  * pointer that lagged the real pointer by a 'tail margin' measured in bytes
@@ -272,6 +278,13 @@ static u32 i915_perf_stream_paranoid = true;
 
 #define INVALID_CTX_ID 0xffffffff
 
+/* On Gen8+ automatically triggered OA reports include a 'reason' field... */
+#define OAREPORT_REASON_MASK           0x3f
+#define OAREPORT_REASON_SHIFT          19
+#define OAREPORT_REASON_TIMER          (1<<0)
+#define OAREPORT_REASON_CTX_SWITCH     (1<<3)
+#define OAREPORT_REASON_CLK_RATIO      (1<<5)
+
 
 /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
  *
@@ -303,6 +316,13 @@ static struct i915_oa_format hsw_oa_formats[I915_OA_FORMAT_MAX] = {
        [I915_OA_FORMAT_C4_B8]      = { 7, 64 },
 };
 
+static struct i915_oa_format gen8_plus_oa_formats[I915_OA_FORMAT_MAX] = {
+       [I915_OA_FORMAT_A12]                = { 0, 64 },
+       [I915_OA_FORMAT_A12_B8_C8]          = { 2, 128 },
+       [I915_OA_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 },
+       [I915_OA_FORMAT_C4_B8]              = { 7, 64 },
+};
+
 #define SAMPLE_OA_REPORT      (1<<0)
 
 /**
@@ -332,8 +352,20 @@ struct perf_open_properties {
        int oa_period_exponent;
 };
 
+static u32 gen8_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+       return I915_READ(GEN8_OATAILPTR) & GEN8_OATAILPTR_MASK;
+}
+
+static u32 gen7_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+       u32 oastatus1 = I915_READ(GEN7_OASTATUS1);
+
+       return oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+}
+
 /**
- * gen7_oa_buffer_check_unlocked - check for data and update tail ptr state
+ * oa_buffer_check_unlocked - check for data and update tail ptr state
  * @dev_priv: i915 device instance
  *
  * This is either called via fops (for blocking reads in user ctx) or the poll
@@ -356,12 +388,11 @@ struct perf_open_properties {
  *
  * Returns: %true if the OA buffer contains data, else %false
  */
-static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
+static bool oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 {
        int report_size = dev_priv->perf.oa.oa_buffer.format_size;
        unsigned long flags;
        unsigned int aged_idx;
-       u32 oastatus1;
        u32 head, hw_tail, aged_tail, aging_tail;
        u64 now;
 
@@ -381,8 +412,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
        aged_tail = dev_priv->perf.oa.oa_buffer.tails[aged_idx].offset;
        aging_tail = dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset;
 
-       oastatus1 = I915_READ(GEN7_OASTATUS1);
-       hw_tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+       hw_tail = dev_priv->perf.oa.ops.oa_hw_tail_read(dev_priv);
 
        /* The tail pointer increases in 64 byte increments,
         * not in report_size steps...
@@ -404,6 +434,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
        if (aging_tail != INVALID_TAIL_PTR &&
            ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
             OA_TAIL_MARGIN_NSEC)) {
+
                aged_idx ^= 1;
                dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
 
@@ -553,6 +584,287 @@ static int append_oa_sample(struct i915_perf_stream *stream,
  *
  * Returns: 0 on success, negative error code on failure.
  */
+static int gen8_append_oa_reports(struct i915_perf_stream *stream,
+                                 char __user *buf,
+                                 size_t count,
+                                 size_t *offset)
+{
+       struct drm_i915_private *dev_priv = stream->dev_priv;
+       int report_size = dev_priv->perf.oa.oa_buffer.format_size;
+       u8 *oa_buf_base = dev_priv->perf.oa.oa_buffer.vaddr;
+       u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+       u32 mask = (OA_BUFFER_SIZE - 1);
+       size_t start_offset = *offset;
+       unsigned long flags;
+       unsigned int aged_tail_idx;
+       u32 head, tail;
+       u32 taken;
+       int ret = 0;
+
+       if (WARN_ON(!stream->enabled))
+               return -EIO;
+
+       spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       head = dev_priv->perf.oa.oa_buffer.head;
+       aged_tail_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+       tail = dev_priv->perf.oa.oa_buffer.tails[aged_tail_idx].offset;
+
+       spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       /* An invalid tail pointer here means we're still waiting for the poll
+        * hrtimer callback to give us a pointer
+        */
+       if (tail == INVALID_TAIL_PTR)
+               return -EAGAIN;
+
+       /* NB: oa_buffer.head/tail include the gtt_offset which we don't want
+        * while indexing relative to oa_buf_base.
+        */
+       head -= gtt_offset;
+       tail -= gtt_offset;
+
+       /* An out of bounds or misaligned head or tail pointer implies a driver
+        * bug since we validate + align the tail pointers we read from the
+        * hardware and we are in full control of the head pointer which should
+        * only be incremented by multiples of the report size (notably also
+        * all a power of two).
+        */
+       if (WARN_ONCE(head > OA_BUFFER_SIZE || head % report_size ||
+                     tail > OA_BUFFER_SIZE || tail % report_size,
+                     "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
+                     head, tail))
+               return -EIO;
+
+
+       for (/* none */;
+            (taken = OA_TAKEN(tail, head));
+            head = (head + report_size) & mask) {
+               u8 *report = oa_buf_base + head;
+               u32 *report32 = (void *)report;
+               u32 ctx_id;
+               u32 reason;
+
+               /* All the report sizes factor neatly into the buffer
+                * size so we never expect to see a report split
+                * between the beginning and end of the buffer.
+                *
+                * Given the initial alignment check a misalignment
+                * here would imply a driver bug that would result
+                * in an overrun.
+                */
+               if (WARN_ON((OA_BUFFER_SIZE - head) < report_size)) {
+                       DRM_ERROR("Spurious OA head ptr: non-integral report offset\n");
+                       break;
+               }
+
+               /* The reason field includes flags identifying what
+                * triggered this specific report (mostly timer
+                * triggered or e.g. due to a context switch).
+                *
+                * This field is never expected to be zero so we can
+                * check that the report isn't invalid before copying
+                * it to userspace...
+                */
+               reason = ((report32[0] >> OAREPORT_REASON_SHIFT) &
+                         OAREPORT_REASON_MASK);
+               if (reason == 0) {
+                       if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+                               DRM_NOTE("Skipping spurious, invalid OA report\n");
+                       continue;
+               }
+
+               /* XXX: Just keep the lower 21 bits for now since I'm not
+                * entirely sure if the HW touches any of the higher bits in
+                * this field
+                */
+               ctx_id = report32[2] & 0x1fffff;
+
+               /* Squash whatever is in the CTX_ID field if it's marked as
+                * invalid to be sure we avoid false-positive, single-context
+                * filtering below...
+                *
+                * Note: that we don't clear the valid_ctx_bit so userspace can
+                * understand that the ID has been squashed by the kernel.
+                */
+               if (!(report32[0] & dev_priv->perf.oa.gen8_valid_ctx_bit))
+                       ctx_id = report32[2] = INVALID_CTX_ID;
+
+               /* NB: For Gen 8 the OA unit no longer supports clock gating
+                * off for a specific context and the kernel can't securely
+                * stop the counters from updating as system-wide / global
+                * values.
+                *
+                * Automatic reports now include a context ID so reports can be
+                * filtered on the cpu but it's not worth trying to
+                * automatically subtract/hide counter progress for other
+                * contexts while filtering since we can't stop userspace
+                * issuing MI_REPORT_PERF_COUNT commands which would still
+                * provide a side-band view of the real values.
+                *
+                * To allow userspace (such as Mesa/GL_INTEL_performance_query)
+                * to normalize counters for a single filtered context then it
+                * needs be forwarded bookend context-switch reports so that it
+                * can track switches in between MI_REPORT_PERF_COUNT commands
+                * and can itself subtract/ignore the progress of counters
+                * associated with other contexts. Note that the hardware
+                * automatically triggers reports when switching to a new
+                * context which are tagged with the ID of the newly active
+                * context. To avoid the complexity (and likely fragility) of
+                * reading ahead while parsing reports to try and minimize
+                * forwarding redundant context switch reports (i.e. between
+                * other, unrelated contexts) we simply elect to forward them
+                * all.
+                *
+                * We don't rely solely on the reason field to identify context
+                * switches since it's not-uncommon for periodic samples to
+                * identify a switch before any 'context switch' report.
+                */
+               if (!dev_priv->perf.oa.exclusive_stream->ctx ||
+                   dev_priv->perf.oa.specific_ctx_id == ctx_id ||
+                   (dev_priv->perf.oa.oa_buffer.last_ctx_id ==
+                    dev_priv->perf.oa.specific_ctx_id) ||
+                   reason & OAREPORT_REASON_CTX_SWITCH) {
+
+                       /* While filtering for a single context we avoid
+                        * leaking the IDs of other contexts.
+                        */
+                       if (dev_priv->perf.oa.exclusive_stream->ctx &&
+                           dev_priv->perf.oa.specific_ctx_id != ctx_id) {
+                               report32[2] = INVALID_CTX_ID;
+                       }
+
+                       ret = append_oa_sample(stream, buf, count, offset,
+                                              report);
+                       if (ret)
+                               break;
+
+                       dev_priv->perf.oa.oa_buffer.last_ctx_id = ctx_id;
+               }
+
+               /* The above reason field sanity check is based on
+                * the assumption that the OA buffer is initially
+                * zeroed and we reset the field after copying so the
+                * check is still meaningful once old reports start
+                * being overwritten.
+                */
+               report32[0] = 0;
+       }
+
+       if (start_offset != *offset) {
+               spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+               /* We removed the gtt_offset for the copy loop above, indexing
+                * relative to oa_buf_base so put back here...
+                */
+               head += gtt_offset;
+
+               I915_WRITE(GEN8_OAHEADPTR, head & GEN8_OAHEADPTR_MASK);
+               dev_priv->perf.oa.oa_buffer.head = head;
+
+               spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+       }
+
+       return ret;
+}
+
+/**
+ * gen8_oa_read - copy status records then buffered OA reports
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Checks OA unit status registers and if necessary appends corresponding
+ * status records for userspace (such as for a buffer full condition) and then
+ * initiate appending any buffered OA reports.
+ *
+ * Updates @offset according to the number of bytes successfully copied into
+ * the userspace buffer.
+ *
+ * NB: some data may be successfully copied to the userspace buffer
+ * even if an error is returned, and this is reflected in the
+ * updated @offset.
+ *
+ * Returns: zero on success or a negative error code
+ */
+static int gen8_oa_read(struct i915_perf_stream *stream,
+                       char __user *buf,
+                       size_t count,
+                       size_t *offset)
+{
+       struct drm_i915_private *dev_priv = stream->dev_priv;
+       u32 oastatus;
+       int ret;
+
+       if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
+               return -EIO;
+
+       oastatus = I915_READ(GEN8_OASTATUS);
+
+       /* We treat OABUFFER_OVERFLOW as a significant error:
+        *
+        * Although theoretically we could handle this more gracefully
+        * sometimes, some Gens don't correctly suppress certain
+        * automatically triggered reports in this condition and so we
+        * have to assume that old reports are now being trampled
+        * over.
+        *
+        * Considering how we don't currently give userspace control
+        * over the OA buffer size and always configure a large 16MB
+        * buffer, then a buffer overflow does anyway likely indicate
+        * that something has gone quite badly wrong.
+        */
+       if (oastatus & GEN8_OASTATUS_OABUFFER_OVERFLOW) {
+               ret = append_oa_status(stream, buf, count, offset,
+                                      DRM_I915_PERF_RECORD_OA_BUFFER_LOST);
+               if (ret)
+                       return ret;
+
+               DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+                         dev_priv->perf.oa.period_exponent);
+
+               dev_priv->perf.oa.ops.oa_disable(dev_priv);
+               dev_priv->perf.oa.ops.oa_enable(dev_priv);
+
+               /* Note: .oa_enable() is expected to re-init the oabuffer
+                * and reset GEN8_OASTATUS for us
+                */
+               oastatus = I915_READ(GEN8_OASTATUS);
+       }
+
+       if (oastatus & GEN8_OASTATUS_REPORT_LOST) {
+               ret = append_oa_status(stream, buf, count, offset,
+                                      DRM_I915_PERF_RECORD_OA_REPORT_LOST);
+               if (ret)
+                       return ret;
+               I915_WRITE(GEN8_OASTATUS,
+                          oastatus & ~GEN8_OASTATUS_REPORT_LOST);
+       }
+
+       return gen8_append_oa_reports(stream, buf, count, offset);
+}
+
+/**
+ * Copies all buffered OA reports into userspace read() buffer.
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Notably any error condition resulting in a short read (-%ENOSPC or
+ * -%EFAULT) will be returned even though one or more records may
+ * have been successfully copied. In this case it's up to the caller
+ * to decide if the error should be squashed before returning to
+ * userspace.
+ *
+ * Note: reports are consumed from the head, and appended to the
+ * tail, so the tail chases the head?... If you think that's mad
+ * and back-to-front you're not alone, but this follows the
+ * Gen PRM naming convention.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
 static int gen7_append_oa_reports(struct i915_perf_stream *stream,
                                  char __user *buf,
                                  size_t count,
@@ -733,7 +1045,8 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
                if (ret)
                        return ret;
 
-               DRM_DEBUG("OA buffer overflow: force restart\n");
+               DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+                         dev_priv->perf.oa.period_exponent);
 
                dev_priv->perf.oa.ops.oa_disable(dev_priv);
                dev_priv->perf.oa.ops.oa_enable(dev_priv);
@@ -776,7 +1089,7 @@ static int i915_oa_wait_unlocked(struct i915_perf_stream *stream)
                return -EIO;
 
        return wait_event_interruptible(dev_priv->perf.oa.poll_wq,
-                                       dev_priv->perf.oa.ops.oa_buffer_check(dev_priv));
+                                       oa_buffer_check_unlocked(dev_priv));
 }
 
 /**
@@ -833,33 +1146,37 @@ static int i915_oa_read(struct i915_perf_stream *stream,
 static int oa_get_render_ctx_id(struct i915_perf_stream *stream)
 {
        struct drm_i915_private *dev_priv = stream->dev_priv;
-       struct intel_engine_cs *engine = dev_priv->engine[RCS];
-       int ret;
 
-       ret = i915_mutex_lock_interruptible(&dev_priv->drm);
-       if (ret)
-               return ret;
+       if (i915.enable_execlists)
+               dev_priv->perf.oa.specific_ctx_id = stream->ctx->hw_id;
+       else {
+               struct intel_engine_cs *engine = dev_priv->engine[RCS];
+               int ret;
 
-       /* As the ID is the gtt offset of the context's vma we pin
-        * the vma to ensure the ID remains fixed.
-        *
-        * NB: implied RCS engine...
-        */
-       ret = engine->context_pin(engine, stream->ctx);
-       if (ret)
-               goto unlock;
+               ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+               if (ret)
+                       return ret;
 
-       /* Explicitly track the ID (instead of calling i915_ggtt_offset()
-        * on the fly) considering the difference with gen8+ and
-        * execlists
-        */
-       dev_priv->perf.oa.specific_ctx_id =
-               i915_ggtt_offset(stream->ctx->engine[engine->id].state);
+               /* As the ID is the gtt offset of the context's vma we pin
+                * the vma to ensure the ID remains fixed.
+                */
+               ret = engine->context_pin(engine, stream->ctx);
+               if (ret) {
+                       mutex_unlock(&dev_priv->drm.struct_mutex);
+                       return ret;
+               }
 
-unlock:
-       mutex_unlock(&dev_priv->drm.struct_mutex);
+               /* Explicitly track the ID (instead of calling
+                * i915_ggtt_offset() on the fly) considering the difference
+                * with gen8+ and execlists
+                */
+               dev_priv->perf.oa.specific_ctx_id =
+                       i915_ggtt_offset(stream->ctx->engine[engine->id].state);
 
-       return ret;
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+       }
+
+       return 0;
 }
 
 /**
@@ -874,12 +1191,16 @@ static void oa_put_render_ctx_id(struct i915_perf_stream *stream)
        struct drm_i915_private *dev_priv = stream->dev_priv;
        struct intel_engine_cs *engine = dev_priv->engine[RCS];
 
-       mutex_lock(&dev_priv->drm.struct_mutex);
+       if (i915.enable_execlists) {
+               dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+       } else {
+               mutex_lock(&dev_priv->drm.struct_mutex);
 
-       dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
-       engine->context_unpin(engine, stream->ctx);
+               dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+               engine->context_unpin(engine, stream->ctx);
 
-       mutex_unlock(&dev_priv->drm.struct_mutex);
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+       }
 }
 
 static void
@@ -969,6 +1290,61 @@ static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
        dev_priv->perf.oa.pollin = false;
 }
 
+static void gen8_init_oa_buffer(struct drm_i915_private *dev_priv)
+{
+       u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+       unsigned long flags;
+
+       spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       I915_WRITE(GEN8_OASTATUS, 0);
+       I915_WRITE(GEN8_OAHEADPTR, gtt_offset);
+       dev_priv->perf.oa.oa_buffer.head = gtt_offset;
+
+       I915_WRITE(GEN8_OABUFFER_UDW, 0);
+
+       /* PRM says:
+        *
+        *  "This MMIO must be set before the OATAILPTR
+        *  register and after the OAHEADPTR register. This is
+        *  to enable proper functionality of the overflow
+        *  bit."
+        */
+       I915_WRITE(GEN8_OABUFFER, gtt_offset |
+                  OABUFFER_SIZE_16M | OA_MEM_SELECT_GGTT);
+       I915_WRITE(GEN8_OATAILPTR, gtt_offset & GEN8_OATAILPTR_MASK);
+
+       /* Mark that we need updated tail pointers to read from... */
+       dev_priv->perf.oa.oa_buffer.tails[0].offset = INVALID_TAIL_PTR;
+       dev_priv->perf.oa.oa_buffer.tails[1].offset = INVALID_TAIL_PTR;
+
+       /* Reset state used to recognise context switches, affecting which
+        * reports we will forward to userspace while filtering for a single
+        * context.
+        */
+       dev_priv->perf.oa.oa_buffer.last_ctx_id = INVALID_CTX_ID;
+
+       spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       /* NB: although the OA buffer will initially be allocated
+        * zeroed via shmfs (and so this memset is redundant when
+        * first allocating), we may re-init the OA buffer, either
+        * when re-enabling a stream or in error/reset paths.
+        *
+        * The reason we clear the buffer for each re-init is for the
+        * sanity check in gen8_append_oa_reports() that looks at the
+        * reason field to make sure it's non-zero which relies on
+        * the assumption that new reports are being written to zeroed
+        * memory...
+        */
+       memset(dev_priv->perf.oa.oa_buffer.vaddr, 0, OA_BUFFER_SIZE);
+
+       /* Maybe make ->pollin per-stream state if we support multiple
+        * concurrent streams in the future.
+        */
+       dev_priv->perf.oa.pollin = false;
+}
+
 static int alloc_oa_buffer(struct drm_i915_private *dev_priv)
 {
        struct drm_i915_gem_object *bo;
@@ -1113,6 +1489,280 @@ static void hsw_disable_metric_set(struct drm_i915_private *dev_priv)
                                      ~GT_NOA_ENABLE));
 }
 
+/*
+ * From Broadwell PRM, 3D-Media-GPGPU -> Register State Context
+ *
+ * MMIO reads or writes to any of the registers listed in the
+ * “Register State Context image” subsections through HOST/IA
+ * MMIO interface for debug purposes must follow the steps below:
+ *
+ * - SW should set the Force Wakeup bit to prevent GT from entering C6.
+ * - Write 0x2050[31:0] = 0x00010001 (disable sequence).
+ * - Disable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010001).
+ * - BDW:  Poll/Wait for register bits of 0x22AC[6:0] turn to 0x30 value.
+ * - SKL+: Poll/Wait for register bits of 0x22A4[6:0] turn to 0x30 value.
+ * - Read/Write to desired MMIO registers.
+ * - Enable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010000).
+ * - Force Wakeup bit should be reset to enable C6 entry.
+ *
+ * XXX: don't nest or overlap calls to this API, it has no ref
+ * counting to track how many entities require the RCS to be
+ * blocked from being idle.
+ */
+static int gen8_begin_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+       i915_reg_t fsm_reg = INTEL_GEN(dev_priv) > 8 ?
+               GEN9_RCS_FE_FSM2 : GEN6_RCS_PWR_FSM;
+       int ret = 0;
+
+       /* There's only no active context while idle in execlist mode
+        * (though we shouldn't be using this in any other case)
+        */
+       if (WARN_ON(!i915.enable_execlists))
+               return -EINVAL;
+
+       intel_uncore_forcewake_get(dev_priv, FORCEWAKE_RENDER);
+
+       I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+                  _MASKED_BIT_ENABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+
+       /* Note: we don't currently have a good handle on the maximum
+        * latency for this wake up so while we only need to hold rcs
+        * busy from process context we at least keep the waiting
+        * interruptible...
+        */
+       ret = intel_wait_for_register(dev_priv, fsm_reg, 0x3f, 0x30, 50);
+       if (ret == -ETIMEDOUT)
+               DRM_ERROR("Timeout waiting to be able to begin per-ctx mmio\n");
+
+       if (ret)
+               intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+
+       return ret;
+}
+
+static void gen8_end_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+       if (WARN_ON(!i915.enable_execlists))
+               return;
+
+       I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+                  _MASKED_BIT_DISABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+       intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+}
+
+/* NB: It must always remain pointer safe to run this even if the OA unit
+ * has been disabled.
+ *
+ * It's fine to put out-of-date values into these per-context registers
+ * in the case that the OA unit has been disabled.
+ */
+static void gen8_update_reg_state_unlocked(struct i915_gem_context *ctx,
+                                          u32 *reg_state)
+{
+       struct drm_i915_private *dev_priv = ctx->i915;
+       const struct i915_oa_reg *flex_regs = dev_priv->perf.oa.flex_regs;
+       int n_flex_regs = dev_priv->perf.oa.flex_regs_len;
+       u32 ctx_oactxctrl = dev_priv->perf.oa.ctx_oactxctrl_offset;
+       u32 ctx_flexeu0 = dev_priv->perf.oa.ctx_flexeu0_offset;
+       /* The MMIO offsets for Flex EU registers aren't contiguous */
+       u32 flex_mmio[] = {
+               i915_mmio_reg_offset(EU_PERF_CNTL0),
+               i915_mmio_reg_offset(EU_PERF_CNTL1),
+               i915_mmio_reg_offset(EU_PERF_CNTL2),
+               i915_mmio_reg_offset(EU_PERF_CNTL3),
+               i915_mmio_reg_offset(EU_PERF_CNTL4),
+               i915_mmio_reg_offset(EU_PERF_CNTL5),
+               i915_mmio_reg_offset(EU_PERF_CNTL6),
+       };
+       int i;
+
+       reg_state[ctx_oactxctrl] = i915_mmio_reg_offset(GEN8_OACTXCONTROL);
+       reg_state[ctx_oactxctrl+1] = (dev_priv->perf.oa.period_exponent <<
+                                     GEN8_OA_TIMER_PERIOD_SHIFT) |
+                                    (dev_priv->perf.oa.periodic ?
+                                     GEN8_OA_TIMER_ENABLE : 0) |
+                                    GEN8_OA_COUNTER_RESUME;
+
+       for (i = 0; i < ARRAY_SIZE(flex_mmio); i++) {
+               u32 state_offset = ctx_flexeu0 + i * 2;
+               u32 mmio = flex_mmio[i];
+
+               /* This arbitrary default will select the 'EU FPU0 Pipeline
+                * Active' event. In the future it's anticipated that there
+                * will be an explicit 'No Event' we can select, but not yet...
+                */
+               u32 value = 0;
+               int j;
+
+               for (j = 0; j < n_flex_regs; j++) {
+                       if (i915_mmio_reg_offset(flex_regs[j].addr) == mmio) {
+                               value = flex_regs[j].value;
+                               break;
+                       }
+               }
+
+               reg_state[state_offset] = mmio;
+               reg_state[state_offset+1] = value;
+       }
+}
+
+/* Manages updating the per-context aspects of the OA stream
+ * configuration across all contexts.
+ *
+ * The awkward consideration here is that OACTXCONTROL controls the
+ * exponent for periodic sampling which is primarily used for system
+ * wide profiling where we'd like a consistent sampling period even in
+ * the face of context switches.
+ *
+ * Our approach of updating the register state context (as opposed to
+ * say using a workaround batch buffer) ensures that the hardware
+ * won't automatically reload an out-of-date timer exponent even
+ * transiently before a WA BB could be parsed.
+ *
+ * This function needs to:
+ * - Ensure the currently running context's per-context OA state is
+ *   updated
+ * - Ensure that all existing contexts will have the correct per-context
+ *   OA state if they are scheduled for use.
+ * - Ensure any new contexts will be initialized with the correct
+ *   per-context OA state.
+ *
+ * Note: it's only the RCS/Render context that has any OA state.
+ */
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1644

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+       struct i915_gem_context *ctx;
+       int ret;
+
+       ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+       if (ret)
+               return ret;
+
+       /* Switch away from any user context.  */
+       ret = i915_gem_switch_to_kernel_context(dev_priv);
+       if (ret)

mutex_unlock(&dev_priv->drm.struct_mutex);

Quoted 33 lines
drivers/gpu/drm/i915/i915_perf.c +33 -0
+               return ret;
+
+       /* The OA register config is setup through the context image. This image
+        * might be written to by the GPU on context switch (in particular on
+        * lite-restore). This means we can't safely update a context's image,
+        * if this context is scheduled/submitted to run on the GPU.
+        *
+        * We could emit the OA register config through the batch buffer but
+        * this might leave small interval of time where the OA unit is
+        * configured at an invalid sampling period.
+        *
+        * So far the best way to work around this issue seems to be draining
+        * the GPU from any submitted work.
+        */
+       ret = i915_gem_wait_for_idle(dev_priv,
+                                    I915_WAIT_INTERRUPTIBLE |
+                                    I915_WAIT_LOCKED);
+       if (ret) {
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+               return ret;
+       }
+
+       /* Update all contexts now that we've stalled the submission. */
+       list_for_each_entry(ctx, &dev_priv->context_list, link) {
+               if (!ctx->engine[RCS].initialised)
+                       continue;
+
+               gen8_update_reg_state_unlocked(ctx,
+                                              ctx->engine[RCS].lrc_reg_state);
+       }
+
+       mutex_unlock(&dev_priv->drm.struct_mutex);
+

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1687

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+       /* Now update the current context.
+        *
+        * Note: Using MMIO to update per-context registers requires
+        * some extra care...
+        */
+       ret = gen8_begin_ctx_mmio(dev_priv);
+       if (ret) {
+               DRM_ERROR("Failed to bring RCS out of idle to update current ctx OA state\n");
+               return ret;
+       }

So do we still need the mmio for the current context, given that we now idle everything and then update the reg state?

Quoted 517 lines
drivers/gpu/drm/i915/i915_perf.c +279 -36
+
+       I915_WRITE(GEN8_OACTXCONTROL, ((dev_priv->perf.oa.period_exponent <<
+                                       GEN8_OA_TIMER_PERIOD_SHIFT) |
+                                     (dev_priv->perf.oa.periodic ?
+                                      GEN8_OA_TIMER_ENABLE : 0) |
+                                     GEN8_OA_COUNTER_RESUME));
+
+       config_oa_regs(dev_priv, dev_priv->perf.oa.flex_regs,
+                       dev_priv->perf.oa.flex_regs_len);
+
+       gen8_end_ctx_mmio(dev_priv);
+
+       return 0;
+}
+
+static int gen8_enable_metric_set(struct drm_i915_private *dev_priv)
+{
+       int ret = dev_priv->perf.oa.ops.select_metric_set(dev_priv);
+
+       if (ret)
+               return ret;
+
+       /* We disable slice/unslice clock ratio change reports on SKL since
+        * they are too noisy. The HW generates a lot of redundant reports
+        * where the ratio hasn't really changed causing a lot of redundant
+        * work to processes and increasing the chances we'll hit buffer
+        * overruns.
+        *
+        * Although we don't currently use the 'disable overrun' OABUFFER
+        * feature it's worth noting that clock ratio reports have to be
+        * disabled before considering to use that feature since the HW doesn't
+        * correctly block these reports.
+        *
+        * Currently none of the high-level metrics we have depend on knowing
+        * this ratio to normalize.
+        *
+        * Note: This register is not power context saved and restored, but
+        * that's OK considering that we disable RC6 while the OA unit is
+        * enabled.
+        *
+        * The _INCLUDE_CLK_RATIO bit allows the slice/unslice frequency to
+        * be read back from automatically triggered reports, as part of the
+        * RPT_ID field.
+        */
+       if (IS_SKYLAKE(dev_priv) || IS_BROXTON(dev_priv)) {
+               I915_WRITE(GEN8_OA_DEBUG,
+                          _MASKED_BIT_ENABLE(GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS |
+                                             GEN9_OA_DEBUG_INCLUDE_CLK_RATIO));
+       }
+
+       I915_WRITE(GDT_CHICKEN_BITS, 0xA0);
+       config_oa_regs(dev_priv, dev_priv->perf.oa.mux_regs,
+                      dev_priv->perf.oa.mux_regs_len);
+       I915_WRITE(GDT_CHICKEN_BITS, 0x80);
+
+       /* It takes a fairly long time for a new MUX configuration to
+        * be be applied after these register writes. This delay
+        * duration is take from Haswell (derived empirically based on
+        * the render_basic config) but hopefully it covers the
+        * maximum configuration latency for Gen8+ too...
+        */
+       usleep_range(15000, 20000);
+
+       config_oa_regs(dev_priv, dev_priv->perf.oa.b_counter_regs,
+                      dev_priv->perf.oa.b_counter_regs_len);
+
+       ret = gen8_configure_all_contexts(dev_priv);
+       if (ret)
+               return ret;
+
+       return 0;
+}
+
+static void gen8_disable_metric_set(struct drm_i915_private *dev_priv)
+{
+       /* NOP */
+}
+
 static void gen7_update_oacontrol_locked(struct drm_i915_private *dev_priv)
 {
        lockdep_assert_held(&dev_priv->perf.hook_lock);
@@ -1157,6 +1807,29 @@ static void gen7_oa_enable(struct drm_i915_private *dev_priv)
        spin_unlock_irqrestore(&dev_priv->perf.hook_lock, flags);
 }
 
+static void gen8_oa_enable(struct drm_i915_private *dev_priv)
+{
+       u32 report_format = dev_priv->perf.oa.oa_buffer.format;
+
+       /* Reset buf pointers so we don't forward reports from before now.
+        *
+        * Think carefully if considering trying to avoid this, since it
+        * also ensures status flags and the buffer itself are cleared
+        * in error paths, and we have checks for invalid reports based
+        * on the assumption that certain fields are written to zeroed
+        * memory which this helps maintains.
+        */
+       gen8_init_oa_buffer(dev_priv);
+
+       /* Note: we don't rely on the hardware to perform single context
+        * filtering and instead filter on the cpu based on the context-id
+        * field of reports
+        */
+       I915_WRITE(GEN8_OACONTROL, (report_format <<
+                                   GEN8_OA_REPORT_FORMAT_SHIFT) |
+                                  GEN8_OA_COUNTER_ENABLE);
+}
+
 /**
  * i915_oa_stream_enable - handle `I915_PERF_IOCTL_ENABLE` for OA stream
  * @stream: An i915 perf stream opened for OA metrics
@@ -1183,6 +1856,11 @@ static void gen7_oa_disable(struct drm_i915_private *dev_priv)
        I915_WRITE(GEN7_OACONTROL, 0);
 }
 
+static void gen8_oa_disable(struct drm_i915_private *dev_priv)
+{
+       I915_WRITE(GEN8_OACONTROL, 0);
+}
+
 /**
  * i915_oa_stream_disable - handle `I915_PERF_IOCTL_DISABLE` for OA stream
  * @stream: An i915 perf stream opened for OA metrics
@@ -1361,6 +2039,21 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream,
        return ret;
 }
 
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+                           struct i915_gem_context *ctx,
+                           u32 *reg_state)
+{
+       struct drm_i915_private *dev_priv = engine->i915;
+
+       if (engine->id != RCS)
+               return;
+
+       if (!dev_priv->perf.initialized)
+               return;
+
+       gen8_update_reg_state_unlocked(ctx, reg_state);
+}
+
 /**
  * i915_perf_read_locked - &i915_perf_stream_ops->read with error normalisation
  * @stream: An i915 perf stream
@@ -1486,7 +2179,7 @@ static enum hrtimer_restart oa_poll_check_timer_cb(struct hrtimer *hrtimer)
                container_of(hrtimer, typeof(*dev_priv),
                             perf.oa.poll_check_timer);
 
-       if (dev_priv->perf.oa.ops.oa_buffer_check(dev_priv)) {
+       if (oa_buffer_check_unlocked(dev_priv)) {
                dev_priv->perf.oa.pollin = true;
                wake_up(&dev_priv->perf.oa.poll_wq);
        }
@@ -1775,6 +2468,7 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
        struct i915_gem_context *specific_ctx = NULL;
        struct i915_perf_stream *stream = NULL;
        unsigned long f_flags = 0;
+       bool privileged_op = true;
        int stream_fd;
        int ret;
 
@@ -1792,12 +2486,28 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
                }
        }
 
+       /* On Haswell the OA unit supports clock gating off for a specific
+        * context and in this mode there's no visibility of metrics for the
+        * rest of the system, which we consider acceptable for a
+        * non-privileged client.
+        *
+        * For Gen8+ the OA unit no longer supports clock gating off for a
+        * specific context and the kernel can't securely stop the counters
+        * from updating as system-wide / global values. Even though we can
+        * filter reports based on the included context ID we can't block
+        * clients from seeing the raw / global counter values via
+        * MI_REPORT_PERF_COUNT commands and so consider it a privileged op to
+        * enable the OA unit by default.
+        */
+       if (IS_HASWELL(dev_priv) && specific_ctx)
+               privileged_op = false;
+
        /* Similar to perf's kernel.perf_paranoid_cpu sysctl option
         * we check a dev.i915.perf_stream_paranoid sysctl option
         * to determine if it's ok to access system wide OA counters
         * without CAP_SYS_ADMIN privileges.
         */
-       if (!specific_ctx &&
+       if (privileged_op &&
            i915_perf_stream_paranoid && !capable(CAP_SYS_ADMIN)) {
                DRM_DEBUG("Insufficient privileges to open system-wide i915 perf stream\n");
                ret = -EACCES;
@@ -2069,9 +2779,6 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data,
  */
 void i915_perf_register(struct drm_i915_private *dev_priv)
 {
-       if (!IS_HASWELL(dev_priv))
-               return;
-
        if (!dev_priv->perf.initialized)
                return;
 
@@ -2087,11 +2794,38 @@ void i915_perf_register(struct drm_i915_private *dev_priv)
        if (!dev_priv->perf.metrics_kobj)
                goto exit;
 
-       if (i915_perf_register_sysfs_hsw(dev_priv)) {
-               kobject_put(dev_priv->perf.metrics_kobj);
-               dev_priv->perf.metrics_kobj = NULL;
+       if (IS_HASWELL(dev_priv)) {
+               if (i915_perf_register_sysfs_hsw(dev_priv))
+                       goto sysfs_error;
+       } else if (IS_BROADWELL(dev_priv)) {
+               if (i915_perf_register_sysfs_bdw(dev_priv))
+                       goto sysfs_error;
+       } else if (IS_CHERRYVIEW(dev_priv)) {
+               if (i915_perf_register_sysfs_chv(dev_priv))
+                       goto sysfs_error;
+       } else if (IS_SKYLAKE(dev_priv)) {
+               if (IS_SKL_GT2(dev_priv)) {
+                       if (i915_perf_register_sysfs_sklgt2(dev_priv))
+                               goto sysfs_error;
+               } else if (IS_SKL_GT3(dev_priv)) {
+                       if (i915_perf_register_sysfs_sklgt3(dev_priv))
+                               goto sysfs_error;
+               } else if (IS_SKL_GT4(dev_priv)) {
+                       if (i915_perf_register_sysfs_sklgt4(dev_priv))
+                               goto sysfs_error;
+               } else
+                       goto sysfs_error;
+       } else if (IS_BROXTON(dev_priv)) {
+               if (i915_perf_register_sysfs_bxt(dev_priv))
+                       goto sysfs_error;
        }
 
+       goto exit;
+
+sysfs_error:
+       kobject_put(dev_priv->perf.metrics_kobj);
+       dev_priv->perf.metrics_kobj = NULL;
+
 exit:
        mutex_unlock(&dev_priv->perf.lock);
 }
@@ -2107,13 +2841,24 @@ void i915_perf_register(struct drm_i915_private *dev_priv)
  */
 void i915_perf_unregister(struct drm_i915_private *dev_priv)
 {
-       if (!IS_HASWELL(dev_priv))
-               return;
-
        if (!dev_priv->perf.metrics_kobj)
                return;
 
-       i915_perf_unregister_sysfs_hsw(dev_priv);
+        if (IS_HASWELL(dev_priv))
+                i915_perf_unregister_sysfs_hsw(dev_priv);
+        else if (IS_BROADWELL(dev_priv))
+                i915_perf_unregister_sysfs_bdw(dev_priv);
+        else if (IS_CHERRYVIEW(dev_priv))
+                i915_perf_unregister_sysfs_chv(dev_priv);
+        else if (IS_SKYLAKE(dev_priv)) {
+               if (IS_SKL_GT2(dev_priv))
+                       i915_perf_unregister_sysfs_sklgt2(dev_priv);
+               else if (IS_SKL_GT3(dev_priv))
+                       i915_perf_unregister_sysfs_sklgt3(dev_priv);
+               else if (IS_SKL_GT4(dev_priv))
+                       i915_perf_unregister_sysfs_sklgt4(dev_priv);
+       } else if (IS_BROXTON(dev_priv))
+                i915_perf_unregister_sysfs_bxt(dev_priv);
 
        kobject_put(dev_priv->perf.metrics_kobj);
        dev_priv->perf.metrics_kobj = NULL;
@@ -2172,36 +2917,105 @@ static struct ctl_table dev_root[] = {
  */
 void i915_perf_init(struct drm_i915_private *dev_priv)
 {
-       if (!IS_HASWELL(dev_priv))
-               return;
-
-       hrtimer_init(&dev_priv->perf.oa.poll_check_timer,
-                    CLOCK_MONOTONIC, HRTIMER_MODE_REL);
-       dev_priv->perf.oa.poll_check_timer.function = oa_poll_check_timer_cb;
-       init_waitqueue_head(&dev_priv->perf.oa.poll_wq);
+       dev_priv->perf.oa.n_builtin_sets = 0;
+
+       if (IS_HASWELL(dev_priv)) {
+               dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
+               dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
+               dev_priv->perf.oa.ops.disable_metric_set = hsw_disable_metric_set;
+               dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
+               dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
+               dev_priv->perf.oa.ops.read = gen7_oa_read;
+               dev_priv->perf.oa.ops.oa_hw_tail_read =
+                       gen7_oa_hw_tail_read;
+
+               dev_priv->perf.oa.oa_formats = hsw_oa_formats;
+
+               dev_priv->perf.oa.n_builtin_sets =
+                       i915_oa_n_builtin_metric_sets_hsw;
+       } else if (i915.enable_execlists) {
+               /* Note: that although we could theoretically also support the
+                * legacy ringbuffer mode on BDW (and earlier iterations of
+                * this driver, before upstreaming did this) it didn't seem
+                * worth the complexity to maintain now that BDW+ enable
+                * execlist mode by default.
+                */
 
-       INIT_LIST_HEAD(&dev_priv->perf.streams);
-       mutex_init(&dev_priv->perf.lock);
-       spin_lock_init(&dev_priv->perf.hook_lock);
-       spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
+               if (IS_GEN8(dev_priv)) {
+                       dev_priv->perf.oa.ctx_oactxctrl_offset = 0x120;
+                       dev_priv->perf.oa.ctx_flexeu0_offset = 0x2ce;
+                       dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<25);
+
+                       if (IS_BROADWELL(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_bdw;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_bdw;
+                       } else if (IS_CHERRYVIEW(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_chv;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_chv;
+                       }
+               } else if (IS_GEN9(dev_priv)) {
+                       dev_priv->perf.oa.ctx_oactxctrl_offset = 0x128;
+                       dev_priv->perf.oa.ctx_flexeu0_offset = 0x3de;
+                       dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<16);
+
+                       if (IS_SKL_GT2(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_sklgt2;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_sklgt2;
+                       } else if (IS_SKL_GT3(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_sklgt3;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_sklgt3;
+                       } else if (IS_SKL_GT4(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_sklgt4;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_sklgt4;
+                       } else if (IS_BROXTON(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_bxt;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_bxt;
+                       }
+               }
 
-       dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
-       dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
-       dev_priv->perf.oa.ops.disable_metric_set = hsw_disable_metric_set;
-       dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
-       dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
-       dev_priv->perf.oa.ops.read = gen7_oa_read;
-       dev_priv->perf.oa.ops.oa_buffer_check =
-               gen7_oa_buffer_check_unlocked;
+               if (dev_priv->perf.oa.n_builtin_sets) {
+                       dev_priv->perf.oa.ops.init_oa_buffer = gen8_init_oa_buffer;
+                       dev_priv->perf.oa.ops.enable_metric_set =
+                               gen8_enable_metric_set;
+                       dev_priv->perf.oa.ops.disable_metric_set =
+                               gen8_disable_metric_set;
+                       dev_priv->perf.oa.ops.oa_enable = gen8_oa_enable;
+                       dev_priv->perf.oa.ops.oa_disable = gen8_oa_disable;
+                       dev_priv->perf.oa.ops.read = gen8_oa_read;
+                       dev_priv->perf.oa.ops.oa_hw_tail_read =
+                               gen8_oa_hw_tail_read;
+
+                       dev_priv->perf.oa.oa_formats = gen8_plus_oa_formats;
+               }
+       }
 
-       dev_priv->perf.oa.oa_formats = hsw_oa_formats;
+       if (dev_priv->perf.oa.n_builtin_sets) {
+               hrtimer_init(&dev_priv->perf.oa.poll_check_timer,
+                               CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+               dev_priv->perf.oa.poll_check_timer.function = oa_poll_check_timer_cb;
+               init_waitqueue_head(&dev_priv->perf.oa.poll_wq);
 
-       dev_priv->perf.oa.n_builtin_sets =
-               i915_oa_n_builtin_metric_sets_hsw;
+               INIT_LIST_HEAD(&dev_priv->perf.streams);
+               mutex_init(&dev_priv->perf.lock);
+               spin_lock_init(&dev_priv->perf.hook_lock);
+               spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
 
-       dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
+               dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
 
-       dev_priv->perf.initialized = true;
+               dev_priv->perf.initialized = true;
+       }
 }
 
 /**
@@ -2216,5 +3030,6 @@ void i915_perf_fini(struct drm_i915_private *dev_priv)
        unregister_sysctl_table(dev_priv->perf.sysctl_header);
 
        memset(&dev_priv->perf.oa.ops, 0, sizeof(dev_priv->perf.oa.ops));
+
        dev_priv->perf.initialized = false;
 }
drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO           (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS      (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12         (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME              (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST         (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0      _MMIO(0xe458)
+#define EU_PERF_CNTL1      _MMIO(0xe558)
+#define EU_PERF_CNTL2      _MMIO(0xe658)
+#define EU_PERF_CNTL3      _MMIO(0xe758)
+#define EU_PERF_CNTL4      _MMIO(0xe45c)
+#define EU_PERF_CNTL5      _MMIO(0xe55c)
+#define EU_PERF_CNTL6      _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE      0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE        (1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE       (1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT                    _MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0          (1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 7df278fe492e..351e231a02f4 100644
@@ -1879,6 +1879,8 @@ static void execlists_init_reg_state(u32 *regs,
                regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
                CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
                        make_rpcs(dev_priv));
+
+               i915_oa_init_reg_state(engine, ctx, regs);
        }
 }
 
include/uapi/drm/i915_drm.h +12 -8
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-       I915_OA_FORMAT_A13 = 1,
-       I915_OA_FORMAT_A29,
-       I915_OA_FORMAT_A13_B8_C8,
-       I915_OA_FORMAT_B4_C8,
-       I915_OA_FORMAT_A45_B8_C8,
-       I915_OA_FORMAT_B4_C8_A16,
-       I915_OA_FORMAT_C4_B8,
+       I915_OA_FORMAT_A13 = 1,     /* HSW only */
+       I915_OA_FORMAT_A29,         /* HSW only */
+       I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+       I915_OA_FORMAT_B4_C8,       /* HSW only */
+       I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+       I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+       I915_OA_FORMAT_C4_B8,       /* HSW+ */
+
+       /* Gen8+ */
+       I915_OA_FORMAT_A12,
+       I915_OA_FORMAT_A12_B8_C8,
+       I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
        I915_OA_FORMAT_MAX          /* non-ABI */
 };
--
Quoted 5 lines

2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#35 lore raw

Re: [PATCH v6 12/15] drm/i915/perf: Add OA unit support for Gen 8+

Quoted 878 lines · Matthew Auld
Quoted 876 lines · Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

---
  drivers/gpu/drm/i915/i915_drv.h  |  45 +-
  drivers/gpu/drm/i915/i915_perf.c | 957 ++++++++++++++++++++++++++++++++++++---
  drivers/gpu/drm/i915/i915_reg.h  |  22 +
  drivers/gpu/drm/i915/intel_lrc.c |   2 +
  include/uapi/drm/i915_drm.h      |  19 +-
  5 files changed, 952 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index ffa1fc5eddfd..676b1227067c 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
         void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
         /**
-        * @enable_metric_set: Applies any MUX configuration to set up the
-        * Boolean and Custom (B/C) counters that are part of the counter
-        * reports being sampled. May apply system constraints such as
+        * @select_metric_set: The auto generated code that checks whether a
+        * requested OA config is applicable to the system and if so sets up
+        * the mux, oa and flex eu register config pointers according to the
+        * current dev_priv->perf.oa.metrics_set.
+        */
+       int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+       /**
+        * @enable_metric_set: Selects and applies any MUX configuration to set
+        * up the Boolean and Custom (B/C) counters that are part of the
+        * counter reports being sampled. May apply system constraints such as
          * disabling EU clock gating as required.
          */
         int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
                     size_t *offset);
 
         /**
-        * @oa_buffer_check: Check for OA buffer data + update tail
-        *
-        * This is either called via fops or the poll check hrtimer (atomic
-        * ctx) without any locks taken.
+        * @oa_hw_tail_read: read the OA tail pointer register
          *
-        * It's safe to read OA config state here unlocked, assuming that this
-        * is only called while the stream is enabled, while the global OA
-        * configuration can't be modified.
-        *
-        * Efficiency is more important than avoiding some false positives
-        * here, which will be handled gracefully - likely resulting in an
-        * %EAGAIN error for userspace.
+        * In particular this enables us to share all the fiddly code for
+        * handling the OA unit tail pointer race that affects multiple
+        * generations.
          */
-       bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+       u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
  };
 
  struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
                         struct {
                                 struct i915_vma *vma;
                                 u8 *vaddr;
+                               u32 last_ctx_id;
                                 int format;
                                 int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
                         } oa_buffer;
 
                         u32 gen7_latched_oastatus1;
+                       u32 ctx_oactxctrl_offset;
+                       u32 ctx_flexeu0_offset;
+
+                       /* The RPT_ID/reason field for Gen8+ includes a bit
+                        * to determine if the CTX ID in the report is valid
+                        * but the specific bit differs between Gen 8 and 9
+                        */
+                       u32 gen8_valid_ctx_bit;
 
                         struct i915_oa_ops ops;
                         const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
  #define IS_KBL_ULX(dev_priv)   (INTEL_DEVID(dev_priv) == 0x590E || \
                                  INTEL_DEVID(dev_priv) == 0x5915 || \
                                  INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)   (IS_SKYLAKE(dev_priv) && \
+                                (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
  #define IS_SKL_GT3(dev_priv)   (IS_SKYLAKE(dev_priv) && \
                                  (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
  #define IS_SKL_GT4(dev_priv)   (IS_SKYLAKE(dev_priv) && \
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
  int i915_perf_open_ioctl(struct drm_device *dev, void *data,
                          struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+                           struct i915_gem_context *ctx,
+                           uint32_t *reg_state);
 
  /* i915_gem_evict.c */
  int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +557 -35
index 3277a52ce98e..f2688dee5a03 100644
@@ -196,6 +196,12 @@
 
  #include "i915_drv.h"
  #include "i915_oa_hsw.h"
+#include "i915_oa_bdw.h"
+#include "i915_oa_chv.h"
+#include "i915_oa_sklgt2.h"
+#include "i915_oa_sklgt3.h"
+#include "i915_oa_sklgt4.h"
+#include "i915_oa_bxt.h"
 
  /* HW requires this to be a power of two, between 128k and 16M, though driver
   * is currently generally designed assuming the largest 16M size is used such
@@ -215,7 +221,7 @@
   *
   * Although this can be observed explicitly while copying reports to userspace
   * by checking for a zeroed report-id field in tail reports, we want to account
- * for this earlier, as part of the _oa_buffer_check to avoid lots of redundant
+ * for this earlier, as part of the oa_buffer_check to avoid lots of redundant
   * read() attempts.
   *
   * In effect we define a tail pointer for reading that lags the real tail
@@ -237,7 +243,7 @@
   * indicates that an updated tail pointer is needed.
   *
   * Most of the implementation details for this workaround are in
- * gen7_oa_buffer_check_unlocked() and gen7_appand_oa_reports()
+ * oa_buffer_check_unlocked() and _append_oa_reports()
   *
   * Note for posterity: previously the driver used to define an effective tail
   * pointer that lagged the real pointer by a 'tail margin' measured in bytes
@@ -272,6 +278,13 @@ static u32 i915_perf_stream_paranoid = true;
 
  #define INVALID_CTX_ID 0xffffffff
 
+/* On Gen8+ automatically triggered OA reports include a 'reason' field... */
+#define OAREPORT_REASON_MASK           0x3f
+#define OAREPORT_REASON_SHIFT          19
+#define OAREPORT_REASON_TIMER          (1<<0)
+#define OAREPORT_REASON_CTX_SWITCH     (1<<3)
+#define OAREPORT_REASON_CLK_RATIO      (1<<5)
+
 
  /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
   *
@@ -303,6 +316,13 @@ static struct i915_oa_format hsw_oa_formats[I915_OA_FORMAT_MAX] = {
         [I915_OA_FORMAT_C4_B8]      = { 7, 64 },
  };
 
+static struct i915_oa_format gen8_plus_oa_formats[I915_OA_FORMAT_MAX] = {
+       [I915_OA_FORMAT_A12]                = { 0, 64 },
+       [I915_OA_FORMAT_A12_B8_C8]          = { 2, 128 },
+       [I915_OA_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 },
+       [I915_OA_FORMAT_C4_B8]              = { 7, 64 },
+};
+
  #define SAMPLE_OA_REPORT      (1<<0)
 
  /**
@@ -332,8 +352,20 @@ struct perf_open_properties {
         int oa_period_exponent;
  };
 
+static u32 gen8_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+       return I915_READ(GEN8_OATAILPTR) & GEN8_OATAILPTR_MASK;
+}
+
+static u32 gen7_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+       u32 oastatus1 = I915_READ(GEN7_OASTATUS1);
+
+       return oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+}
+
  /**
- * gen7_oa_buffer_check_unlocked - check for data and update tail ptr state
+ * oa_buffer_check_unlocked - check for data and update tail ptr state
   * @dev_priv: i915 device instance
   *
   * This is either called via fops (for blocking reads in user ctx) or the poll
@@ -356,12 +388,11 @@ struct perf_open_properties {
   *
   * Returns: %true if the OA buffer contains data, else %false
   */
-static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
+static bool oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
  {
         int report_size = dev_priv->perf.oa.oa_buffer.format_size;
         unsigned long flags;
         unsigned int aged_idx;
-       u32 oastatus1;
         u32 head, hw_tail, aged_tail, aging_tail;
         u64 now;
 
@@ -381,8 +412,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
         aged_tail = dev_priv->perf.oa.oa_buffer.tails[aged_idx].offset;
         aging_tail = dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset;
 
-       oastatus1 = I915_READ(GEN7_OASTATUS1);
-       hw_tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+       hw_tail = dev_priv->perf.oa.ops.oa_hw_tail_read(dev_priv);
 
         /* The tail pointer increases in 64 byte increments,
          * not in report_size steps...
@@ -404,6 +434,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
         if (aging_tail != INVALID_TAIL_PTR &&
             ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
              OA_TAIL_MARGIN_NSEC)) {
+
                 aged_idx ^= 1;
                 dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
 
@@ -553,6 +584,287 @@ static int append_oa_sample(struct i915_perf_stream *stream,
   *
   * Returns: 0 on success, negative error code on failure.
   */
+static int gen8_append_oa_reports(struct i915_perf_stream *stream,
+                                 char __user *buf,
+                                 size_t count,
+                                 size_t *offset)
+{
+       struct drm_i915_private *dev_priv = stream->dev_priv;
+       int report_size = dev_priv->perf.oa.oa_buffer.format_size;
+       u8 *oa_buf_base = dev_priv->perf.oa.oa_buffer.vaddr;
+       u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+       u32 mask = (OA_BUFFER_SIZE - 1);
+       size_t start_offset = *offset;
+       unsigned long flags;
+       unsigned int aged_tail_idx;
+       u32 head, tail;
+       u32 taken;
+       int ret = 0;
+
+       if (WARN_ON(!stream->enabled))
+               return -EIO;
+
+       spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       head = dev_priv->perf.oa.oa_buffer.head;
+       aged_tail_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+       tail = dev_priv->perf.oa.oa_buffer.tails[aged_tail_idx].offset;
+
+       spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       /* An invalid tail pointer here means we're still waiting for the poll
+        * hrtimer callback to give us a pointer
+        */
+       if (tail == INVALID_TAIL_PTR)
+               return -EAGAIN;
+
+       /* NB: oa_buffer.head/tail include the gtt_offset which we don't want
+        * while indexing relative to oa_buf_base.
+        */
+       head -= gtt_offset;
+       tail -= gtt_offset;
+
+       /* An out of bounds or misaligned head or tail pointer implies a driver
+        * bug since we validate + align the tail pointers we read from the
+        * hardware and we are in full control of the head pointer which should
+        * only be incremented by multiples of the report size (notably also
+        * all a power of two).
+        */
+       if (WARN_ONCE(head > OA_BUFFER_SIZE || head % report_size ||
+                     tail > OA_BUFFER_SIZE || tail % report_size,
+                     "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
+                     head, tail))
+               return -EIO;
+
+
+       for (/* none */;
+            (taken = OA_TAKEN(tail, head));
+            head = (head + report_size) & mask) {
+               u8 *report = oa_buf_base + head;
+               u32 *report32 = (void *)report;
+               u32 ctx_id;
+               u32 reason;
+
+               /* All the report sizes factor neatly into the buffer
+                * size so we never expect to see a report split
+                * between the beginning and end of the buffer.
+                *
+                * Given the initial alignment check a misalignment
+                * here would imply a driver bug that would result
+                * in an overrun.
+                */
+               if (WARN_ON((OA_BUFFER_SIZE - head) < report_size)) {
+                       DRM_ERROR("Spurious OA head ptr: non-integral report offset\n");
+                       break;
+               }
+
+               /* The reason field includes flags identifying what
+                * triggered this specific report (mostly timer
+                * triggered or e.g. due to a context switch).
+                *
+                * This field is never expected to be zero so we can
+                * check that the report isn't invalid before copying
+                * it to userspace...
+                */
+               reason = ((report32[0] >> OAREPORT_REASON_SHIFT) &
+                         OAREPORT_REASON_MASK);
+               if (reason == 0) {
+                       if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+                               DRM_NOTE("Skipping spurious, invalid OA report\n");
+                       continue;
+               }
+
+               /* XXX: Just keep the lower 21 bits for now since I'm not
+                * entirely sure if the HW touches any of the higher bits in
+                * this field
+                */
+               ctx_id = report32[2] & 0x1fffff;
+
+               /* Squash whatever is in the CTX_ID field if it's marked as
+                * invalid to be sure we avoid false-positive, single-context
+                * filtering below...
+                *
+                * Note: that we don't clear the valid_ctx_bit so userspace can
+                * understand that the ID has been squashed by the kernel.
+                */
+               if (!(report32[0] & dev_priv->perf.oa.gen8_valid_ctx_bit))
+                       ctx_id = report32[2] = INVALID_CTX_ID;
+
+               /* NB: For Gen 8 the OA unit no longer supports clock gating
+                * off for a specific context and the kernel can't securely
+                * stop the counters from updating as system-wide / global
+                * values.
+                *
+                * Automatic reports now include a context ID so reports can be
+                * filtered on the cpu but it's not worth trying to
+                * automatically subtract/hide counter progress for other
+                * contexts while filtering since we can't stop userspace
+                * issuing MI_REPORT_PERF_COUNT commands which would still
+                * provide a side-band view of the real values.
+                *
+                * To allow userspace (such as Mesa/GL_INTEL_performance_query)
+                * to normalize counters for a single filtered context then it
+                * needs be forwarded bookend context-switch reports so that it
+                * can track switches in between MI_REPORT_PERF_COUNT commands
+                * and can itself subtract/ignore the progress of counters
+                * associated with other contexts. Note that the hardware
+                * automatically triggers reports when switching to a new
+                * context which are tagged with the ID of the newly active
+                * context. To avoid the complexity (and likely fragility) of
+                * reading ahead while parsing reports to try and minimize
+                * forwarding redundant context switch reports (i.e. between
+                * other, unrelated contexts) we simply elect to forward them
+                * all.
+                *
+                * We don't rely solely on the reason field to identify context
+                * switches since it's not-uncommon for periodic samples to
+                * identify a switch before any 'context switch' report.
+                */
+               if (!dev_priv->perf.oa.exclusive_stream->ctx ||
+                   dev_priv->perf.oa.specific_ctx_id == ctx_id ||
+                   (dev_priv->perf.oa.oa_buffer.last_ctx_id ==
+                    dev_priv->perf.oa.specific_ctx_id) ||
+                   reason & OAREPORT_REASON_CTX_SWITCH) {
+
+                       /* While filtering for a single context we avoid
+                        * leaking the IDs of other contexts.
+                        */
+                       if (dev_priv->perf.oa.exclusive_stream->ctx &&
+                           dev_priv->perf.oa.specific_ctx_id != ctx_id) {
+                               report32[2] = INVALID_CTX_ID;
+                       }
+
+                       ret = append_oa_sample(stream, buf, count, offset,
+                                              report);
+                       if (ret)
+                               break;
+
+                       dev_priv->perf.oa.oa_buffer.last_ctx_id = ctx_id;
+               }
+
+               /* The above reason field sanity check is based on
+                * the assumption that the OA buffer is initially
+                * zeroed and we reset the field after copying so the
+                * check is still meaningful once old reports start
+                * being overwritten.
+                */
+               report32[0] = 0;
+       }
+
+       if (start_offset != *offset) {
+               spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+               /* We removed the gtt_offset for the copy loop above, indexing
+                * relative to oa_buf_base so put back here...
+                */
+               head += gtt_offset;
+
+               I915_WRITE(GEN8_OAHEADPTR, head & GEN8_OAHEADPTR_MASK);
+               dev_priv->perf.oa.oa_buffer.head = head;
+
+               spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+       }
+
+       return ret;
+}
+
+/**
+ * gen8_oa_read - copy status records then buffered OA reports
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Checks OA unit status registers and if necessary appends corresponding
+ * status records for userspace (such as for a buffer full condition) and then
+ * initiate appending any buffered OA reports.
+ *
+ * Updates @offset according to the number of bytes successfully copied into
+ * the userspace buffer.
+ *
+ * NB: some data may be successfully copied to the userspace buffer
+ * even if an error is returned, and this is reflected in the
+ * updated @offset.
+ *
+ * Returns: zero on success or a negative error code
+ */
+static int gen8_oa_read(struct i915_perf_stream *stream,
+                       char __user *buf,
+                       size_t count,
+                       size_t *offset)
+{
+       struct drm_i915_private *dev_priv = stream->dev_priv;
+       u32 oastatus;
+       int ret;
+
+       if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
+               return -EIO;
+
+       oastatus = I915_READ(GEN8_OASTATUS);
+
+       /* We treat OABUFFER_OVERFLOW as a significant error:
+        *
+        * Although theoretically we could handle this more gracefully
+        * sometimes, some Gens don't correctly suppress certain
+        * automatically triggered reports in this condition and so we
+        * have to assume that old reports are now being trampled
+        * over.
+        *
+        * Considering how we don't currently give userspace control
+        * over the OA buffer size and always configure a large 16MB
+        * buffer, then a buffer overflow does anyway likely indicate
+        * that something has gone quite badly wrong.
+        */
+       if (oastatus & GEN8_OASTATUS_OABUFFER_OVERFLOW) {
+               ret = append_oa_status(stream, buf, count, offset,
+                                      DRM_I915_PERF_RECORD_OA_BUFFER_LOST);
+               if (ret)
+                       return ret;
+
+               DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+                         dev_priv->perf.oa.period_exponent);
+
+               dev_priv->perf.oa.ops.oa_disable(dev_priv);
+               dev_priv->perf.oa.ops.oa_enable(dev_priv);
+
+               /* Note: .oa_enable() is expected to re-init the oabuffer
+                * and reset GEN8_OASTATUS for us
+                */
+               oastatus = I915_READ(GEN8_OASTATUS);
+       }
+
+       if (oastatus & GEN8_OASTATUS_REPORT_LOST) {
+               ret = append_oa_status(stream, buf, count, offset,
+                                      DRM_I915_PERF_RECORD_OA_REPORT_LOST);
+               if (ret)
+                       return ret;
+               I915_WRITE(GEN8_OASTATUS,
+                          oastatus & ~GEN8_OASTATUS_REPORT_LOST);
+       }
+
+       return gen8_append_oa_reports(stream, buf, count, offset);
+}
+
+/**
+ * Copies all buffered OA reports into userspace read() buffer.
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Notably any error condition resulting in a short read (-%ENOSPC or
+ * -%EFAULT) will be returned even though one or more records may
+ * have been successfully copied. In this case it's up to the caller
+ * to decide if the error should be squashed before returning to
+ * userspace.
+ *
+ * Note: reports are consumed from the head, and appended to the
+ * tail, so the tail chases the head?... If you think that's mad
+ * and back-to-front you're not alone, but this follows the
+ * Gen PRM naming convention.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
  static int gen7_append_oa_reports(struct i915_perf_stream *stream,
                                   char __user *buf,
                                   size_t count,
@@ -733,7 +1045,8 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
                 if (ret)
                         return ret;
 
-               DRM_DEBUG("OA buffer overflow: force restart\n");
+               DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+                         dev_priv->perf.oa.period_exponent);
 
                 dev_priv->perf.oa.ops.oa_disable(dev_priv);
                 dev_priv->perf.oa.ops.oa_enable(dev_priv);
@@ -776,7 +1089,7 @@ static int i915_oa_wait_unlocked(struct i915_perf_stream *stream)
                 return -EIO;
 
         return wait_event_interruptible(dev_priv->perf.oa.poll_wq,
-                                       dev_priv->perf.oa.ops.oa_buffer_check(dev_priv));
+                                       oa_buffer_check_unlocked(dev_priv));
  }
 
  /**
@@ -833,33 +1146,37 @@ static int i915_oa_read(struct i915_perf_stream *stream,
  static int oa_get_render_ctx_id(struct i915_perf_stream *stream)
  {
         struct drm_i915_private *dev_priv = stream->dev_priv;
-       struct intel_engine_cs *engine = dev_priv->engine[RCS];
-       int ret;
 
-       ret = i915_mutex_lock_interruptible(&dev_priv->drm);
-       if (ret)
-               return ret;
+       if (i915.enable_execlists)
+               dev_priv->perf.oa.specific_ctx_id = stream->ctx->hw_id;
+       else {
+               struct intel_engine_cs *engine = dev_priv->engine[RCS];
+               int ret;
 
-       /* As the ID is the gtt offset of the context's vma we pin
-        * the vma to ensure the ID remains fixed.
-        *
-        * NB: implied RCS engine...
-        */
-       ret = engine->context_pin(engine, stream->ctx);
-       if (ret)
-               goto unlock;
+               ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+               if (ret)
+                       return ret;
 
-       /* Explicitly track the ID (instead of calling i915_ggtt_offset()
-        * on the fly) considering the difference with gen8+ and
-        * execlists
-        */
-       dev_priv->perf.oa.specific_ctx_id =
-               i915_ggtt_offset(stream->ctx->engine[engine->id].state);
+               /* As the ID is the gtt offset of the context's vma we pin
+                * the vma to ensure the ID remains fixed.
+                */
+               ret = engine->context_pin(engine, stream->ctx);
+               if (ret) {
+                       mutex_unlock(&dev_priv->drm.struct_mutex);
+                       return ret;
+               }
 
-unlock:
-       mutex_unlock(&dev_priv->drm.struct_mutex);
+               /* Explicitly track the ID (instead of calling
+                * i915_ggtt_offset() on the fly) considering the difference
+                * with gen8+ and execlists
+                */
+               dev_priv->perf.oa.specific_ctx_id =
+                       i915_ggtt_offset(stream->ctx->engine[engine->id].state);
 
-       return ret;
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+       }
+
+       return 0;
  }
 
  /**
@@ -874,12 +1191,16 @@ static void oa_put_render_ctx_id(struct i915_perf_stream *stream)
         struct drm_i915_private *dev_priv = stream->dev_priv;
         struct intel_engine_cs *engine = dev_priv->engine[RCS];
 
-       mutex_lock(&dev_priv->drm.struct_mutex);
+       if (i915.enable_execlists) {
+               dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+       } else {
+               mutex_lock(&dev_priv->drm.struct_mutex);
 
-       dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
-       engine->context_unpin(engine, stream->ctx);
+               dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+               engine->context_unpin(engine, stream->ctx);
 
-       mutex_unlock(&dev_priv->drm.struct_mutex);
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+       }
  }
 
  static void
@@ -969,6 +1290,61 @@ static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
         dev_priv->perf.oa.pollin = false;
  }
 
+static void gen8_init_oa_buffer(struct drm_i915_private *dev_priv)
+{
+       u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+       unsigned long flags;
+
+       spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       I915_WRITE(GEN8_OASTATUS, 0);
+       I915_WRITE(GEN8_OAHEADPTR, gtt_offset);
+       dev_priv->perf.oa.oa_buffer.head = gtt_offset;
+
+       I915_WRITE(GEN8_OABUFFER_UDW, 0);
+
+       /* PRM says:
+        *
+        *  "This MMIO must be set before the OATAILPTR
+        *  register and after the OAHEADPTR register. This is
+        *  to enable proper functionality of the overflow
+        *  bit."
+        */
+       I915_WRITE(GEN8_OABUFFER, gtt_offset |
+                  OABUFFER_SIZE_16M | OA_MEM_SELECT_GGTT);
+       I915_WRITE(GEN8_OATAILPTR, gtt_offset & GEN8_OATAILPTR_MASK);
+
+       /* Mark that we need updated tail pointers to read from... */
+       dev_priv->perf.oa.oa_buffer.tails[0].offset = INVALID_TAIL_PTR;
+       dev_priv->perf.oa.oa_buffer.tails[1].offset = INVALID_TAIL_PTR;
+
+       /* Reset state used to recognise context switches, affecting which
+        * reports we will forward to userspace while filtering for a single
+        * context.
+        */
+       dev_priv->perf.oa.oa_buffer.last_ctx_id = INVALID_CTX_ID;
+
+       spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       /* NB: although the OA buffer will initially be allocated
+        * zeroed via shmfs (and so this memset is redundant when
+        * first allocating), we may re-init the OA buffer, either
+        * when re-enabling a stream or in error/reset paths.
+        *
+        * The reason we clear the buffer for each re-init is for the
+        * sanity check in gen8_append_oa_reports() that looks at the
+        * reason field to make sure it's non-zero which relies on
+        * the assumption that new reports are being written to zeroed
+        * memory...
+        */
+       memset(dev_priv->perf.oa.oa_buffer.vaddr, 0, OA_BUFFER_SIZE);
+
+       /* Maybe make ->pollin per-stream state if we support multiple
+        * concurrent streams in the future.
+        */
+       dev_priv->perf.oa.pollin = false;
+}
+
  static int alloc_oa_buffer(struct drm_i915_private *dev_priv)
  {
         struct drm_i915_gem_object *bo;
@@ -1113,6 +1489,280 @@ static void hsw_disable_metric_set(struct drm_i915_private *dev_priv)
                                       ~GT_NOA_ENABLE));
  }
 
+/*
+ * From Broadwell PRM, 3D-Media-GPGPU -> Register State Context
+ *
+ * MMIO reads or writes to any of the registers listed in the
+ * “Register State Context image” subsections through HOST/IA
+ * MMIO interface for debug purposes must follow the steps below:
+ *
+ * - SW should set the Force Wakeup bit to prevent GT from entering C6.
+ * - Write 0x2050[31:0] = 0x00010001 (disable sequence).
+ * - Disable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010001).
+ * - BDW:  Poll/Wait for register bits of 0x22AC[6:0] turn to 0x30 value.
+ * - SKL+: Poll/Wait for register bits of 0x22A4[6:0] turn to 0x30 value.
+ * - Read/Write to desired MMIO registers.
+ * - Enable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010000).
+ * - Force Wakeup bit should be reset to enable C6 entry.
+ *
+ * XXX: don't nest or overlap calls to this API, it has no ref
+ * counting to track how many entities require the RCS to be
+ * blocked from being idle.
+ */
+static int gen8_begin_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+       i915_reg_t fsm_reg = INTEL_GEN(dev_priv) > 8 ?
+               GEN9_RCS_FE_FSM2 : GEN6_RCS_PWR_FSM;
+       int ret = 0;
+
+       /* There's only no active context while idle in execlist mode
+        * (though we shouldn't be using this in any other case)
+        */
+       if (WARN_ON(!i915.enable_execlists))
+               return -EINVAL;
+
+       intel_uncore_forcewake_get(dev_priv, FORCEWAKE_RENDER);
+
+       I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+                  _MASKED_BIT_ENABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+
+       /* Note: we don't currently have a good handle on the maximum
+        * latency for this wake up so while we only need to hold rcs
+        * busy from process context we at least keep the waiting
+        * interruptible...
+        */
+       ret = intel_wait_for_register(dev_priv, fsm_reg, 0x3f, 0x30, 50);
+       if (ret == -ETIMEDOUT)
+               DRM_ERROR("Timeout waiting to be able to begin per-ctx mmio\n");
+
+       if (ret)
+               intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+
+       return ret;
+}
+
+static void gen8_end_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+       if (WARN_ON(!i915.enable_execlists))
+               return;
+
+       I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+                  _MASKED_BIT_DISABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+       intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+}
+
+/* NB: It must always remain pointer safe to run this even if the OA unit
+ * has been disabled.
+ *
+ * It's fine to put out-of-date values into these per-context registers
+ * in the case that the OA unit has been disabled.
+ */
+static void gen8_update_reg_state_unlocked(struct i915_gem_context *ctx,
+                                          u32 *reg_state)
+{
+       struct drm_i915_private *dev_priv = ctx->i915;
+       const struct i915_oa_reg *flex_regs = dev_priv->perf.oa.flex_regs;
+       int n_flex_regs = dev_priv->perf.oa.flex_regs_len;
+       u32 ctx_oactxctrl = dev_priv->perf.oa.ctx_oactxctrl_offset;
+       u32 ctx_flexeu0 = dev_priv->perf.oa.ctx_flexeu0_offset;
+       /* The MMIO offsets for Flex EU registers aren't contiguous */
+       u32 flex_mmio[] = {
+               i915_mmio_reg_offset(EU_PERF_CNTL0),
+               i915_mmio_reg_offset(EU_PERF_CNTL1),
+               i915_mmio_reg_offset(EU_PERF_CNTL2),
+               i915_mmio_reg_offset(EU_PERF_CNTL3),
+               i915_mmio_reg_offset(EU_PERF_CNTL4),
+               i915_mmio_reg_offset(EU_PERF_CNTL5),
+               i915_mmio_reg_offset(EU_PERF_CNTL6),
+       };
+       int i;
+
+       reg_state[ctx_oactxctrl] = i915_mmio_reg_offset(GEN8_OACTXCONTROL);
+       reg_state[ctx_oactxctrl+1] = (dev_priv->perf.oa.period_exponent <<
+                                     GEN8_OA_TIMER_PERIOD_SHIFT) |
+                                    (dev_priv->perf.oa.periodic ?
+                                     GEN8_OA_TIMER_ENABLE : 0) |
+                                    GEN8_OA_COUNTER_RESUME;
+
+       for (i = 0; i < ARRAY_SIZE(flex_mmio); i++) {
+               u32 state_offset = ctx_flexeu0 + i * 2;
+               u32 mmio = flex_mmio[i];
+
+               /* This arbitrary default will select the 'EU FPU0 Pipeline
+                * Active' event. In the future it's anticipated that there
+                * will be an explicit 'No Event' we can select, but not yet...
+                */
+               u32 value = 0;
+               int j;
+
+               for (j = 0; j < n_flex_regs; j++) {
+                       if (i915_mmio_reg_offset(flex_regs[j].addr) == mmio) {
+                               value = flex_regs[j].value;
+                               break;
+                       }
+               }
+
+               reg_state[state_offset] = mmio;
+               reg_state[state_offset+1] = value;
+       }
+}
+
+/* Manages updating the per-context aspects of the OA stream
+ * configuration across all contexts.
+ *
+ * The awkward consideration here is that OACTXCONTROL controls the
+ * exponent for periodic sampling which is primarily used for system
+ * wide profiling where we'd like a consistent sampling period even in
+ * the face of context switches.
+ *
+ * Our approach of updating the register state context (as opposed to
+ * say using a workaround batch buffer) ensures that the hardware
+ * won't automatically reload an out-of-date timer exponent even
+ * transiently before a WA BB could be parsed.
+ *
+ * This function needs to:
+ * - Ensure the currently running context's per-context OA state is
+ *   updated
+ * - Ensure that all existing contexts will have the correct per-context
+ *   OA state if they are scheduled for use.
+ * - Ensure any new contexts will be initialized with the correct
+ *   per-context OA state.
+ *
+ * Note: it's only the RCS/Render context that has any OA state.
+ */
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{
+       struct i915_gem_context *ctx;
+       int ret;
+
Quoted 8 lines

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1644

Quoted 7 lines
drivers/gpu/drm/i915/i915_perf.c +7 -0
+       ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+       if (ret)
+               return ret;
+
+       /* Switch away from any user context.  */
+       ret = i915_gem_switch_to_kernel_context(dev_priv);
+       if (ret)

mutex_unlock(&dev_priv->drm.struct_mutex);

Duh! Thanks!

Quoted 38 lines
Quoted 38 lines
drivers/gpu/drm/i915/i915_perf.c +37 -0
+               return ret;
+
+       /* The OA register config is setup through the context image. This image
+        * might be written to by the GPU on context switch (in particular on
+        * lite-restore). This means we can't safely update a context's image,
+        * if this context is scheduled/submitted to run on the GPU.
+        *
+        * We could emit the OA register config through the batch buffer but
+        * this might leave small interval of time where the OA unit is
+        * configured at an invalid sampling period.
+        *
+        * So far the best way to work around this issue seems to be draining
+        * the GPU from any submitted work.
+        */
+       ret = i915_gem_wait_for_idle(dev_priv,
+                                    I915_WAIT_INTERRUPTIBLE |
+                                    I915_WAIT_LOCKED);
+       if (ret) {
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+               return ret;
+       }
+
+       /* Update all contexts now that we've stalled the submission. */
+       list_for_each_entry(ctx, &dev_priv->context_list, link) {
+               if (!ctx->engine[RCS].initialised)
+                       continue;
+
+               gen8_update_reg_state_unlocked(ctx,
+                                              ctx->engine[RCS].lrc_reg_state);
+       }
+
+       mutex_unlock(&dev_priv->drm.struct_mutex);
+
+       /* Now update the current context.
+        *
+        * Note: Using MMIO to update per-context registers requires
+        * some extra care...
Quoted 8 lines

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1687

Quoted 6 lines
drivers/gpu/drm/i915/i915_perf.c +6 -0
+        */
+       ret = gen8_begin_ctx_mmio(dev_priv);
+       if (ret) {
+               DRM_ERROR("Failed to bring RCS out of idle to update current ctx OA state\n");
+               return ret;
+       }

So do we still need the mmio for the current context, given that we now idle everything and then update the reg state?

I would say yes, because we don't have any guarantee that another context is going to be scheduled after that. Yet we want the config to be applied before we return.

Quoted 514 lines
Quoted 514 lines
drivers/gpu/drm/i915/i915_perf.c +279 -36
+
+       I915_WRITE(GEN8_OACTXCONTROL, ((dev_priv->perf.oa.period_exponent <<
+                                       GEN8_OA_TIMER_PERIOD_SHIFT) |
+                                     (dev_priv->perf.oa.periodic ?
+                                      GEN8_OA_TIMER_ENABLE : 0) |
+                                     GEN8_OA_COUNTER_RESUME));
+
+       config_oa_regs(dev_priv, dev_priv->perf.oa.flex_regs,
+                       dev_priv->perf.oa.flex_regs_len);
+
+       gen8_end_ctx_mmio(dev_priv);
+
+       return 0;
+}
+
+static int gen8_enable_metric_set(struct drm_i915_private *dev_priv)
+{
+       int ret = dev_priv->perf.oa.ops.select_metric_set(dev_priv);
+
+       if (ret)
+               return ret;
+
+       /* We disable slice/unslice clock ratio change reports on SKL since
+        * they are too noisy. The HW generates a lot of redundant reports
+        * where the ratio hasn't really changed causing a lot of redundant
+        * work to processes and increasing the chances we'll hit buffer
+        * overruns.
+        *
+        * Although we don't currently use the 'disable overrun' OABUFFER
+        * feature it's worth noting that clock ratio reports have to be
+        * disabled before considering to use that feature since the HW doesn't
+        * correctly block these reports.
+        *
+        * Currently none of the high-level metrics we have depend on knowing
+        * this ratio to normalize.
+        *
+        * Note: This register is not power context saved and restored, but
+        * that's OK considering that we disable RC6 while the OA unit is
+        * enabled.
+        *
+        * The _INCLUDE_CLK_RATIO bit allows the slice/unslice frequency to
+        * be read back from automatically triggered reports, as part of the
+        * RPT_ID field.
+        */
+       if (IS_SKYLAKE(dev_priv) || IS_BROXTON(dev_priv)) {
+               I915_WRITE(GEN8_OA_DEBUG,
+                          _MASKED_BIT_ENABLE(GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS |
+                                             GEN9_OA_DEBUG_INCLUDE_CLK_RATIO));
+       }
+
+       I915_WRITE(GDT_CHICKEN_BITS, 0xA0);
+       config_oa_regs(dev_priv, dev_priv->perf.oa.mux_regs,
+                      dev_priv->perf.oa.mux_regs_len);
+       I915_WRITE(GDT_CHICKEN_BITS, 0x80);
+
+       /* It takes a fairly long time for a new MUX configuration to
+        * be be applied after these register writes. This delay
+        * duration is take from Haswell (derived empirically based on
+        * the render_basic config) but hopefully it covers the
+        * maximum configuration latency for Gen8+ too...
+        */
+       usleep_range(15000, 20000);
+
+       config_oa_regs(dev_priv, dev_priv->perf.oa.b_counter_regs,
+                      dev_priv->perf.oa.b_counter_regs_len);
+
+       ret = gen8_configure_all_contexts(dev_priv);
+       if (ret)
+               return ret;
+
+       return 0;
+}
+
+static void gen8_disable_metric_set(struct drm_i915_private *dev_priv)
+{
+       /* NOP */
+}
+
  static void gen7_update_oacontrol_locked(struct drm_i915_private *dev_priv)
  {
         lockdep_assert_held(&dev_priv->perf.hook_lock);
@@ -1157,6 +1807,29 @@ static void gen7_oa_enable(struct drm_i915_private *dev_priv)
         spin_unlock_irqrestore(&dev_priv->perf.hook_lock, flags);
  }
 
+static void gen8_oa_enable(struct drm_i915_private *dev_priv)
+{
+       u32 report_format = dev_priv->perf.oa.oa_buffer.format;
+
+       /* Reset buf pointers so we don't forward reports from before now.
+        *
+        * Think carefully if considering trying to avoid this, since it
+        * also ensures status flags and the buffer itself are cleared
+        * in error paths, and we have checks for invalid reports based
+        * on the assumption that certain fields are written to zeroed
+        * memory which this helps maintains.
+        */
+       gen8_init_oa_buffer(dev_priv);
+
+       /* Note: we don't rely on the hardware to perform single context
+        * filtering and instead filter on the cpu based on the context-id
+        * field of reports
+        */
+       I915_WRITE(GEN8_OACONTROL, (report_format <<
+                                   GEN8_OA_REPORT_FORMAT_SHIFT) |
+                                  GEN8_OA_COUNTER_ENABLE);
+}
+
  /**
   * i915_oa_stream_enable - handle `I915_PERF_IOCTL_ENABLE` for OA stream
   * @stream: An i915 perf stream opened for OA metrics
@@ -1183,6 +1856,11 @@ static void gen7_oa_disable(struct drm_i915_private *dev_priv)
         I915_WRITE(GEN7_OACONTROL, 0);
  }
 
+static void gen8_oa_disable(struct drm_i915_private *dev_priv)
+{
+       I915_WRITE(GEN8_OACONTROL, 0);
+}
+
  /**
   * i915_oa_stream_disable - handle `I915_PERF_IOCTL_DISABLE` for OA stream
   * @stream: An i915 perf stream opened for OA metrics
@@ -1361,6 +2039,21 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream,
         return ret;
  }
 
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+                           struct i915_gem_context *ctx,
+                           u32 *reg_state)
+{
+       struct drm_i915_private *dev_priv = engine->i915;
+
+       if (engine->id != RCS)
+               return;
+
+       if (!dev_priv->perf.initialized)
+               return;
+
+       gen8_update_reg_state_unlocked(ctx, reg_state);
+}
+
  /**
   * i915_perf_read_locked - &i915_perf_stream_ops->read with error normalisation
   * @stream: An i915 perf stream
@@ -1486,7 +2179,7 @@ static enum hrtimer_restart oa_poll_check_timer_cb(struct hrtimer *hrtimer)
                 container_of(hrtimer, typeof(*dev_priv),
                              perf.oa.poll_check_timer);
 
-       if (dev_priv->perf.oa.ops.oa_buffer_check(dev_priv)) {
+       if (oa_buffer_check_unlocked(dev_priv)) {
                 dev_priv->perf.oa.pollin = true;
                 wake_up(&dev_priv->perf.oa.poll_wq);
         }
@@ -1775,6 +2468,7 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
         struct i915_gem_context *specific_ctx = NULL;
         struct i915_perf_stream *stream = NULL;
         unsigned long f_flags = 0;
+       bool privileged_op = true;
         int stream_fd;
         int ret;
 
@@ -1792,12 +2486,28 @@ i915_perf_open_ioctl_locked(struct drm_i915_private *dev_priv,
                 }
         }
 
+       /* On Haswell the OA unit supports clock gating off for a specific
+        * context and in this mode there's no visibility of metrics for the
+        * rest of the system, which we consider acceptable for a
+        * non-privileged client.
+        *
+        * For Gen8+ the OA unit no longer supports clock gating off for a
+        * specific context and the kernel can't securely stop the counters
+        * from updating as system-wide / global values. Even though we can
+        * filter reports based on the included context ID we can't block
+        * clients from seeing the raw / global counter values via
+        * MI_REPORT_PERF_COUNT commands and so consider it a privileged op to
+        * enable the OA unit by default.
+        */
+       if (IS_HASWELL(dev_priv) && specific_ctx)
+               privileged_op = false;
+
         /* Similar to perf's kernel.perf_paranoid_cpu sysctl option
          * we check a dev.i915.perf_stream_paranoid sysctl option
          * to determine if it's ok to access system wide OA counters
          * without CAP_SYS_ADMIN privileges.
          */
-       if (!specific_ctx &&
+       if (privileged_op &&
             i915_perf_stream_paranoid && !capable(CAP_SYS_ADMIN)) {
                 DRM_DEBUG("Insufficient privileges to open system-wide i915 perf stream\n");
                 ret = -EACCES;
@@ -2069,9 +2779,6 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data,
   */
  void i915_perf_register(struct drm_i915_private *dev_priv)
  {
-       if (!IS_HASWELL(dev_priv))
-               return;
-
         if (!dev_priv->perf.initialized)
                 return;
 
@@ -2087,11 +2794,38 @@ void i915_perf_register(struct drm_i915_private *dev_priv)
         if (!dev_priv->perf.metrics_kobj)
                 goto exit;
 
-       if (i915_perf_register_sysfs_hsw(dev_priv)) {
-               kobject_put(dev_priv->perf.metrics_kobj);
-               dev_priv->perf.metrics_kobj = NULL;
+       if (IS_HASWELL(dev_priv)) {
+               if (i915_perf_register_sysfs_hsw(dev_priv))
+                       goto sysfs_error;
+       } else if (IS_BROADWELL(dev_priv)) {
+               if (i915_perf_register_sysfs_bdw(dev_priv))
+                       goto sysfs_error;
+       } else if (IS_CHERRYVIEW(dev_priv)) {
+               if (i915_perf_register_sysfs_chv(dev_priv))
+                       goto sysfs_error;
+       } else if (IS_SKYLAKE(dev_priv)) {
+               if (IS_SKL_GT2(dev_priv)) {
+                       if (i915_perf_register_sysfs_sklgt2(dev_priv))
+                               goto sysfs_error;
+               } else if (IS_SKL_GT3(dev_priv)) {
+                       if (i915_perf_register_sysfs_sklgt3(dev_priv))
+                               goto sysfs_error;
+               } else if (IS_SKL_GT4(dev_priv)) {
+                       if (i915_perf_register_sysfs_sklgt4(dev_priv))
+                               goto sysfs_error;
+               } else
+                       goto sysfs_error;
+       } else if (IS_BROXTON(dev_priv)) {
+               if (i915_perf_register_sysfs_bxt(dev_priv))
+                       goto sysfs_error;
         }
 
+       goto exit;
+
+sysfs_error:
+       kobject_put(dev_priv->perf.metrics_kobj);
+       dev_priv->perf.metrics_kobj = NULL;
+
  exit:
         mutex_unlock(&dev_priv->perf.lock);
  }
@@ -2107,13 +2841,24 @@ void i915_perf_register(struct drm_i915_private *dev_priv)
   */
  void i915_perf_unregister(struct drm_i915_private *dev_priv)
  {
-       if (!IS_HASWELL(dev_priv))
-               return;
-
         if (!dev_priv->perf.metrics_kobj)
                 return;
 
-       i915_perf_unregister_sysfs_hsw(dev_priv);
+        if (IS_HASWELL(dev_priv))
+                i915_perf_unregister_sysfs_hsw(dev_priv);
+        else if (IS_BROADWELL(dev_priv))
+                i915_perf_unregister_sysfs_bdw(dev_priv);
+        else if (IS_CHERRYVIEW(dev_priv))
+                i915_perf_unregister_sysfs_chv(dev_priv);
+        else if (IS_SKYLAKE(dev_priv)) {
+               if (IS_SKL_GT2(dev_priv))
+                       i915_perf_unregister_sysfs_sklgt2(dev_priv);
+               else if (IS_SKL_GT3(dev_priv))
+                       i915_perf_unregister_sysfs_sklgt3(dev_priv);
+               else if (IS_SKL_GT4(dev_priv))
+                       i915_perf_unregister_sysfs_sklgt4(dev_priv);
+       } else if (IS_BROXTON(dev_priv))
+                i915_perf_unregister_sysfs_bxt(dev_priv);
 
         kobject_put(dev_priv->perf.metrics_kobj);
         dev_priv->perf.metrics_kobj = NULL;
@@ -2172,36 +2917,105 @@ static struct ctl_table dev_root[] = {
   */
  void i915_perf_init(struct drm_i915_private *dev_priv)
  {
-       if (!IS_HASWELL(dev_priv))
-               return;
-
-       hrtimer_init(&dev_priv->perf.oa.poll_check_timer,
-                    CLOCK_MONOTONIC, HRTIMER_MODE_REL);
-       dev_priv->perf.oa.poll_check_timer.function = oa_poll_check_timer_cb;
-       init_waitqueue_head(&dev_priv->perf.oa.poll_wq);
+       dev_priv->perf.oa.n_builtin_sets = 0;
+
+       if (IS_HASWELL(dev_priv)) {
+               dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
+               dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
+               dev_priv->perf.oa.ops.disable_metric_set = hsw_disable_metric_set;
+               dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
+               dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
+               dev_priv->perf.oa.ops.read = gen7_oa_read;
+               dev_priv->perf.oa.ops.oa_hw_tail_read =
+                       gen7_oa_hw_tail_read;
+
+               dev_priv->perf.oa.oa_formats = hsw_oa_formats;
+
+               dev_priv->perf.oa.n_builtin_sets =
+                       i915_oa_n_builtin_metric_sets_hsw;
+       } else if (i915.enable_execlists) {
+               /* Note: that although we could theoretically also support the
+                * legacy ringbuffer mode on BDW (and earlier iterations of
+                * this driver, before upstreaming did this) it didn't seem
+                * worth the complexity to maintain now that BDW+ enable
+                * execlist mode by default.
+                */
 
-       INIT_LIST_HEAD(&dev_priv->perf.streams);
-       mutex_init(&dev_priv->perf.lock);
-       spin_lock_init(&dev_priv->perf.hook_lock);
-       spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
+               if (IS_GEN8(dev_priv)) {
+                       dev_priv->perf.oa.ctx_oactxctrl_offset = 0x120;
+                       dev_priv->perf.oa.ctx_flexeu0_offset = 0x2ce;
+                       dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<25);
+
+                       if (IS_BROADWELL(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_bdw;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_bdw;
+                       } else if (IS_CHERRYVIEW(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_chv;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_chv;
+                       }
+               } else if (IS_GEN9(dev_priv)) {
+                       dev_priv->perf.oa.ctx_oactxctrl_offset = 0x128;
+                       dev_priv->perf.oa.ctx_flexeu0_offset = 0x3de;
+                       dev_priv->perf.oa.gen8_valid_ctx_bit = (1<<16);
+
+                       if (IS_SKL_GT2(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_sklgt2;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_sklgt2;
+                       } else if (IS_SKL_GT3(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_sklgt3;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_sklgt3;
+                       } else if (IS_SKL_GT4(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_sklgt4;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_sklgt4;
+                       } else if (IS_BROXTON(dev_priv)) {
+                               dev_priv->perf.oa.n_builtin_sets =
+                                       i915_oa_n_builtin_metric_sets_bxt;
+                               dev_priv->perf.oa.ops.select_metric_set =
+                                       i915_oa_select_metric_set_bxt;
+                       }
+               }
 
-       dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
-       dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
-       dev_priv->perf.oa.ops.disable_metric_set = hsw_disable_metric_set;
-       dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
-       dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
-       dev_priv->perf.oa.ops.read = gen7_oa_read;
-       dev_priv->perf.oa.ops.oa_buffer_check =
-               gen7_oa_buffer_check_unlocked;
+               if (dev_priv->perf.oa.n_builtin_sets) {
+                       dev_priv->perf.oa.ops.init_oa_buffer = gen8_init_oa_buffer;
+                       dev_priv->perf.oa.ops.enable_metric_set =
+                               gen8_enable_metric_set;
+                       dev_priv->perf.oa.ops.disable_metric_set =
+                               gen8_disable_metric_set;
+                       dev_priv->perf.oa.ops.oa_enable = gen8_oa_enable;
+                       dev_priv->perf.oa.ops.oa_disable = gen8_oa_disable;
+                       dev_priv->perf.oa.ops.read = gen8_oa_read;
+                       dev_priv->perf.oa.ops.oa_hw_tail_read =
+                               gen8_oa_hw_tail_read;
+
+                       dev_priv->perf.oa.oa_formats = gen8_plus_oa_formats;
+               }
+       }
 
-       dev_priv->perf.oa.oa_formats = hsw_oa_formats;
+       if (dev_priv->perf.oa.n_builtin_sets) {
+               hrtimer_init(&dev_priv->perf.oa.poll_check_timer,
+                               CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+               dev_priv->perf.oa.poll_check_timer.function = oa_poll_check_timer_cb;
+               init_waitqueue_head(&dev_priv->perf.oa.poll_wq);
 
-       dev_priv->perf.oa.n_builtin_sets =
-               i915_oa_n_builtin_metric_sets_hsw;
+               INIT_LIST_HEAD(&dev_priv->perf.streams);
+               mutex_init(&dev_priv->perf.lock);
+               spin_lock_init(&dev_priv->perf.hook_lock);
+               spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
 
-       dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
+               dev_priv->perf.sysctl_header = register_sysctl_table(dev_root);
 
-       dev_priv->perf.initialized = true;
+               dev_priv->perf.initialized = true;
+       }
  }
 
  /**
@@ -2216,5 +3030,6 @@ void i915_perf_fini(struct drm_i915_private *dev_priv)
         unregister_sysctl_table(dev_priv->perf.sysctl_header);
 
         memset(&dev_priv->perf.oa.ops, 0, sizeof(dev_priv->perf.oa.ops));
+
         dev_priv->perf.initialized = false;
  }
drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
  #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO           (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS      (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
  #define GEN8_OACONTROL _MMIO(0x2B00)
  #define  GEN8_OA_REPORT_FORMAT_A12         (0<<2)
  #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
  #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
  #define  GEN7_OABUFFER_RESUME              (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
  #define GEN8_OABUFFER _MMIO(0x2b14)
 
  #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
  #define  GEN8_OASTATUS_REPORT_LOST         (1<<0)
 
  #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
  #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
  #define OABUFFER_SIZE_128K  (0<<3)
  #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
  #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
  #define EU_PERF_CNTL0      _MMIO(0xe458)
+#define EU_PERF_CNTL1      _MMIO(0xe558)
+#define EU_PERF_CNTL2      _MMIO(0xe658)
+#define EU_PERF_CNTL3      _MMIO(0xe758)
+#define EU_PERF_CNTL4      _MMIO(0xe45c)
+#define EU_PERF_CNTL5      _MMIO(0xe55c)
+#define EU_PERF_CNTL6      _MMIO(0xe65c)
 
  #define GDT_CHICKEN_BITS    _MMIO(0x9840)
  #define GT_NOA_ENABLE      0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
  #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE        (1 << 12)
  #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE       (1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
  /* Fuse readout registers for GT */
  #define CHV_FUSE_GT                    _MMIO(VLV_DISPLAY_BASE + 0x2168)
  #define   CHV_FGT_DISABLE_SS0          (1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 7df278fe492e..351e231a02f4 100644
@@ -1879,6 +1879,8 @@ static void execlists_init_reg_state(u32 *regs,
                 regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
                 CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
                         make_rpcs(dev_priv));
+
+               i915_oa_init_reg_state(engine, ctx, regs);
         }
  }
 
include/uapi/drm/i915_drm.h +12 -7
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
  };
 
  enum drm_i915_oa_format {
-       I915_OA_FORMAT_A13 = 1,
-       I915_OA_FORMAT_A29,
-       I915_OA_FORMAT_A13_B8_C8,
-       I915_OA_FORMAT_B4_C8,
-       I915_OA_FORMAT_A45_B8_C8,
-       I915_OA_FORMAT_B4_C8_A16,
-       I915_OA_FORMAT_C4_B8,
+       I915_OA_FORMAT_A13 = 1,     /* HSW only */
+       I915_OA_FORMAT_A29,         /* HSW only */
+       I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+       I915_OA_FORMAT_B4_C8,       /* HSW only */
+       I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+       I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+       I915_OA_FORMAT_C4_B8,       /* HSW+ */
+
+       /* Gen8+ */
+       I915_OA_FORMAT_A12,
+       I915_OA_FORMAT_A12_B8_C8,
+       I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
Quoted 8 lines
Quoted 8 lines
include/uapi/drm/i915_drm.h +0 -1
         I915_OA_FORMAT_MAX          /* non-ABI */
  };
--

2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#36 lore raw

Re: [PATCH v6 12/15] drm/i915/perf: Add OA unit support for Gen 8+

Quoted 1,064 lines · Lionel Landwerlin
Quoted 995 lines · Matthew Auld
Quoted 992 lines · Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

---
  drivers/gpu/drm/i915/i915_drv.h  |  45 +-
  drivers/gpu/drm/i915/i915_perf.c | 957
++++++++++++++++++++++++++++++++++++---
  drivers/gpu/drm/i915/i915_reg.h  |  22 +
  drivers/gpu/drm/i915/intel_lrc.c |   2 +
  include/uapi/drm/i915_drm.h      |  19 +-
  5 files changed, 952 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +0 -1
index ffa1fc5eddfd..676b1227067c 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
         void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
         /**
-        * @enable_metric_set: Applies any MUX configuration to set up

the

drivers/gpu/drm/i915/i915_drv.h +1 -2
-        * Boolean and Custom (B/C) counters that are part of the counter
-        * reports being sampled. May apply system constraints such as
+        * @select_metric_set: The auto generated code that checks

whether a
+ * requested OA config is applicable to the system and if so sets
up
+ * the mux, oa and flex eu register config pointers according to
the

drivers/gpu/drm/i915/i915_drv.h +6 -0
+        * current dev_priv->perf.oa.metrics_set.
+        */
+       int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+       /**
+        * @enable_metric_set: Selects and applies any MUX configuration

to set

drivers/gpu/drm/i915/i915_drv.h +2 -0
+        * up the Boolean and Custom (B/C) counters that are part of the
+        * counter reports being sampled. May apply system constraints

such as

          * disabling EU clock gating as required.
          */
         int (*enable_metric_set)(struct drm_i915_private *dev_priv);
drivers/gpu/drm/i915/i915_drv.h +0 -3
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
                     size_t *offset);
 
         /**
-        * @oa_buffer_check: Check for OA buffer data + update tail
-        *
-        * This is either called via fops or the poll check hrtimer

(atomic

drivers/gpu/drm/i915/i915_drv.h +1 -2
-        * ctx) without any locks taken.
+        * @oa_hw_tail_read: read the OA tail pointer register
          *
-        * It's safe to read OA config state here unlocked, assuming that

this
- * is only called while the stream is enabled, while the global
OA

drivers/gpu/drm/i915/i915_drv.h +0 -3
-        * configuration can't be modified.
-        *
-        * Efficiency is more important than avoiding some false

positives
- * here, which will be handled gracefully - likely resulting in
an

drivers/gpu/drm/i915/i915_drv.h +9 -2
-        * %EAGAIN error for userspace.
+        * In particular this enables us to share all the fiddly code for
+        * handling the OA unit tail pointer race that affects multiple
+        * generations.
          */
-       bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+       u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
  };
 
  struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
                         struct {
                                 struct i915_vma *vma;
                                 u8 *vaddr;
+                               u32 last_ctx_id;
                                 int format;
                                 int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
                         } oa_buffer;
 
                         u32 gen7_latched_oastatus1;
+                       u32 ctx_oactxctrl_offset;
+                       u32 ctx_flexeu0_offset;
+
+                       /* The RPT_ID/reason field for Gen8+ includes a

bit
+ * to determine if the CTX ID in the report is
valid
+ * but the specific bit differs between Gen 8 and
9

drivers/gpu/drm/i915/i915_drv.h +4 -0
+                        */
+                       u32 gen8_valid_ctx_bit;
 
                         struct i915_oa_ops ops;
                         const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
  #define IS_KBL_ULX(dev_priv)   (INTEL_DEVID(dev_priv) == 0x590E || \
                                  INTEL_DEVID(dev_priv) == 0x5915 || \
                                  INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)   (IS_SKYLAKE(dev_priv) && \
+                                (INTEL_DEVID(dev_priv) & 0x00F0) ==
 0x0010)
  #define IS_SKL_GT3(dev_priv)   (IS_SKYLAKE(dev_priv) && \
                                  (INTEL_DEVID(dev_priv) & 0x00F0) ==

0x0020)
#define IS_SKL_GT4(dev_priv) (IS_SKYLAKE(dev_priv) && \

drivers/gpu/drm/i915/i915_drv.h +3 -0
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct
 i915_gem_context *ctx,
 
  int i915_perf_open_ioctl(struct drm_device *dev, void *data,
                          struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+                           struct i915_gem_context *ctx,
+                           uint32_t *reg_state);
 
  /* i915_gem_evict.c */

int __must_check i915_gem_evict_something(struct i915_address_space
*vm,

drivers/gpu/drm/i915/i915_perf.c +6 -0
index 3277a52ce98e..f2688dee5a03 100644
@@ -196,6 +196,12 @@
 
  #include "i915_drv.h"
  #include "i915_oa_hsw.h"
+#include "i915_oa_bdw.h"
+#include "i915_oa_chv.h"
+#include "i915_oa_sklgt2.h"
+#include "i915_oa_sklgt3.h"
+#include "i915_oa_sklgt4.h"
+#include "i915_oa_bxt.h"
 
  /* HW requires this to be a power of two, between 128k and 16M, though

driver
* is currently generally designed assuming the largest 16M size is
used such

drivers/gpu/drm/i915/i915_perf.c +0 -0
@@ -215,7 +221,7 @@
   *
   * Although this can be observed explicitly while copying reports to

userspace
* by checking for a zeroed report-id field in tail reports, we want to
account
- * for this earlier, as part of the _oa_buffer_check to avoid lots of redundant
+ * for this earlier, as part of the oa_buffer_check to avoid lots of redundant
* read() attempts.
*
* In effect we define a tail pointer for reading that lags the real
tail

drivers/gpu/drm/i915/i915_perf.c +1 -1
@@ -237,7 +243,7 @@
   * indicates that an updated tail pointer is needed.
   *
   * Most of the implementation details for this workaround are in
- * gen7_oa_buffer_check_unlocked() and gen7_appand_oa_reports()
+ * oa_buffer_check_unlocked() and _append_oa_reports()
   *
   * Note for posterity: previously the driver used to define an

effective tail
* pointer that lagged the real pointer by a 'tail margin' measured in
bytes

drivers/gpu/drm/i915/i915_perf.c +1 -0
@@ -272,6 +278,13 @@ static u32 i915_perf_stream_paranoid = true;
 
  #define INVALID_CTX_ID 0xffffffff
 
+/* On Gen8+ automatically triggered OA reports include a 'reason'

field... */

drivers/gpu/drm/i915/i915_perf.c +13 -0
+#define OAREPORT_REASON_MASK           0x3f
+#define OAREPORT_REASON_SHIFT          19
+#define OAREPORT_REASON_TIMER          (1<<0)
+#define OAREPORT_REASON_CTX_SWITCH     (1<<3)
+#define OAREPORT_REASON_CLK_RATIO      (1<<5)
+
 
  /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
   *
@@ -303,6 +316,13 @@ static struct i915_oa_format
 hsw_oa_formats[I915_OA_FORMAT_MAX] = {
         [I915_OA_FORMAT_C4_B8]      = { 7, 64 },
  };
 
+static struct i915_oa_format gen8_plus_oa_formats[I915_OA_FORMAT_MAX] =
 {
+       [I915_OA_FORMAT_A12]                = { 0, 64 },
+       [I915_OA_FORMAT_A12_B8_C8]          = { 2, 128 },
+       [I915_OA_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 },
+       [I915_OA_FORMAT_C4_B8]              = { 7, 64 },
+};
+
  #define SAMPLE_OA_REPORT      (1<<0)

/**

drivers/gpu/drm/i915/i915_perf.c +12 -1
@@ -332,8 +352,20 @@ struct perf_open_properties {
         int oa_period_exponent;
  };
 
+static u32 gen8_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+       return I915_READ(GEN8_OATAILPTR) & GEN8_OATAILPTR_MASK;
+}
+
+static u32 gen7_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+       u32 oastatus1 = I915_READ(GEN7_OASTATUS1);
+
+       return oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+}
+
  /**
- * gen7_oa_buffer_check_unlocked - check for data and update tail ptr

state
+ * oa_buffer_check_unlocked - check for data and update tail ptr state
* @dev_priv: i915 device instance
*
* This is either called via fops (for blocking reads in user ctx) or
the poll

drivers/gpu/drm/i915/i915_perf.c +2 -4
@@ -356,12 +388,11 @@ struct perf_open_properties {
   *
   * Returns: %true if the OA buffer contains data, else %false
   */
-static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private
 *dev_priv)
+static bool oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
  {
         int report_size = dev_priv->perf.oa.oa_buffer.format_size;
         unsigned long flags;
         unsigned int aged_idx;
-       u32 oastatus1;
         u32 head, hw_tail, aged_tail, aging_tail;
         u64 now;
@@ -381,8 +412,7 @@ static bool gen7_oa_buffer_check_unlocked(struct
 drm_i915_private *dev_priv)
         aged_tail = dev_priv->perf.oa.oa_buffer.tails[aged_idx].offset;
         aging_tail =
 dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset;
 
-       oastatus1 = I915_READ(GEN7_OASTATUS1);
-       hw_tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+       hw_tail = dev_priv->perf.oa.ops.oa_hw_tail_read(dev_priv);
 

/* The tail pointer increases in 64 byte increments,
* not in report_size steps...

drivers/gpu/drm/i915/i915_perf.c +30 -0
@@ -404,6 +434,7 @@ static bool gen7_oa_buffer_check_unlocked(struct
 drm_i915_private *dev_priv)
         if (aging_tail != INVALID_TAIL_PTR &&
             ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
              OA_TAIL_MARGIN_NSEC)) {
+
                 aged_idx ^= 1;
                 dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
@@ -553,6 +584,287 @@ static int append_oa_sample(struct i915_perf_stream
 *stream,
   *
   * Returns: 0 on success, negative error code on failure.
   */
+static int gen8_append_oa_reports(struct i915_perf_stream *stream,
+                                 char __user *buf,
+                                 size_t count,
+                                 size_t *offset)
+{
+       struct drm_i915_private *dev_priv = stream->dev_priv;
+       int report_size = dev_priv->perf.oa.oa_buffer.format_size;
+       u8 *oa_buf_base = dev_priv->perf.oa.oa_buffer.vaddr;
+       u32 gtt_offset =
 i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+       u32 mask = (OA_BUFFER_SIZE - 1);
+       size_t start_offset = *offset;
+       unsigned long flags;
+       unsigned int aged_tail_idx;
+       u32 head, tail;
+       u32 taken;
+       int ret = 0;
+
+       if (WARN_ON(!stream->enabled))
+               return -EIO;
+
+       spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       head = dev_priv->perf.oa.oa_buffer.head;
+       aged_tail_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+       tail = dev_priv->perf.oa.oa_buffer.tails[aged_tail_idx].offset;
+
+       spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock,
 flags);
+
+       /* An invalid tail pointer here means we're still waiting for the

poll

drivers/gpu/drm/i915/i915_perf.c +6 -0
+        * hrtimer callback to give us a pointer
+        */
+       if (tail == INVALID_TAIL_PTR)
+               return -EAGAIN;
+
+       /* NB: oa_buffer.head/tail include the gtt_offset which we don't

want

drivers/gpu/drm/i915/i915_perf.c +6 -0
+        * while indexing relative to oa_buf_base.
+        */
+       head -= gtt_offset;
+       tail -= gtt_offset;
+
+       /* An out of bounds or misaligned head or tail pointer implies a

driver
+ * bug since we validate + align the tail pointers we read from
the
+ * hardware and we are in full control of the head pointer which
should
+ * only be incremented by multiples of the report size (notably
also

drivers/gpu/drm/i915/i915_perf.c +5 -0
+        * all a power of two).
+        */
+       if (WARN_ONCE(head > OA_BUFFER_SIZE || head % report_size ||
+                     tail > OA_BUFFER_SIZE || tail % report_size,
+                     "Inconsistent OA buffer pointers: head = %u, tail =

%u\n",

drivers/gpu/drm/i915/i915_perf.c +22 -0
+                     head, tail))
+               return -EIO;
+
+
+       for (/* none */;
+            (taken = OA_TAKEN(tail, head));
+            head = (head + report_size) & mask) {
+               u8 *report = oa_buf_base + head;
+               u32 *report32 = (void *)report;
+               u32 ctx_id;
+               u32 reason;
+
+               /* All the report sizes factor neatly into the buffer
+                * size so we never expect to see a report split
+                * between the beginning and end of the buffer.
+                *
+                * Given the initial alignment check a misalignment
+                * here would imply a driver bug that would result
+                * in an overrun.
+                */
+               if (WARN_ON((OA_BUFFER_SIZE - head) < report_size)) {
+                       DRM_ERROR("Spurious OA head ptr: non-integral

report offset\n");

drivers/gpu/drm/i915/i915_perf.c +15 -0
+                       break;
+               }
+
+               /* The reason field includes flags identifying what
+                * triggered this specific report (mostly timer
+                * triggered or e.g. due to a context switch).
+                *
+                * This field is never expected to be zero so we can
+                * check that the report isn't invalid before copying
+                * it to userspace...
+                */
+               reason = ((report32[0] >> OAREPORT_REASON_SHIFT) &
+                         OAREPORT_REASON_MASK);
+               if (reason == 0) {
+                       if

(__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+ DRM_NOTE("Skipping spurious, invalid OA
report\n");

drivers/gpu/drm/i915/i915_perf.c +5 -0
+                       continue;
+               }
+
+               /* XXX: Just keep the lower 21 bits for now since I'm not
+                * entirely sure if the HW touches any of the higher bits

in

drivers/gpu/drm/i915/i915_perf.c +5 -0
+                * this field
+                */
+               ctx_id = report32[2] & 0x1fffff;
+
+               /* Squash whatever is in the CTX_ID field if it's marked

as
+ * invalid to be sure we avoid false-positive,
single-context

drivers/gpu/drm/i915/i915_perf.c +3 -0
+                * filtering below...
+                *
+                * Note: that we don't clear the valid_ctx_bit so

userspace can
+ * understand that the ID has been squashed by the
kernel.

drivers/gpu/drm/i915/i915_perf.c +2 -0
+                */
+               if (!(report32[0] &

dev_priv->perf.oa.gen8_valid_ctx_bit))

drivers/gpu/drm/i915/i915_perf.c +3 -0
+                       ctx_id = report32[2] = INVALID_CTX_ID;
+
+               /* NB: For Gen 8 the OA unit no longer supports clock

gating
+ * off for a specific context and the kernel can't
securely
+ * stop the counters from updating as system-wide /
global

drivers/gpu/drm/i915/i915_perf.c +3 -0
+                * values.
+                *
+                * Automatic reports now include a context ID so reports

can be

drivers/gpu/drm/i915/i915_perf.c +4 -0
+                * filtered on the cpu but it's not worth trying to
+                * automatically subtract/hide counter progress for other
+                * contexts while filtering since we can't stop userspace
+                * issuing MI_REPORT_PERF_COUNT commands which would

still

drivers/gpu/drm/i915/i915_perf.c +3 -0
+                * provide a side-band view of the real values.
+                *
+                * To allow userspace (such as

Mesa/GL_INTEL_performance_query)
+ * to normalize counters for a single filtered context
then it
+ * needs be forwarded bookend context-switch reports so
that it
+ * can track switches in between MI_REPORT_PERF_COUNT
commands
+ * and can itself subtract/ignore the progress of
counters

drivers/gpu/drm/i915/i915_perf.c +3 -0
+                * associated with other contexts. Note that the hardware
+                * automatically triggers reports when switching to a new
+                * context which are tagged with the ID of the newly

active
+ * context. To avoid the complexity (and likely
fragility) of
+ * reading ahead while parsing reports to try and
minimize
+ * forwarding redundant context switch reports (i.e.
between
+ * other, unrelated contexts) we simply elect to forward
them

drivers/gpu/drm/i915/i915_perf.c +3 -0
+                * all.
+                *
+                * We don't rely solely on the reason field to identify

context
+ * switches since it's not-uncommon for periodic samples
to

drivers/gpu/drm/i915/i915_perf.c +13 -0
+                * identify a switch before any 'context switch' report.
+                */
+               if (!dev_priv->perf.oa.exclusive_stream->ctx ||
+                   dev_priv->perf.oa.specific_ctx_id == ctx_id ||
+                   (dev_priv->perf.oa.oa_buffer.last_ctx_id ==
+                    dev_priv->perf.oa.specific_ctx_id) ||
+                   reason & OAREPORT_REASON_CTX_SWITCH) {
+
+                       /* While filtering for a single context we avoid
+                        * leaking the IDs of other contexts.
+                        */
+                       if (dev_priv->perf.oa.exclusive_stream->ctx &&
+                           dev_priv->perf.oa.specific_ctx_id != ctx_id)

{

drivers/gpu/drm/i915/i915_perf.c +4 -0
+                               report32[2] = INVALID_CTX_ID;
+                       }
+
+                       ret = append_oa_sample(stream, buf, count,

offset,

drivers/gpu/drm/i915/i915_perf.c +18 -0
+                                              report);
+                       if (ret)
+                               break;
+
+                       dev_priv->perf.oa.oa_buffer.last_ctx_id = ctx_id;
+               }
+
+               /* The above reason field sanity check is based on
+                * the assumption that the OA buffer is initially
+                * zeroed and we reset the field after copying so the
+                * check is still meaningful once old reports start
+                * being overwritten.
+                */
+               report32[0] = 0;
+       }
+
+       if (start_offset != *offset) {
+               spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock,

flags);

drivers/gpu/drm/i915/i915_perf.c +2 -0
+
+               /* We removed the gtt_offset for the copy loop above,

indexing

drivers/gpu/drm/i915/i915_perf.c +8 -0
+                * relative to oa_buf_base so put back here...
+                */
+               head += gtt_offset;
+
+               I915_WRITE(GEN8_OAHEADPTR, head & GEN8_OAHEADPTR_MASK);
+               dev_priv->perf.oa.oa_buffer.head = head;
+
+

spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);

drivers/gpu/drm/i915/i915_perf.c +13 -0
+       }
+
+       return ret;
+}
+
+/**
+ * gen8_oa_read - copy status records then buffered OA reports
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Checks OA unit status registers and if necessary appends

corresponding
+ * status records for userspace (such as for a buffer full condition) and then
+ * initiate appending any buffered OA reports.
+ *
+ * Updates @offset according to the number of bytes successfully copied into
+ * the userspace buffer.
+ *
+ * NB: some data may be successfully copied to the userspace buffer
+ * even if an error is returned, and this is reflected in the
+ * updated @offset.
+ *
+ * Returns: zero on success or a negative error code
+ */

drivers/gpu/drm/i915/i915_perf.c +30 -0
+static int gen8_oa_read(struct i915_perf_stream *stream,
+                       char __user *buf,
+                       size_t count,
+                       size_t *offset)
+{
+       struct drm_i915_private *dev_priv = stream->dev_priv;
+       u32 oastatus;
+       int ret;
+
+       if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
+               return -EIO;
+
+       oastatus = I915_READ(GEN8_OASTATUS);
+
+       /* We treat OABUFFER_OVERFLOW as a significant error:
+        *
+        * Although theoretically we could handle this more gracefully
+        * sometimes, some Gens don't correctly suppress certain
+        * automatically triggered reports in this condition and so we
+        * have to assume that old reports are now being trampled
+        * over.
+        *
+        * Considering how we don't currently give userspace control
+        * over the OA buffer size and always configure a large 16MB
+        * buffer, then a buffer overflow does anyway likely indicate
+        * that something has gone quite badly wrong.
+        */
+       if (oastatus & GEN8_OASTATUS_OABUFFER_OVERFLOW) {
+               ret = append_oa_status(stream, buf, count, offset,
+

DRM_I915_PERF_RECORD_OA_BUFFER_LOST);

drivers/gpu/drm/i915/i915_perf.c +4 -0
+               if (ret)
+                       return ret;
+
+               DRM_DEBUG("OA buffer overflow (exponent = %d): force

restart\n",

drivers/gpu/drm/i915/i915_perf.c +14 -0
+                         dev_priv->perf.oa.period_exponent);
+
+               dev_priv->perf.oa.ops.oa_disable(dev_priv);
+               dev_priv->perf.oa.ops.oa_enable(dev_priv);
+
+               /* Note: .oa_enable() is expected to re-init the oabuffer
+                * and reset GEN8_OASTATUS for us
+                */
+               oastatus = I915_READ(GEN8_OASTATUS);
+       }
+
+       if (oastatus & GEN8_OASTATUS_REPORT_LOST) {
+               ret = append_oa_status(stream, buf, count, offset,
+

DRM_I915_PERF_RECORD_OA_REPORT_LOST);

drivers/gpu/drm/i915/i915_perf.c +31 -1
+               if (ret)
+                       return ret;
+               I915_WRITE(GEN8_OASTATUS,
+                          oastatus & ~GEN8_OASTATUS_REPORT_LOST);
+       }
+
+       return gen8_append_oa_reports(stream, buf, count, offset);
+}
+
+/**
+ * Copies all buffered OA reports into userspace read() buffer.
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Notably any error condition resulting in a short read (-%ENOSPC or
+ * -%EFAULT) will be returned even though one or more records may
+ * have been successfully copied. In this case it's up to the caller
+ * to decide if the error should be squashed before returning to
+ * userspace.
+ *
+ * Note: reports are consumed from the head, and appended to the
+ * tail, so the tail chases the head?... If you think that's mad
+ * and back-to-front you're not alone, but this follows the
+ * Gen PRM naming convention.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
  static int gen7_append_oa_reports(struct i915_perf_stream *stream,
                                   char __user *buf,
                                   size_t count,
@@ -733,7 +1045,8 @@ static int gen7_oa_read(struct i915_perf_stream
 *stream,
                 if (ret)
                         return ret;
 
-               DRM_DEBUG("OA buffer overflow: force restart\n");
+               DRM_DEBUG("OA buffer overflow (exponent = %d): force
 restart\n",
+                         dev_priv->perf.oa.period_exponent);
 
                 dev_priv->perf.oa.ops.oa_disable(dev_priv);
                 dev_priv->perf.oa.ops.oa_enable(dev_priv);
drivers/gpu/drm/i915/i915_perf.c +1 -1
@@ -776,7 +1089,7 @@ static int i915_oa_wait_unlocked(struct
 i915_perf_stream *stream)
                 return -EIO;
 
         return wait_event_interruptible(dev_priv->perf.oa.poll_wq,
-
 dev_priv->perf.oa.ops.oa_buffer_check(dev_priv));
+
 oa_buffer_check_unlocked(dev_priv));
  }

/**

drivers/gpu/drm/i915/i915_perf.c +9 -19
@@ -833,33 +1146,37 @@ static int i915_oa_read(struct i915_perf_stream
 *stream,
  static int oa_get_render_ctx_id(struct i915_perf_stream *stream)
  {
         struct drm_i915_private *dev_priv = stream->dev_priv;
-       struct intel_engine_cs *engine = dev_priv->engine[RCS];
-       int ret;
 
-       ret = i915_mutex_lock_interruptible(&dev_priv->drm);
-       if (ret)
-               return ret;
+       if (i915.enable_execlists)
+               dev_priv->perf.oa.specific_ctx_id = stream->ctx->hw_id;
+       else {
+               struct intel_engine_cs *engine = dev_priv->engine[RCS];
+               int ret;
 
-       /* As the ID is the gtt offset of the context's vma we pin
-        * the vma to ensure the ID remains fixed.
-        *
-        * NB: implied RCS engine...
-        */
-       ret = engine->context_pin(engine, stream->ctx);
-       if (ret)
-               goto unlock;
+               ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+               if (ret)
+                       return ret;
 
-       /* Explicitly track the ID (instead of calling i915_ggtt_offset()
-        * on the fly) considering the difference with gen8+ and
-        * execlists
-        */
-       dev_priv->perf.oa.specific_ctx_id =
-               i915_ggtt_offset(stream->ctx->engine[engine->id].state);
+               /* As the ID is the gtt offset of the context's vma we

pin

drivers/gpu/drm/i915/i915_perf.c +9 -2
+                * the vma to ensure the ID remains fixed.
+                */
+               ret = engine->context_pin(engine, stream->ctx);
+               if (ret) {
+                       mutex_unlock(&dev_priv->drm.struct_mutex);
+                       return ret;
+               }
 
-unlock:
-       mutex_unlock(&dev_priv->drm.struct_mutex);
+               /* Explicitly track the ID (instead of calling
+                * i915_ggtt_offset() on the fly) considering the

difference

drivers/gpu/drm/i915/i915_perf.c +4 -0
+                * with gen8+ and execlists
+                */
+               dev_priv->perf.oa.specific_ctx_id =
+

i915_ggtt_offset(stream->ctx->engine[engine->id].state);

drivers/gpu/drm/i915/i915_perf.c +12 -5
-       return ret;
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+       }
+
+       return 0;
  }
 
  /**
@@ -874,12 +1191,16 @@ static void oa_put_render_ctx_id(struct
 i915_perf_stream *stream)
         struct drm_i915_private *dev_priv = stream->dev_priv;
         struct intel_engine_cs *engine = dev_priv->engine[RCS];
 
-       mutex_lock(&dev_priv->drm.struct_mutex);
+       if (i915.enable_execlists) {
+               dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+       } else {
+               mutex_lock(&dev_priv->drm.struct_mutex);
 
-       dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
-       engine->context_unpin(engine, stream->ctx);
+               dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+               engine->context_unpin(engine, stream->ctx);
 
-       mutex_unlock(&dev_priv->drm.struct_mutex);
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+       }
  }
 

static void

drivers/gpu/drm/i915/i915_perf.c +29 -0
@@ -969,6 +1290,61 @@ static void gen7_init_oa_buffer(struct
 drm_i915_private *dev_priv)
         dev_priv->perf.oa.pollin = false;
  }
 
+static void gen8_init_oa_buffer(struct drm_i915_private *dev_priv)
+{
+       u32 gtt_offset =
 i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+       unsigned long flags;
+
+       spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+       I915_WRITE(GEN8_OASTATUS, 0);
+       I915_WRITE(GEN8_OAHEADPTR, gtt_offset);
+       dev_priv->perf.oa.oa_buffer.head = gtt_offset;
+
+       I915_WRITE(GEN8_OABUFFER_UDW, 0);
+
+       /* PRM says:
+        *
+        *  "This MMIO must be set before the OATAILPTR
+        *  register and after the OAHEADPTR register. This is
+        *  to enable proper functionality of the overflow
+        *  bit."
+        */
+       I915_WRITE(GEN8_OABUFFER, gtt_offset |
+                  OABUFFER_SIZE_16M | OA_MEM_SELECT_GGTT);
+       I915_WRITE(GEN8_OATAILPTR, gtt_offset & GEN8_OATAILPTR_MASK);
+
+       /* Mark that we need updated tail pointers to read from... */
+       dev_priv->perf.oa.oa_buffer.tails[0].offset = INVALID_TAIL_PTR;
+       dev_priv->perf.oa.oa_buffer.tails[1].offset = INVALID_TAIL_PTR;
+
+       /* Reset state used to recognise context switches, affecting

which
+ * reports we will forward to userspace while filtering for a
single

drivers/gpu/drm/i915/i915_perf.c +5 -0
+        * context.
+        */
+       dev_priv->perf.oa.oa_buffer.last_ctx_id = INVALID_CTX_ID;
+
+       spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock,

flags);

drivers/gpu/drm/i915/i915_perf.c +31 -0
+
+       /* NB: although the OA buffer will initially be allocated
+        * zeroed via shmfs (and so this memset is redundant when
+        * first allocating), we may re-init the OA buffer, either
+        * when re-enabling a stream or in error/reset paths.
+        *
+        * The reason we clear the buffer for each re-init is for the
+        * sanity check in gen8_append_oa_reports() that looks at the
+        * reason field to make sure it's non-zero which relies on
+        * the assumption that new reports are being written to zeroed
+        * memory...
+        */
+       memset(dev_priv->perf.oa.oa_buffer.vaddr, 0, OA_BUFFER_SIZE);
+
+       /* Maybe make ->pollin per-stream state if we support multiple
+        * concurrent streams in the future.
+        */
+       dev_priv->perf.oa.pollin = false;
+}
+
  static int alloc_oa_buffer(struct drm_i915_private *dev_priv)
  {
         struct drm_i915_gem_object *bo;
@@ -1113,6 +1489,280 @@ static void hsw_disable_metric_set(struct
 drm_i915_private *dev_priv)
                                       ~GT_NOA_ENABLE));
  }
 
+/*
+ * From Broadwell PRM, 3D-Media-GPGPU -> Register State Context
+ *
+ * MMIO reads or writes to any of the registers listed in the
+ * “Register State Context image” subsections through HOST/IA
+ * MMIO interface for debug purposes must follow the steps below:
+ *
+ * - SW should set the Force Wakeup bit to prevent GT from entering C6.
+ * - Write 0x2050[31:0] = 0x00010001 (disable sequence).
+ * - Disable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010001).
+ * - BDW:  Poll/Wait for register bits of 0x22AC[6:0] turn to 0x30

value.
+ * - SKL+: Poll/Wait for register bits of 0x22A4[6:0] turn to 0x30 value.
+ * - Read/Write to desired MMIO registers.
+ * - Enable IDLE messaging in CS (Write 0x2050[31:0] = 0x00010000).
+ * - Force Wakeup bit should be reset to enable C6 entry.
+ *
+ * XXX: don't nest or overlap calls to this API, it has no ref
+ * counting to track how many entities require the RCS to be
+ * blocked from being idle.
+ */

drivers/gpu/drm/i915/i915_perf.c +25 -0
+static int gen8_begin_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+       i915_reg_t fsm_reg = INTEL_GEN(dev_priv) > 8 ?
+               GEN9_RCS_FE_FSM2 : GEN6_RCS_PWR_FSM;
+       int ret = 0;
+
+       /* There's only no active context while idle in execlist mode
+        * (though we shouldn't be using this in any other case)
+        */
+       if (WARN_ON(!i915.enable_execlists))
+               return -EINVAL;
+
+       intel_uncore_forcewake_get(dev_priv, FORCEWAKE_RENDER);
+
+       I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+                  _MASKED_BIT_ENABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+
+       /* Note: we don't currently have a good handle on the maximum
+        * latency for this wake up so while we only need to hold rcs
+        * busy from process context we at least keep the waiting
+        * interruptible...
+        */
+       ret = intel_wait_for_register(dev_priv, fsm_reg, 0x3f, 0x30, 50);
+       if (ret == -ETIMEDOUT)
+               DRM_ERROR("Timeout waiting to be able to begin per-ctx

mmio\n");

drivers/gpu/drm/i915/i915_perf.c +18 -0
+
+       if (ret)
+               intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+
+       return ret;
+}
+
+static void gen8_end_ctx_mmio(struct drm_i915_private *dev_priv)
+{
+       if (WARN_ON(!i915.enable_execlists))
+               return;
+
+       I915_WRITE(GEN6_RC_SLEEP_PSMI_CONTROL,
+                  _MASKED_BIT_DISABLE(GEN6_PSMI_SLEEP_MSG_DISABLE));
+       intel_uncore_forcewake_put(dev_priv, FORCEWAKE_RENDER);
+}
+
+/* NB: It must always remain pointer safe to run this even if the OA

unit
+ * has been disabled.
+ *
+ * It's fine to put out-of-date values into these per-context registers
+ * in the case that the OA unit has been disabled.
+ */

drivers/gpu/drm/i915/i915_perf.c +5 -0
+static void gen8_update_reg_state_unlocked(struct i915_gem_context *ctx,
+                                          u32 *reg_state)
+{
+       struct drm_i915_private *dev_priv = ctx->i915;
+       const struct i915_oa_reg *flex_regs =

dev_priv->perf.oa.flex_regs;

drivers/gpu/drm/i915/i915_perf.c +16 -0
+       int n_flex_regs = dev_priv->perf.oa.flex_regs_len;
+       u32 ctx_oactxctrl = dev_priv->perf.oa.ctx_oactxctrl_offset;
+       u32 ctx_flexeu0 = dev_priv->perf.oa.ctx_flexeu0_offset;
+       /* The MMIO offsets for Flex EU registers aren't contiguous */
+       u32 flex_mmio[] = {
+               i915_mmio_reg_offset(EU_PERF_CNTL0),
+               i915_mmio_reg_offset(EU_PERF_CNTL1),
+               i915_mmio_reg_offset(EU_PERF_CNTL2),
+               i915_mmio_reg_offset(EU_PERF_CNTL3),
+               i915_mmio_reg_offset(EU_PERF_CNTL4),
+               i915_mmio_reg_offset(EU_PERF_CNTL5),
+               i915_mmio_reg_offset(EU_PERF_CNTL6),
+       };
+       int i;
+
+       reg_state[ctx_oactxctrl] =

i915_mmio_reg_offset(GEN8_OACTXCONTROL);
+ reg_state[ctx_oactxctrl+1] = (dev_priv->perf.oa.period_exponent
<<

drivers/gpu/drm/i915/i915_perf.c +10 -0
+                                     GEN8_OA_TIMER_PERIOD_SHIFT) |
+                                    (dev_priv->perf.oa.periodic ?
+                                     GEN8_OA_TIMER_ENABLE : 0) |
+                                    GEN8_OA_COUNTER_RESUME;
+
+       for (i = 0; i < ARRAY_SIZE(flex_mmio); i++) {
+               u32 state_offset = ctx_flexeu0 + i * 2;
+               u32 mmio = flex_mmio[i];
+
+               /* This arbitrary default will select the 'EU FPU0

Pipeline
+ * Active' event. In the future it's anticipated that
there
+ * will be an explicit 'No Event' we can select, but not
yet...

drivers/gpu/drm/i915/i915_perf.c +6 -0
+                */
+               u32 value = 0;
+               int j;
+
+               for (j = 0; j < n_flex_regs; j++) {
+                       if (i915_mmio_reg_offset(flex_regs[j].addr) ==

mmio) {

drivers/gpu/drm/i915/i915_perf.c +34 -0
+                               value = flex_regs[j].value;
+                               break;
+                       }
+               }
+
+               reg_state[state_offset] = mmio;
+               reg_state[state_offset+1] = value;
+       }
+}
+
+/* Manages updating the per-context aspects of the OA stream
+ * configuration across all contexts.
+ *
+ * The awkward consideration here is that OACTXCONTROL controls the
+ * exponent for periodic sampling which is primarily used for system
+ * wide profiling where we'd like a consistent sampling period even in
+ * the face of context switches.
+ *
+ * Our approach of updating the register state context (as opposed to
+ * say using a workaround batch buffer) ensures that the hardware
+ * won't automatically reload an out-of-date timer exponent even
+ * transiently before a WA BB could be parsed.
+ *
+ * This function needs to:
+ * - Ensure the currently running context's per-context OA state is
+ *   updated
+ * - Ensure that all existing contexts will have the correct per-context
+ *   OA state if they are scheduled for use.
+ * - Ensure any new contexts will be initialized with the correct
+ *   per-context OA state.
+ *
+ * Note: it's only the RCS/Render context that has any OA state.
+ */
+static int gen8_configure_all_contexts(struct drm_i915_private

*dev_priv)

drivers/gpu/drm/i915/i915_perf.c +5 -0
+{
+       struct i915_gem_context *ctx;
+       int ret;
+
+       ret = i915_mutex_lock_interruptible(&dev_priv->drm);
Quoted 8 lines

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1644

Quoted 6 lines
drivers/gpu/drm/i915/i915_perf.c +6 -0
+       if (ret)
+               return ret;
+
+       /* Switch away from any user context.  */
+       ret = i915_gem_switch_to_kernel_context(dev_priv);
+       if (ret)
  mutex_unlock(&dev_priv->drm.struct_mutex);

Duh! Thanks!

Quoted 46 lines
Quoted 46 lines
drivers/gpu/drm/i915/i915_perf.c +3 -0
+               return ret;
+
+       /* The OA register config is setup through the context image.

This image
+ * might be written to by the GPU on context switch (in
particular on
+ * lite-restore). This means we can't safely update a context's
image,

drivers/gpu/drm/i915/i915_perf.c +3 -0
+        * if this context is scheduled/submitted to run on the GPU.
+        *
+        * We could emit the OA register config through the batch buffer

but

drivers/gpu/drm/i915/i915_perf.c +4 -0
+        * this might leave small interval of time where the OA unit is
+        * configured at an invalid sampling period.
+        *
+        * So far the best way to work around this issue seems to be

draining

drivers/gpu/drm/i915/i915_perf.c +17 -0
+        * the GPU from any submitted work.
+        */
+       ret = i915_gem_wait_for_idle(dev_priv,
+                                    I915_WAIT_INTERRUPTIBLE |
+                                    I915_WAIT_LOCKED);
+       if (ret) {
+               mutex_unlock(&dev_priv->drm.struct_mutex);
+               return ret;
+       }
+
+       /* Update all contexts now that we've stalled the submission. */
+       list_for_each_entry(ctx, &dev_priv->context_list, link) {
+               if (!ctx->engine[RCS].initialised)
+                       continue;
+
+               gen8_update_reg_state_unlocked(ctx,
+

ctx->engine[RCS].lrc_reg_state);

drivers/gpu/drm/i915/i915_perf.c +10 -0
+       }
+
+       mutex_unlock(&dev_priv->drm.struct_mutex);
+
+       /* Now update the current context.
+        *
+        * Note: Using MMIO to update per-context registers requires
+        * some extra care...
+        */
+       ret = gen8_begin_ctx_mmio(dev_priv);
Quoted 8 lines

drivers/gpu/drm/i915/i915_perf.c

Quoted 5 lines
drivers/gpu/drm/i915/i915_perf.c +2 -0
+       if (ret) {
+               DRM_ERROR("Failed to bring RCS out of idle to update

current ctx OA state\n");

drivers/gpu/drm/i915/i915_perf.c +2 -0
+               return ret;
+       }

So do we still need the mmio for the current context, given that we now idle everything and then update the reg state?

Quoted 3 lines

I would say yes, because we don't have any guarantee that another context is going to be scheduled after that.
Yet we want the config to be applied before we return.

Okay, feel free to keep my r-b then.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#37 lore raw

PATCH v7 12/15 drm/i915/perf: Add OA unit support for Gen 8+

In reply to Matthew Auld

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

v7: Add missing mutex_unlock() if switching to kernel context fails
(Matthew)

---
 drivers/gpu/drm/i915/i915_drv.h  |  45 +-
 drivers/gpu/drm/i915/i915_perf.c | 959 ++++++++++++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/i915_reg.h  |  22 +
 drivers/gpu/drm/i915/intel_lrc.c |   2 +
 include/uapi/drm/i915_drm.h      |  19 +-
 5 files changed, 954 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index ffa1fc5eddfd..676b1227067c 100644
@@ -2064,9 +2064,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2097,20 +2105,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2472,6 +2473,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2541,6 +2543,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2851,6 +2861,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3615,6 +3627,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +888 -71

1,148 lines not on this page · raw message

drivers/gpu/drm/i915/i915_reg.h +22 -0
index 4c72adae368b..c0bbba6d5b78 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 7df278fe492e..351e231a02f4 100644
@@ -1879,6 +1879,8 @@ static void execlists_init_reg_state(u32 *regs,
 		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
 		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
 			make_rpcs(dev_priv));
+
+		i915_oa_init_reg_state(engine, ctx, regs);
 	}
 }
 
include/uapi/drm/i915_drm.h +12 -7
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
 	I915_OA_FORMAT_MAX	    /* non-ABI */
 };

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#38 lore raw

Re: [PATCH v7 12/15] drm/i915/perf: Add OA unit support for Gen 8+

Quoted 31 lines · Lionel Landwerlin
drivers/gpu/drm/i915/i915_perf.c +31 -0
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{
+	struct i915_gem_context *ctx;
+	int ret;
+
+	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+	if (ret)
+		return ret;
+
+	/* Switch away from any user context.  */
+	ret = i915_gem_switch_to_kernel_context(dev_priv);
+	if (ret) {
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+		return ret;
+	}
+
+	/* The OA register config is setup through the context image. This image
+	 * might be written to by the GPU on context switch (in particular on
+	 * lite-restore). This means we can't safely update a context's image,
+	 * if this context is scheduled/submitted to run on the GPU.
+	 *
+	 * We could emit the OA register config through the batch buffer but
+	 * this might leave small interval of time where the OA unit is
+	 * configured at an invalid sampling period.
+	 *
+	 * So far the best way to work around this issue seems to be draining
+	 * the GPU from any submitted work.
+	 */
+	ret = i915_gem_wait_for_idle(dev_priv,
+				     I915_WAIT_INTERRUPTIBLE |
+				     I915_WAIT_LOCKED);

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1673

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+	if (ret) {
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+		return ret;
+	}
+
+	/* Update all contexts now that we've stalled the submission. */
+	list_for_each_entry(ctx, &dev_priv->context_list, link) {
+		if (!ctx->engine[RCS].initialised)
+			continue;
+

You need to pin the context here, otherwise there is not guarrantee that the lrc_reg_state exists, or map it directly.

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1680

Quoted 7 lines
drivers/gpu/drm/i915/i915_perf.c +7 -0
+		gen8_update_reg_state_unlocked(ctx,
+					       ctx->engine[RCS].lrc_reg_state);
+	}
+
+	mutex_unlock(&dev_priv->drm.struct_mutex);
+
+	/* Now update the current context.

You don't need to. The current context is the kernel context, it is scratch and never used by userspace so no oa-reports.

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+	 *
+	 * Note: Using MMIO to update per-context registers requires
+	 * some extra care...
+	 */
+	ret = gen8_begin_ctx_mmio(dev_priv);
+	if (ret) {
+		DRM_ERROR("Failed to bring RCS out of idle to update current ctx OA state\n");
+		return ret;
+	}
+

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1700

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+	I915_WRITE(GEN8_OACTXCONTROL, ((dev_priv->perf.oa.period_exponent <<
+					GEN8_OA_TIMER_PERIOD_SHIFT) |
+				      (dev_priv->perf.oa.periodic ?
+				       GEN8_OA_TIMER_ENABLE : 0) |
+				      GEN8_OA_COUNTER_RESUME));
+
+	config_oa_regs(dev_priv, dev_priv->perf.oa.flex_regs,
+			dev_priv->perf.oa.flex_regs_len);
+
+	gen8_end_ctx_mmio(dev_priv);

This entire chunk can go and I don't need to critique gen8_begin_ctx_mmio() -- it needs a bit of tlc.

This patch is not ready.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#39 lore raw

PATCH v9 12/15 drm/i915/perf: Add OA unit support for Gen 8+

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

v7: Add missing mutex_unlock() if switching to kernel context fails
(Matthew)

v8: Simplify OA period/flex-eu-counters programming by using the
batchbuffer instead of modifying ctx-image (Lionel)

v9: Back to updating the context image (due to erroneous testing,
batchbuffer programming the OA unit doesn't actually work) (Lionel)
Pin context before updating context image (Chris)
Drop MMIO programming now that we switch to a kernel context with right values in initial context image (Chris)

---
 drivers/gpu/drm/i915/i915_drv.h  |  45 +-
 drivers/gpu/drm/i915/i915_perf.c | 879 +++++++++++++++++++++++++++++++++++----
 drivers/gpu/drm/i915/i915_reg.h  |  22 +
 drivers/gpu/drm/i915/intel_lrc.c |   2 +
 include/uapi/drm/i915_drm.h      |  19 +-
 5 files changed, 874 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index f15751534c29..eaf4ce5e5790 100644
@@ -2063,9 +2063,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2096,20 +2104,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2470,6 +2471,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2539,6 +2541,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2849,6 +2859,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3612,6 +3624,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +808 -71

1,068 lines not on this page · raw message

drivers/gpu/drm/i915/i915_reg.h +22 -0
index ee8170cda93e..f8acde02c314 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 0909549ad320..58216c4bfbc1 100644
@@ -1877,6 +1877,8 @@ static void execlists_init_reg_state(u32 *regs,
 		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
 		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
 			make_rpcs(dev_priv));
+
+		i915_oa_init_reg_state(engine, ctx, regs);
 	}
 }
 
include/uapi/drm/i915_drm.h +12 -7
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
 	I915_OA_FORMAT_MAX	    /* non-ABI */
 };

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#40 lore raw

Re: [PATCH v9 12/15] drm/i915/perf: Add OA unit support for Gen 8+

On Mon, May 01, 2017 at 06:17:09PM -0700, Lionel Landwerlin wrote:

Focusing on the bit I know best and leaving the hw mumbo jumbo to one side...

Quoted 37 lines
drivers/gpu/drm/i915/i915_perf.c +37 -0
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{
+	struct intel_engine_cs *engine = dev_priv->engine[RCS];
+	struct i915_gem_context *ctx;
+	int ret;
+
+	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+	if (ret)
+		return ret;
+
+	/* Switch away from any user context.  */
+	ret = i915_gem_switch_to_kernel_context(dev_priv);
+	if (ret) {
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+		return ret;
+	}
+
+	/* The OA register config is setup through the context image. This image
+	 * might be written to by the GPU on context switch (in particular on
+	 * lite-restore). This means we can't safely update a context's image,
+	 * if this context is scheduled/submitted to run on the GPU.
+	 *
+	 * We could emit the OA register config through the batch buffer but
+	 * this might leave small interval of time where the OA unit is
+	 * configured at an invalid sampling period.
+	 *
+	 * So far the best way to work around this issue seems to be draining
+	 * the GPU from any submitted work.
+	 */
+	ret = i915_gem_wait_for_idle(dev_priv,
+				     I915_WAIT_INTERRUPTIBLE |
+				     I915_WAIT_LOCKED);
+	if (ret) {
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+		return ret;
+	}
+

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1617

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+	/* Update all contexts now that we've stalled the submission. */
+	list_for_each_entry(ctx, &dev_priv->context_list, link) {
+		ret = engine->context_pin(engine, ctx);
+		if (ret) {
+			mutex_unlock(&dev_priv->drm.struct_mutex);
+			return ret;
+		}
+
+		gen8_update_reg_state_unlocked(ctx,
+					       ctx->engine[RCS].lrc_reg_state);

engine->context_unpin is missing.

Since you do populate the OA settings for contexts initialised later, you can skip the context_pin here (keeping the deferred initialisation for currently unused contexts) and use

	list_for_each_entry(ctx, &dev_priv->context_list, link) {
		struct intel_context *ce = &ctx->engine[RCS];
		u32 *regs;

		if (!ce->state) /* OA settings will be set upon first use */
			continue;

		/* Hmm, export this from intel_lrc.c? */
		regs = i915_gem_object_pin_map(ce->state->obj, I915_MAP_WB);
		if (IS_ERR(regs)) {
			/* teardown? */
			mutex_unlock(&ctx->i915->drm.struct_mutex);
			return PTR_ERR(regs);
		}
		ce->state->obj->mm.dirty = true;
		regs += LRC_STATE_PN * PAGE_SIZE / sizeof(*regs);
		/* Probably best since we don't really want to expose
		 * the LRC_STATE_PN trick.
		 */

		gen8_update_reg_state_unlocked(ctx, regs);

		i915_gem_object_unpin_map(ce->state->obj);
	}
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#41 lore raw

PATCH v10 12/15 drm/i915/perf: Add OA unit support for Gen 8+

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

v7: Add missing mutex_unlock() if switching to kernel context fails
(Matthew)

v8: Simplify OA period/flex-eu-counters programming by using the
batchbuffer instead of modifying ctx-image (Lionel)

v9: Back to updating the context image (due to erroneous testing,
batchbuffer programming the OA unit doesn't actually work) (Lionel)
Pin context before updating context image (Chris)
Drop MMIO programming now that we switch to a kernel context with right values in initial context image (Chris)

v10: Just pin_map the contexts we want to modify or let the
configuration happen on first use (Chris)

---
 drivers/gpu/drm/i915/i915_drv.h  |  45 +-
 drivers/gpu/drm/i915/i915_perf.c | 889 +++++++++++++++++++++++++++++++++++----
 drivers/gpu/drm/i915/i915_reg.h  |  22 +
 drivers/gpu/drm/i915/intel_lrc.c |   2 +
 include/uapi/drm/i915_drm.h      |  19 +-
 5 files changed, 884 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index f15751534c29..eaf4ce5e5790 100644
@@ -2063,9 +2063,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2096,20 +2104,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2470,6 +2471,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2539,6 +2541,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2849,6 +2859,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3612,6 +3624,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +818 -71

1,078 lines not on this page · raw message

drivers/gpu/drm/i915/i915_reg.h +22 -0
index ee8170cda93e..f8acde02c314 100644
@@ -653,6 +653,12 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define GEN8_OACTXID _MMIO(0x2364)
 
+#define GEN8_OA_DEBUG _MMIO(0x2B04)
+#define  GEN9_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS    (1<<5)
+#define  GEN9_OA_DEBUG_INCLUDE_CLK_RATIO	    (1<<6)
+#define  GEN9_OA_DEBUG_DISABLE_GO_1_0_REPORTS	    (1<<2)
+#define  GEN9_OA_DEBUG_DISABLE_CTX_SWITCH_REPORTS   (1<<1)
+
 #define GEN8_OACONTROL _MMIO(0x2B00)
 #define  GEN8_OA_REPORT_FORMAT_A12	    (0<<2)
 #define  GEN8_OA_REPORT_FORMAT_A12_B8_C8    (2<<2)
@@ -674,6 +680,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN7_OABUFFER_STOP_RESUME_ENABLE   (1<<1)
 #define  GEN7_OABUFFER_RESUME		    (1<<0)
 
+#define GEN8_OABUFFER_UDW _MMIO(0x23b4)
 #define GEN8_OABUFFER _MMIO(0x2b14)
 
 #define GEN7_OASTATUS1 _MMIO(0x2364)
@@ -692,7 +699,9 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 #define  GEN8_OASTATUS_REPORT_LOST	    (1<<0)
 
 #define GEN8_OAHEADPTR _MMIO(0x2B0C)
+#define GEN8_OAHEADPTR_MASK    0xffffffc0
 #define GEN8_OATAILPTR _MMIO(0x2B10)
+#define GEN8_OATAILPTR_MASK    0xffffffc0
 
 #define OABUFFER_SIZE_128K  (0<<3)
 #define OABUFFER_SIZE_256K  (1<<3)
@@ -705,7 +714,17 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)
 
 #define OA_MEM_SELECT_GGTT  (1<<0)
 
+/*
+ * Flexible, Aggregate EU Counter Registers.
+ * Note: these aren't contiguous
+ */
 #define EU_PERF_CNTL0	    _MMIO(0xe458)
+#define EU_PERF_CNTL1	    _MMIO(0xe558)
+#define EU_PERF_CNTL2	    _MMIO(0xe658)
+#define EU_PERF_CNTL3	    _MMIO(0xe758)
+#define EU_PERF_CNTL4	    _MMIO(0xe45c)
+#define EU_PERF_CNTL5	    _MMIO(0xe55c)
+#define EU_PERF_CNTL6	    _MMIO(0xe65c)
 
 #define GDT_CHICKEN_BITS    _MMIO(0x9840)
 #define GT_NOA_ENABLE	    0x00000080
@@ -2325,6 +2344,9 @@ enum skl_disp_power_wells {
 #define   GEN8_RC_SEMA_IDLE_MSG_DISABLE	(1 << 12)
 #define   GEN8_FF_DOP_CLOCK_GATE_DISABLE	(1<<10)
 
+#define GEN6_RCS_PWR_FSM _MMIO(0x22ac)
+#define GEN9_RCS_FE_FSM2 _MMIO(0x22a4)
+
 /* Fuse readout registers for GT */
 #define CHV_FUSE_GT			_MMIO(VLV_DISPLAY_BASE + 0x2168)
 #define   CHV_FGT_DISABLE_SS0		(1 << 10)
drivers/gpu/drm/i915/intel_lrc.c +2 -0
index 0909549ad320..58216c4bfbc1 100644
@@ -1877,6 +1877,8 @@ static void execlists_init_reg_state(u32 *regs,
 		regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
 		CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
 			make_rpcs(dev_priv));
+
+		i915_oa_init_reg_state(engine, ctx, regs);
 	}
 }
 
include/uapi/drm/i915_drm.h +12 -7
index 464547d08173..15bc9f78ba4d 100644
@@ -1316,13 +1316,18 @@ struct drm_i915_gem_context_param {
 };
 
 enum drm_i915_oa_format {
-	I915_OA_FORMAT_A13 = 1,
-	I915_OA_FORMAT_A29,
-	I915_OA_FORMAT_A13_B8_C8,
-	I915_OA_FORMAT_B4_C8,
-	I915_OA_FORMAT_A45_B8_C8,
-	I915_OA_FORMAT_B4_C8_A16,
-	I915_OA_FORMAT_C4_B8,
+	I915_OA_FORMAT_A13 = 1,	    /* HSW only */
+	I915_OA_FORMAT_A29,	    /* HSW only */
+	I915_OA_FORMAT_A13_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8,	    /* HSW only */
+	I915_OA_FORMAT_A45_B8_C8,   /* HSW only */
+	I915_OA_FORMAT_B4_C8_A16,   /* HSW only */
+	I915_OA_FORMAT_C4_B8,	    /* HSW+ */
+
+	/* Gen8+ */
+	I915_OA_FORMAT_A12,
+	I915_OA_FORMAT_A12_B8_C8,
+	I915_OA_FORMAT_A32u40_A4u32_B8_C8,
 
 	I915_OA_FORMAT_MAX	    /* non-ABI */
 };

--
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Reviewed-by on v10 12/15
#42 lore raw

Re: [PATCH v10 12/15] drm/i915/perf: Add OA unit support for Gen 8+

Quoted 867 lines · Lionel Landwerlin

From: Robert Bragg <[email protected]>

Enables access to OA unit metrics for BDW, CHV, SKL and BXT which all share (more-or-less) the same OA unit design.

Of particular note in comparison to Haswell: some OA unit HW config state has become per-context state and as a consequence it is somewhat more complicated to manage synchronous state changes from the cpu while there's no guarantee of what context (if any) is currently actively running on the gpu.

The periodic sampling frequency which can be particularly useful for system-wide analysis (as opposed to command stream synchronised MI_REPORT_PERF_COUNT commands) is perhaps the most surprising state to have become per-context save and restored (while the OABUFFER destination is still a shared, system-wide resource).

This support for gen8+ takes care to consider a number of timing challenges involved in synchronously updating per-context state primarily by programming all config state from the cpu and updating all current and saved contexts synchronously while the OA unit is still disabled.

The driver intentionally avoids depending on command streamer programming to update OA state considering the lack of synchronization between the automatic loading of OACTXCONTROL state (that includes the periodic sampling state and enable state) on context restore and the parsing of any general purpose BB the driver can control. I.e. this implementation is careful to avoid the possibility of a context restore temporarily enabling any out-of-date periodic sampling state. In addition to the risk of transiently-out-of-date state being loaded automatically; there are also internal HW latencies involved in the loading of MUX configurations which would be difficult to account for from the command streamer (and we only want to enable the unit when once the MUX configuration is complete).

Since the Gen8+ OA unit design no longer supports clock gating the unit off for a single given context (which effectively stopped any progress of counters while any other context was running) and instead supports tagging OA reports with a context ID for filtering on the CPU, it means we can no longer hide the system-wide progress of counters from a non-privileged application only interested in metrics for its own context. Although we could theoretically try and subtract the progress of other contexts before forwarding reports via read() we aren't in a position to filter reports captured via MI_REPORT_PERF_COUNT commands. As a result, for Gen8+, we always require the dev.i915.perf_stream_paranoid to be unset for any access to OA metrics if not root.

v5: Drain submitted requests when enabling metric set to ensure no
lite-restore erases the context image we just updated (Lionel)

v6: In addition to drain, switch to kernel context & update all
context in place (Chris)

v7: Add missing mutex_unlock() if switching to kernel context fails
(Matthew)

v8: Simplify OA period/flex-eu-counters programming by using the
batchbuffer instead of modifying ctx-image (Lionel)

v9: Back to updating the context image (due to erroneous testing,
batchbuffer programming the OA unit doesn't actually work) (Lionel)
Pin context before updating context image (Chris)
Drop MMIO programming now that we switch to a kernel context with right values in initial context image (Chris)

v10: Just pin_map the contexts we want to modify or let the
configuration happen on first use (Chris)

---
 drivers/gpu/drm/i915/i915_drv.h  |  45 +-
 drivers/gpu/drm/i915/i915_perf.c | 889 +++++++++++++++++++++++++++++++++++----
 drivers/gpu/drm/i915/i915_reg.h  |  22 +
 drivers/gpu/drm/i915/intel_lrc.c |   2 +
 include/uapi/drm/i915_drm.h      |  19 +-
 5 files changed, 884 insertions(+), 93 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +30 -15
index f15751534c29..eaf4ce5e5790 100644
@@ -2063,9 +2063,17 @@ struct i915_oa_ops {
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
-	 * Boolean and Custom (B/C) counters that are part of the counter
-	 * reports being sampled. May apply system constraints such as
+	 * @select_metric_set: The auto generated code that checks whether a
+	 * requested OA config is applicable to the system and if so sets up
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
@@ -2096,20 +2104,13 @@ struct i915_oa_ops {
 		    size_t *offset);
 
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
+	 * @oa_hw_tail_read: read the OA tail pointer register
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
-	 * is only called while the stream is enabled, while the global OA
-	 * configuration can't be modified.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
+	 * In particular this enables us to share all the fiddly code for
+	 * handling the OA unit tail pointer race that affects multiple
+	 * generations.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 
 struct intel_cdclk_state {
@@ -2470,6 +2471,7 @@ struct drm_i915_private {
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 
@@ -2539,6 +2541,14 @@ struct drm_i915_private {
 			} oa_buffer;
 
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+
+			/* The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
@@ -2849,6 +2859,8 @@ intel_info(const struct drm_i915_private *dev_priv)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
@@ -3612,6 +3624,9 @@ i915_gem_context_lookup_timeline(struct i915_gem_context *ctx,
 
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
drivers/gpu/drm/i915/i915_perf.c +532 -35
index 3277a52ce98e..7a82681d122e 100644
@@ -196,6 +196,12 @@
 
 #include "i915_drv.h"
 #include "i915_oa_hsw.h"
+#include "i915_oa_bdw.h"
+#include "i915_oa_chv.h"
+#include "i915_oa_sklgt2.h"
+#include "i915_oa_sklgt3.h"
+#include "i915_oa_sklgt4.h"
+#include "i915_oa_bxt.h"
 
 /* HW requires this to be a power of two, between 128k and 16M, though driver
  * is currently generally designed assuming the largest 16M size is used such
@@ -215,7 +221,7 @@
  *
  * Although this can be observed explicitly while copying reports to userspace
  * by checking for a zeroed report-id field in tail reports, we want to account
- * for this earlier, as part of the _oa_buffer_check to avoid lots of redundant
+ * for this earlier, as part of the oa_buffer_check to avoid lots of redundant
  * read() attempts.
  *
  * In effect we define a tail pointer for reading that lags the real tail
@@ -237,7 +243,7 @@
  * indicates that an updated tail pointer is needed.
  *
  * Most of the implementation details for this workaround are in
- * gen7_oa_buffer_check_unlocked() and gen7_appand_oa_reports()
+ * oa_buffer_check_unlocked() and _append_oa_reports()
  *
  * Note for posterity: previously the driver used to define an effective tail
  * pointer that lagged the real pointer by a 'tail margin' measured in bytes
@@ -272,6 +278,13 @@ static u32 i915_perf_stream_paranoid = true;
 
 #define INVALID_CTX_ID 0xffffffff
 
+/* On Gen8+ automatically triggered OA reports include a 'reason' field... */
+#define OAREPORT_REASON_MASK           0x3f
+#define OAREPORT_REASON_SHIFT          19
+#define OAREPORT_REASON_TIMER          (1<<0)
+#define OAREPORT_REASON_CTX_SWITCH     (1<<3)
+#define OAREPORT_REASON_CLK_RATIO      (1<<5)
+
 
 /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate
  *
@@ -303,6 +316,13 @@ static struct i915_oa_format hsw_oa_formats[I915_OA_FORMAT_MAX] = {
 	[I915_OA_FORMAT_C4_B8]	    = { 7, 64 },
 };
 
+static struct i915_oa_format gen8_plus_oa_formats[I915_OA_FORMAT_MAX] = {
+	[I915_OA_FORMAT_A12]		    = { 0, 64 },
+	[I915_OA_FORMAT_A12_B8_C8]	    = { 2, 128 },
+	[I915_OA_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 },
+	[I915_OA_FORMAT_C4_B8]		    = { 7, 64 },
+};
+
 #define SAMPLE_OA_REPORT      (1<<0)
 
 /**
@@ -332,8 +352,20 @@ struct perf_open_properties {
 	int oa_period_exponent;
 };
 
+static u32 gen8_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+	return I915_READ(GEN8_OATAILPTR) & GEN8_OATAILPTR_MASK;
+}
+
+static u32 gen7_oa_hw_tail_read(struct drm_i915_private *dev_priv)
+{
+	u32 oastatus1 = I915_READ(GEN7_OASTATUS1);
+
+	return oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+}
+
 /**
- * gen7_oa_buffer_check_unlocked - check for data and update tail ptr state
+ * oa_buffer_check_unlocked - check for data and update tail ptr state
  * @dev_priv: i915 device instance
  *
  * This is either called via fops (for blocking reads in user ctx) or the poll
@@ -356,12 +388,11 @@ struct perf_open_properties {
  *
  * Returns: %true if the OA buffer contains data, else %false
  */
-static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
+static bool oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 {
 	int report_size = dev_priv->perf.oa.oa_buffer.format_size;
 	unsigned long flags;
 	unsigned int aged_idx;
-	u32 oastatus1;
 	u32 head, hw_tail, aged_tail, aging_tail;
 	u64 now;
 
@@ -381,8 +412,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 	aged_tail = dev_priv->perf.oa.oa_buffer.tails[aged_idx].offset;
 	aging_tail = dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset;
 
-	oastatus1 = I915_READ(GEN7_OASTATUS1);
-	hw_tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+	hw_tail = dev_priv->perf.oa.ops.oa_hw_tail_read(dev_priv);
 
 	/* The tail pointer increases in 64 byte increments,
 	 * not in report_size steps...
@@ -404,6 +434,7 @@ static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
 	if (aging_tail != INVALID_TAIL_PTR &&
 	    ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
 	     OA_TAIL_MARGIN_NSEC)) {
+
 		aged_idx ^= 1;
 		dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
 
@@ -553,6 +584,287 @@ static int append_oa_sample(struct i915_perf_stream *stream,
  *
  * Returns: 0 on success, negative error code on failure.
  */
+static int gen8_append_oa_reports(struct i915_perf_stream *stream,
+				  char __user *buf,
+				  size_t count,
+				  size_t *offset)
+{
+	struct drm_i915_private *dev_priv = stream->dev_priv;
+	int report_size = dev_priv->perf.oa.oa_buffer.format_size;
+	u8 *oa_buf_base = dev_priv->perf.oa.oa_buffer.vaddr;
+	u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+	u32 mask = (OA_BUFFER_SIZE - 1);
+	size_t start_offset = *offset;
+	unsigned long flags;
+	unsigned int aged_tail_idx;
+	u32 head, tail;
+	u32 taken;
+	int ret = 0;
+
+	if (WARN_ON(!stream->enabled))
+		return -EIO;
+
+	spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	head = dev_priv->perf.oa.oa_buffer.head;
+	aged_tail_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+	tail = dev_priv->perf.oa.oa_buffer.tails[aged_tail_idx].offset;
+
+	spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	/* An invalid tail pointer here means we're still waiting for the poll
+	 * hrtimer callback to give us a pointer
+	 */
+	if (tail == INVALID_TAIL_PTR)
+		return -EAGAIN;
+
+	/* NB: oa_buffer.head/tail include the gtt_offset which we don't want
+	 * while indexing relative to oa_buf_base.
+	 */
+	head -= gtt_offset;
+	tail -= gtt_offset;
+
+	/* An out of bounds or misaligned head or tail pointer implies a driver
+	 * bug since we validate + align the tail pointers we read from the
+	 * hardware and we are in full control of the head pointer which should
+	 * only be incremented by multiples of the report size (notably also
+	 * all a power of two).
+	 */
+	if (WARN_ONCE(head > OA_BUFFER_SIZE || head % report_size ||
+		      tail > OA_BUFFER_SIZE || tail % report_size,
+		      "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
+		      head, tail))
+		return -EIO;
+
+
+	for (/* none */;
+	     (taken = OA_TAKEN(tail, head));
+	     head = (head + report_size) & mask) {
+		u8 *report = oa_buf_base + head;
+		u32 *report32 = (void *)report;
+		u32 ctx_id;
+		u32 reason;
+
+		/* All the report sizes factor neatly into the buffer
+		 * size so we never expect to see a report split
+		 * between the beginning and end of the buffer.
+		 *
+		 * Given the initial alignment check a misalignment
+		 * here would imply a driver bug that would result
+		 * in an overrun.
+		 */
+		if (WARN_ON((OA_BUFFER_SIZE - head) < report_size)) {
+			DRM_ERROR("Spurious OA head ptr: non-integral report offset\n");
+			break;
+		}
+
+		/* The reason field includes flags identifying what
+		 * triggered this specific report (mostly timer
+		 * triggered or e.g. due to a context switch).
+		 *
+		 * This field is never expected to be zero so we can
+		 * check that the report isn't invalid before copying
+		 * it to userspace...
+		 */
+		reason = ((report32[0] >> OAREPORT_REASON_SHIFT) &
+			  OAREPORT_REASON_MASK);
+		if (reason == 0) {
+			if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+				DRM_NOTE("Skipping spurious, invalid OA report\n");
+			continue;
+		}
+
+		/* XXX: Just keep the lower 21 bits for now since I'm not
+		 * entirely sure if the HW touches any of the higher bits in
+		 * this field
+		 */
+		ctx_id = report32[2] & 0x1fffff;
+
+		/* Squash whatever is in the CTX_ID field if it's marked as
+		 * invalid to be sure we avoid false-positive, single-context
+		 * filtering below...
+		 *
+		 * Note: that we don't clear the valid_ctx_bit so userspace can
+		 * understand that the ID has been squashed by the kernel.
+		 */
+		if (!(report32[0] & dev_priv->perf.oa.gen8_valid_ctx_bit))
+			ctx_id = report32[2] = INVALID_CTX_ID;
+
+		/* NB: For Gen 8 the OA unit no longer supports clock gating
+		 * off for a specific context and the kernel can't securely
+		 * stop the counters from updating as system-wide / global
+		 * values.
+		 *
+		 * Automatic reports now include a context ID so reports can be
+		 * filtered on the cpu but it's not worth trying to
+		 * automatically subtract/hide counter progress for other
+		 * contexts while filtering since we can't stop userspace
+		 * issuing MI_REPORT_PERF_COUNT commands which would still
+		 * provide a side-band view of the real values.
+		 *
+		 * To allow userspace (such as Mesa/GL_INTEL_performance_query)
+		 * to normalize counters for a single filtered context then it
+		 * needs be forwarded bookend context-switch reports so that it
+		 * can track switches in between MI_REPORT_PERF_COUNT commands
+		 * and can itself subtract/ignore the progress of counters
+		 * associated with other contexts. Note that the hardware
+		 * automatically triggers reports when switching to a new
+		 * context which are tagged with the ID of the newly active
+		 * context. To avoid the complexity (and likely fragility) of
+		 * reading ahead while parsing reports to try and minimize
+		 * forwarding redundant context switch reports (i.e. between
+		 * other, unrelated contexts) we simply elect to forward them
+		 * all.
+		 *
+		 * We don't rely solely on the reason field to identify context
+		 * switches since it's not-uncommon for periodic samples to
+		 * identify a switch before any 'context switch' report.
+		 */
+		if (!dev_priv->perf.oa.exclusive_stream->ctx ||
+		    dev_priv->perf.oa.specific_ctx_id == ctx_id ||
+		    (dev_priv->perf.oa.oa_buffer.last_ctx_id ==
+		     dev_priv->perf.oa.specific_ctx_id) ||
+		    reason & OAREPORT_REASON_CTX_SWITCH) {
+
+			/* While filtering for a single context we avoid
+			 * leaking the IDs of other contexts.
+			 */
+			if (dev_priv->perf.oa.exclusive_stream->ctx &&
+			    dev_priv->perf.oa.specific_ctx_id != ctx_id) {
+				report32[2] = INVALID_CTX_ID;
+			}
+
+			ret = append_oa_sample(stream, buf, count, offset,
+					       report);
+			if (ret)
+				break;
+
+			dev_priv->perf.oa.oa_buffer.last_ctx_id = ctx_id;
+		}
+
+		/* The above reason field sanity check is based on
+		 * the assumption that the OA buffer is initially
+		 * zeroed and we reset the field after copying so the
+		 * check is still meaningful once old reports start
+		 * being overwritten.
+		 */
+		report32[0] = 0;
+	}
+
+	if (start_offset != *offset) {
+		spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+		/* We removed the gtt_offset for the copy loop above, indexing
+		 * relative to oa_buf_base so put back here...
+		 */
+		head += gtt_offset;
+
+		I915_WRITE(GEN8_OAHEADPTR, head & GEN8_OAHEADPTR_MASK);
+		dev_priv->perf.oa.oa_buffer.head = head;
+
+		spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+	}
+
+	return ret;
+}
+
+/**
+ * gen8_oa_read - copy status records then buffered OA reports
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Checks OA unit status registers and if necessary appends corresponding
+ * status records for userspace (such as for a buffer full condition) and then
+ * initiate appending any buffered OA reports.
+ *
+ * Updates @offset according to the number of bytes successfully copied into
+ * the userspace buffer.
+ *
+ * NB: some data may be successfully copied to the userspace buffer
+ * even if an error is returned, and this is reflected in the
+ * updated @offset.
+ *
+ * Returns: zero on success or a negative error code
+ */
+static int gen8_oa_read(struct i915_perf_stream *stream,
+			char __user *buf,
+			size_t count,
+			size_t *offset)
+{
+	struct drm_i915_private *dev_priv = stream->dev_priv;
+	u32 oastatus;
+	int ret;
+
+	if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
+		return -EIO;
+
+	oastatus = I915_READ(GEN8_OASTATUS);
+
+	/* We treat OABUFFER_OVERFLOW as a significant error:
+	 *
+	 * Although theoretically we could handle this more gracefully
+	 * sometimes, some Gens don't correctly suppress certain
+	 * automatically triggered reports in this condition and so we
+	 * have to assume that old reports are now being trampled
+	 * over.
+	 *
+	 * Considering how we don't currently give userspace control
+	 * over the OA buffer size and always configure a large 16MB
+	 * buffer, then a buffer overflow does anyway likely indicate
+	 * that something has gone quite badly wrong.
+	 */
+	if (oastatus & GEN8_OASTATUS_OABUFFER_OVERFLOW) {
+		ret = append_oa_status(stream, buf, count, offset,
+				       DRM_I915_PERF_RECORD_OA_BUFFER_LOST);
+		if (ret)
+			return ret;
+
+		DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+			  dev_priv->perf.oa.period_exponent);
+
+		dev_priv->perf.oa.ops.oa_disable(dev_priv);
+		dev_priv->perf.oa.ops.oa_enable(dev_priv);
+
+		/* Note: .oa_enable() is expected to re-init the oabuffer
+		 * and reset GEN8_OASTATUS for us
+		 */
+		oastatus = I915_READ(GEN8_OASTATUS);
+	}
+
+	if (oastatus & GEN8_OASTATUS_REPORT_LOST) {
+		ret = append_oa_status(stream, buf, count, offset,
+				       DRM_I915_PERF_RECORD_OA_REPORT_LOST);
+		if (ret)
+			return ret;
+		I915_WRITE(GEN8_OASTATUS,
+			   oastatus & ~GEN8_OASTATUS_REPORT_LOST);
+	}
+
+	return gen8_append_oa_reports(stream, buf, count, offset);
+}
+
+/**
+ * Copies all buffered OA reports into userspace read() buffer.
+ * @stream: An i915-perf stream opened for OA metrics
+ * @buf: destination buffer given by userspace
+ * @count: the number of bytes userspace wants to read
+ * @offset: (inout): the current position for writing into @buf
+ *
+ * Notably any error condition resulting in a short read (-%ENOSPC or
+ * -%EFAULT) will be returned even though one or more records may
+ * have been successfully copied. In this case it's up to the caller
+ * to decide if the error should be squashed before returning to
+ * userspace.
+ *
+ * Note: reports are consumed from the head, and appended to the
+ * tail, so the tail chases the head?... If you think that's mad
+ * and back-to-front you're not alone, but this follows the
+ * Gen PRM naming convention.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
 static int gen7_append_oa_reports(struct i915_perf_stream *stream,
 				  char __user *buf,
 				  size_t count,
@@ -733,7 +1045,8 @@ static int gen7_oa_read(struct i915_perf_stream *stream,
 		if (ret)
 			return ret;
 
-		DRM_DEBUG("OA buffer overflow: force restart\n");
+		DRM_DEBUG("OA buffer overflow (exponent = %d): force restart\n",
+			  dev_priv->perf.oa.period_exponent);
 
 		dev_priv->perf.oa.ops.oa_disable(dev_priv);
 		dev_priv->perf.oa.ops.oa_enable(dev_priv);
@@ -776,7 +1089,7 @@ static int i915_oa_wait_unlocked(struct i915_perf_stream *stream)
 		return -EIO;
 
 	return wait_event_interruptible(dev_priv->perf.oa.poll_wq,
-					dev_priv->perf.oa.ops.oa_buffer_check(dev_priv));
+					oa_buffer_check_unlocked(dev_priv));
 }
 
 /**
@@ -833,33 +1146,37 @@ static int i915_oa_read(struct i915_perf_stream *stream,
 static int oa_get_render_ctx_id(struct i915_perf_stream *stream)
 {
 	struct drm_i915_private *dev_priv = stream->dev_priv;
-	struct intel_engine_cs *engine = dev_priv->engine[RCS];
-	int ret;
 
-	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
-	if (ret)
-		return ret;
+	if (i915.enable_execlists)
+		dev_priv->perf.oa.specific_ctx_id = stream->ctx->hw_id;
+	else {
+		struct intel_engine_cs *engine = dev_priv->engine[RCS];
+		int ret;
 
-	/* As the ID is the gtt offset of the context's vma we pin
-	 * the vma to ensure the ID remains fixed.
-	 *
-	 * NB: implied RCS engine...
-	 */
-	ret = engine->context_pin(engine, stream->ctx);
-	if (ret)
-		goto unlock;
+		ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+		if (ret)
+			return ret;
 
-	/* Explicitly track the ID (instead of calling i915_ggtt_offset()
-	 * on the fly) considering the difference with gen8+ and
-	 * execlists
-	 */
-	dev_priv->perf.oa.specific_ctx_id =
-		i915_ggtt_offset(stream->ctx->engine[engine->id].state);
+		/* As the ID is the gtt offset of the context's vma we pin
+		 * the vma to ensure the ID remains fixed.
+		 */
+		ret = engine->context_pin(engine, stream->ctx);
+		if (ret) {
+			mutex_unlock(&dev_priv->drm.struct_mutex);
+			return ret;
+		}
 
-unlock:
-	mutex_unlock(&dev_priv->drm.struct_mutex);
+		/* Explicitly track the ID (instead of calling
+		 * i915_ggtt_offset() on the fly) considering the difference
+		 * with gen8+ and execlists
+		 */
+		dev_priv->perf.oa.specific_ctx_id =
+			i915_ggtt_offset(stream->ctx->engine[engine->id].state);
 
-	return ret;
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+	}
+
+	return 0;
 }
 
 /**
@@ -874,12 +1191,16 @@ static void oa_put_render_ctx_id(struct i915_perf_stream *stream)
 	struct drm_i915_private *dev_priv = stream->dev_priv;
 	struct intel_engine_cs *engine = dev_priv->engine[RCS];
 
-	mutex_lock(&dev_priv->drm.struct_mutex);
+	if (i915.enable_execlists) {
+		dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+	} else {
+		mutex_lock(&dev_priv->drm.struct_mutex);
 
-	dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
-	engine->context_unpin(engine, stream->ctx);
+		dev_priv->perf.oa.specific_ctx_id = INVALID_CTX_ID;
+		engine->context_unpin(engine, stream->ctx);
 
-	mutex_unlock(&dev_priv->drm.struct_mutex);
+		mutex_unlock(&dev_priv->drm.struct_mutex);
+	}
 }
 
 static void
@@ -969,6 +1290,61 @@ static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
 	dev_priv->perf.oa.pollin = false;
 }
 
+static void gen8_init_oa_buffer(struct drm_i915_private *dev_priv)
+{
+	u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+	unsigned long flags;
+
+	spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	I915_WRITE(GEN8_OASTATUS, 0);
+	I915_WRITE(GEN8_OAHEADPTR, gtt_offset);
+	dev_priv->perf.oa.oa_buffer.head = gtt_offset;
+
+	I915_WRITE(GEN8_OABUFFER_UDW, 0);
+
+	/* PRM says:
+	 *
+	 *  "This MMIO must be set before the OATAILPTR
+	 *  register and after the OAHEADPTR register. This is
+	 *  to enable proper functionality of the overflow
+	 *  bit."
+	 */
+	I915_WRITE(GEN8_OABUFFER, gtt_offset |
+		   OABUFFER_SIZE_16M | OA_MEM_SELECT_GGTT);
+	I915_WRITE(GEN8_OATAILPTR, gtt_offset & GEN8_OATAILPTR_MASK);
+
+	/* Mark that we need updated tail pointers to read from... */
+	dev_priv->perf.oa.oa_buffer.tails[0].offset = INVALID_TAIL_PTR;
+	dev_priv->perf.oa.oa_buffer.tails[1].offset = INVALID_TAIL_PTR;
+
+	/* Reset state used to recognise context switches, affecting which
+	 * reports we will forward to userspace while filtering for a single
+	 * context.
+	 */
+	dev_priv->perf.oa.oa_buffer.last_ctx_id = INVALID_CTX_ID;
+
+	spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+	/* NB: although the OA buffer will initially be allocated
+	 * zeroed via shmfs (and so this memset is redundant when
+	 * first allocating), we may re-init the OA buffer, either
+	 * when re-enabling a stream or in error/reset paths.
+	 *
+	 * The reason we clear the buffer for each re-init is for the
+	 * sanity check in gen8_append_oa_reports() that looks at the
+	 * reason field to make sure it's non-zero which relies on
+	 * the assumption that new reports are being written to zeroed
+	 * memory...
+	 */
+	memset(dev_priv->perf.oa.oa_buffer.vaddr, 0, OA_BUFFER_SIZE);
+
+	/* Maybe make ->pollin per-stream state if we support multiple
+	 * concurrent streams in the future.
+	 */
+	dev_priv->perf.oa.pollin = false;
+}
+
 static int alloc_oa_buffer(struct drm_i915_private *dev_priv)
 {
 	struct drm_i915_gem_object *bo;
@@ -1113,6 +1489,212 @@ static void hsw_disable_metric_set(struct drm_i915_private *dev_priv)
 				      ~GT_NOA_ENABLE));
 }
 
+/* NB: It must always remain pointer safe to run this even if the OA unit
+ * has been disabled.
+ *
+ * It's fine to put out-of-date values into these per-context registers
+ * in the case that the OA unit has been disabled.
+ */
+static void gen8_update_reg_state_unlocked(struct i915_gem_context *ctx,
+					   u32 *reg_state)
+{
+	struct drm_i915_private *dev_priv = ctx->i915;
+	const struct i915_oa_reg *flex_regs = dev_priv->perf.oa.flex_regs;
+	int n_flex_regs = dev_priv->perf.oa.flex_regs_len;
+	u32 ctx_oactxctrl = dev_priv->perf.oa.ctx_oactxctrl_offset;
+	u32 ctx_flexeu0 = dev_priv->perf.oa.ctx_flexeu0_offset;
+	/* The MMIO offsets for Flex EU registers aren't contiguous */
+	u32 flex_mmio[] = {
+		i915_mmio_reg_offset(EU_PERF_CNTL0),
+		i915_mmio_reg_offset(EU_PERF_CNTL1),
+		i915_mmio_reg_offset(EU_PERF_CNTL2),
+		i915_mmio_reg_offset(EU_PERF_CNTL3),
+		i915_mmio_reg_offset(EU_PERF_CNTL4),
+		i915_mmio_reg_offset(EU_PERF_CNTL5),
+		i915_mmio_reg_offset(EU_PERF_CNTL6),
+	};
+	int i;
+
+	reg_state[ctx_oactxctrl] = i915_mmio_reg_offset(GEN8_OACTXCONTROL);
+	reg_state[ctx_oactxctrl+1] = (dev_priv->perf.oa.period_exponent <<
+				      GEN8_OA_TIMER_PERIOD_SHIFT) |
+				     (dev_priv->perf.oa.periodic ?
+				      GEN8_OA_TIMER_ENABLE : 0) |
+				     GEN8_OA_COUNTER_RESUME;
+
+	for (i = 0; i < ARRAY_SIZE(flex_mmio); i++) {
+		u32 state_offset = ctx_flexeu0 + i * 2;
+		u32 mmio = flex_mmio[i];
+
+		/* This arbitrary default will select the 'EU FPU0 Pipeline
+		 * Active' event. In the future it's anticipated that there
+		 * will be an explicit 'No Event' we can select, but not yet...
+		 */
+		u32 value = 0;
+		int j;
+
+		for (j = 0; j < n_flex_regs; j++) {
+			if (i915_mmio_reg_offset(flex_regs[j].addr) == mmio) {
+				value = flex_regs[j].value;
+				break;
+			}
+		}
+
+		reg_state[state_offset] = mmio;
+		reg_state[state_offset+1] = value;
+	}
+}
+
+/* Manages updating the per-context aspects of the OA stream
+ * configuration across all contexts.
+ *
+ * The awkward consideration here is that OACTXCONTROL controls the
+ * exponent for periodic sampling which is primarily used for system
+ * wide profiling where we'd like a consistent sampling period even in
+ * the face of context switches.
+ *
+ * Our approach of updating the register state context (as opposed to
+ * say using a workaround batch buffer) ensures that the hardware
+ * won't automatically reload an out-of-date timer exponent even
+ * transiently before a WA BB could be parsed.
+ *
+ * This function needs to:
+ * - Ensure the currently running context's per-context OA state is
+ *   updated
+ * - Ensure that all existing contexts will have the correct per-context
+ *   OA state if they are scheduled for use.
+ * - Ensure any new contexts will be initialized with the correct
+ *   per-context OA state.
+ *
+ * Note: it's only the RCS/Render context that has any OA state.
+ */
+static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv)
+{
+	struct i915_gem_context *ctx;
+	int ret;
+
+	ret = i915_mutex_lock_interruptible(&dev_priv->drm);
+	if (ret)
+		return ret;
+
+	/* Switch away from any user context.  */
+	ret = i915_gem_switch_to_kernel_context(dev_priv);
+	if (ret)
+		goto out;
+
+	/* The OA register config is setup through the context image. This image
+	 * might be written to by the GPU on context switch (in particular on
+	 * lite-restore). This means we can't safely update a context's image,
+	 * if this context is scheduled/submitted to run on the GPU.
+	 *
+	 * We could emit the OA register config through the batch buffer but
+	 * this might leave small interval of time where the OA unit is
+	 * configured at an invalid sampling period.
+	 *
+	 * So far the best way to work around this issue seems to be draining
+	 * the GPU from any submitted work.
+	 */
+	ret = i915_gem_wait_for_idle(dev_priv,
+				     I915_WAIT_INTERRUPTIBLE |
+				     I915_WAIT_LOCKED);
+	if (ret)
+		goto out;
+
+	/* Update all contexts now that we've stalled the submission. */
+	list_for_each_entry(ctx, &dev_priv->context_list, link) {
+		struct intel_context *ce = &ctx->engine[RCS];
+		u32 *regs;
+
+		/* OA settings will be set upon first use */
+		if (!ce->state)
+			continue;
+
+		regs = i915_gem_object_pin_map(ce->state->obj, I915_MAP_WB);

drivers/gpu/drm/i915/i915_perf.c static void hsw_disable_metric_set(struct drm_i915_private *dev_priv) · line 1622

Quoted 10 lines
drivers/gpu/drm/i915/i915_perf.c +10 -0
+		if (IS_ERR(regs)) {
+			ret = PTR_ERR(regs);
+			goto out;
+		}
+
+		ce->state->obj->mm.dirty = true;
+		regs += LRC_STATE_PN * PAGE_SIZE / sizeof(*regs);
+		/* Probably best since we don't really want to expose
+		 * the LRC_STATE_PN trick.
+		 */

Sorry, was writing comments to myself :) This is not about the code, just about the interface to export mapping the context from intel_lrc.c

The only worry I have here is a failure whilst pinning the contexts, leaving some updated, some not, and upon error we will assume the change did not occur.

Other than that, this looks correct, and fwiw I do much prefer the idea of the OA adjustment being synchronous wrt to request submission (that all previous requests complete using the current OA settings and all new submissions will use the next OA config). The downside is that it may stall the driver for an indefinite period of time - but for a global config that seems acceptable in order to have it well defined. Per-context OA configs presumably are faster to apply?

For the context image adjustment,

and Matthew can take responsibility for having checked the rest ;) -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx