Skip to contentSkip to conversation

Page faults to help user space debug

PATCH In review Ben Widawsky 4 participants 18 messages Jun 28, 2013 – Jul 6, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Page faults to help user space debug

This series originated from the request from Paul, "can you enable page faults"? After some though and discussion, we came up with 3 debug features to implement:

1. echo 1 > /sys/kernel/debug/dri/0/i915_debug_flags Make all scratch pages fault on access

2. echo 2 > /sys/kernel/debug/dri/0/i915_debug_flags
Unbind all buffers synchronously, so any batch which has bad code will fault every time and not get lucky by hitting a buffer it bound. This will not catch cases where it accidentally reads/writes someone else's buffer.

3. echo 4 > /sys/kernel/debug/dri/0/i915_debug_flags
Add an extra page to all user allocated objects which will fault when accessed. This is meant to catch the case where mesa overruns one of it's local buffers. It's meant to be a finer grained version of #1, and was a specific request.

Personally, I don't know how useful page faults are. The one time I tried to turn it on previously, I remember getting a hard hang, but my memory if very fuzzy.

Putt this out there for review, and hopefully for Paul to test as soon as he has something read.

Ben Widawsky (6):
drm/i915: Faults for scratch PTEs
drm/i915: Debugfs for setting debug_flags drm/i915: Reset scratch pages when using debug_flags drm/i915: Synchronous execbuf for debug drm/i915: Let userspace create a faultable pad page drm/i915: distinguish pad and fault pages

 drivers/gpu/drm/i915/i915_debugfs.c        | 53 +++++++++++++++++++
 drivers/gpu/drm/i915/i915_drv.c            |  3 ++
 drivers/gpu/drm/i915/i915_drv.h            | 17 +++++--
 drivers/gpu/drm/i915/i915_gem.c            |  7 +++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  7 ++-
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 81 +++++++++++++++++++++++++-----
 6 files changed, 151 insertions(+), 17 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/6 drm/i915: Faults for scratch PTEs

Using a new "debug_flags" we'll be able to configure some options which make it easier for user space to quickly identify when there is a bug in their code.

There is no way to yet turn it on.

---
 drivers/gpu/drm/i915/i915_debugfs.c | 1 +
 drivers/gpu/drm/i915/i915_drv.c     | 3 +++
 drivers/gpu/drm/i915/i915_drv.h     | 4 ++++
 drivers/gpu/drm/i915/i915_gem_gtt.c | 9 +++++++++
 4 files changed, 17 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +1 -0
index b3a2dbe..4d9d8a1 100644
@@ -812,6 +812,7 @@ static int i915_error_state(struct i915_error_state_file_priv *error_priv,
 	err_printf(m, "FORCEWAKE: 0x%08x\n", error->forcewake);
 	err_printf(m, "DERRMR: 0x%08x\n", error->derrmr);
 	err_printf(m, "CCID: 0x%08x\n", error->ccid);
+	err_printf(m, "debug_flags: 0x%llx\n", dev_priv->debug_flags);
 
 	for (i = 0; i < dev_priv->num_fence_regs; i++)
 		err_printf(m, "  fence[%d] = %08llx\n", i, error->fence[i]);
drivers/gpu/drm/i915/i915_drv.c +3 -0
index 062cbda..5718f45 100644
@@ -940,6 +940,9 @@ int i915_reset(struct drm_device *dev)
 		return ret;
 	}
 
+	/* Assume debug flags may have been the cause of the problem */
+	dev_priv->debug_flags = 0;
+
 	/* Ok, now get things going again... */
 
 	/*
drivers/gpu/drm/i915/i915_drv.h +4 -0
index 56bd82b..7a98f7e 100644
@@ -1176,6 +1176,10 @@ typedef struct drm_i915_private {
 	/* Old dri1 support infrastructure, beware the dragons ya fools entering
 	 * here! */
 	struct i915_dri1_state dri1;
+
+#define I915_DEBUG_NONE 0
+#define I915_SCRATCH_FAULTS (1<<0)
+	u64 debug_flags;
 } drm_i915_private_t;
 
 /* Iterate over initialised rings */
drivers/gpu/drm/i915/i915_gem_gtt.c +9 -0
index 5101ab6..982c791 100644
@@ -41,6 +41,8 @@
 #define GEN6_PTE_CACHE_LLC		(2 << 1)
 #define GEN6_PTE_CACHE_LLC_MLC		(3 << 1)
 #define GEN6_PTE_ADDR_ENCODE(addr)	GEN6_GTT_ADDR_ENCODE(addr)
+/* Use a pattern to make debug a bit easier */
+#define GEN6_PTE_FAULT			0xbaddc0de
 
 static gen6_gtt_pte_t gen6_pte_encode(struct drm_device *dev,
 				      dma_addr_t addr,
@@ -185,6 +187,7 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 				   unsigned first_entry,
 				   unsigned num_entries)
 {
+	struct drm_i915_private *dev_priv = ppgtt->dev->dev_private;
 	gen6_gtt_pte_t *pt_vaddr, scratch_pte;
 	unsigned act_pt = first_entry / I915_PPGTT_PT_ENTRIES;
 	unsigned first_pte = first_entry % I915_PPGTT_PT_ENTRIES;
@@ -194,6 +197,9 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 					ppgtt->scratch_page_dma_addr,
 					I915_CACHE_LLC);
 
+	if (unlikely(dev_priv->debug_flags & I915_SCRATCH_FAULTS))
+		scratch_pte = GEN6_PTE_FAULT;
+
 	while (num_entries) {
 		last_pte = first_pte + num_entries;
 		if (last_pte > I915_PPGTT_PT_ENTRIES)
@@ -521,6 +527,8 @@ static void gen6_ggtt_clear_range(struct drm_device *dev,
 	scratch_pte = dev_priv->gtt.pte_encode(dev,
 					       dev_priv->gtt.scratch_page_dma,
 					       I915_CACHE_LLC);
+	if (unlikely(dev_priv->debug_flags & I915_SCRATCH_FAULTS))
+		scratch_pte = GEN6_PTE_FAULT;
 	for (i = 0; i < num_entries; i++)
 		iowrite32(scratch_pte, &gtt_base[i]);
 	readl(gtt_base);
@@ -623,6 +631,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	struct drm_i915_gem_object *obj;
 	unsigned long hole_start, hole_end;
 
+	BUILD_BUG_ON(GEN6_PTE_FAULT & GEN6_PTE_VALID);
 	BUG_ON(mappable_end > end);
 
 	/* Subtract the guard page ... */
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH 2/6 drm/i915: Debugfs for setting debug_flags

In reply to Ben Widawsky

The lock in the setter isn't really necessary - however we're doing some more stuff there presently, so just leave it in...

---
 drivers/gpu/drm/i915/i915_debugfs.c | 45 +++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +45 -0
index 4d9d8a1..5d6a020 100644
@@ -2182,6 +2182,42 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_cache_sharing_fops,
 			i915_cache_sharing_get, i915_cache_sharing_set,
 			"%llu\n");
 
+static int
+i915_debug_flags_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+
+	*val = dev_priv->debug_flags;
+
+	return 0;
+}
+
+static int
+i915_debug_flags_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	int ret;
+
+	/* FIXME?: Nothing prevents this from working pre GEN6; but don't want
+	 * to touch AGP code.
+	 */
+	if (INTEL_INFO(dev)->gen < 6 && val & I915_SCRATCH_FAULTS)
+		DRM_DEBUG_DRIVER("Tried to set unsupported SCRATCH_FAULTS\n");
+
+	ret = mutex_lock_interruptible(&dev->struct_mutex);
+	if (ret)
+		return ret;
+	dev_priv->debug_flags = val;
+	mutex_unlock(&dev_priv->dev->struct_mutex);
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_debug_flag_fops,
+			i915_debug_flags_get, i915_debug_flags_set,
+			"%llu\n");
+
 /* As the drm_debugfs_init() routines are called before dev->dev_private is
  * allocated we need to hook into the minor for release. */
 static int
@@ -2366,6 +2402,13 @@ int i915_debugfs_init(struct drm_minor *minor)
 	if (ret)
 		return ret;
 
+	ret = i915_debugfs_create(minor->debugfs_root, minor,
+				 "i915_debug_flags",
+				 &i915_debug_flag_fops);
+	if (ret)
+		return ret;
+
+
 	return drm_debugfs_create_files(i915_debugfs_list,
 					I915_DEBUGFS_ENTRIES,
 					minor->debugfs_root, minor);
@@ -2393,6 +2436,8 @@ void i915_debugfs_cleanup(struct drm_minor *minor)
 				 1, minor);
 	drm_debugfs_remove_files((struct drm_info_list *) &i915_next_seqno_fops,
 				 1, minor);
+	drm_debugfs_remove_files((struct drm_info_list *) &i915_debug_flag_fops,
+				 1, minor);
 }
 
 #endif /* CONFIG_DEBUG_FS */
Signature · 1 line
1.8.3.1
#3 lore raw

PATCH 3/6 drm/i915: Reset scratch pages when using debug_flags

In reply to Ben Widawsky

Instead of waiting an indeterminate amount of time for ranges to get cleared with the PTEs that will fault, do it immediately.

---
 drivers/gpu/drm/i915/i915_debugfs.c |  7 +++++++
 drivers/gpu/drm/i915/i915_drv.h     |  4 +++-
 drivers/gpu/drm/i915/i915_gem_gtt.c | 20 ++++++++++++++++++++
 3 files changed, 30 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_debugfs.c +7 -0
index 5d6a020..86e750c 100644
@@ -2198,6 +2198,7 @@ i915_debug_flags_set(void *data, u64 val)
 {
 	struct drm_device *dev = data;
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	u64 old_flags = dev_priv->debug_flags;
 	int ret;
 
 	/* FIXME?: Nothing prevents this from working pre GEN6; but don't want
@@ -2209,7 +2210,13 @@ i915_debug_flags_set(void *data, u64 val)
 	ret = mutex_lock_interruptible(&dev->struct_mutex);
 	if (ret)
 		return ret;
+
 	dev_priv->debug_flags = val;
+
+	/* If setting or unsetting scratch faults, we need special handling */
+	if ((old_flags ^ val) & I915_SCRATCH_FAULTS)
+		i915_gem_rewrite_scratch_ptes(dev);
+
 	mutex_unlock(&dev_priv->dev->struct_mutex);
 	return 0;
 }
drivers/gpu/drm/i915/i915_drv.h +3 -1
index 7a98f7e..b99087f8 100644
@@ -1865,7 +1865,9 @@ static inline void i915_gem_chipset_flush(struct drm_device *dev)
 	if (INTEL_INFO(dev)->gen < 6)
 		intel_gtt_chipset_flush();
 }
-
+#ifdef CONFIG_DEBUG_FS
+void i915_gem_rewrite_scratch_ptes(struct drm_device *dev);
+#endif
 
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct drm_device *dev, int min_size,
drivers/gpu/drm/i915/i915_gem_gtt.c +20 -0
index 982c791..f71636e 100644
@@ -889,3 +889,23 @@ int i915_gem_gtt_init(struct drm_device *dev)
 
 	return 0;
 }
+
+#ifdef CONFIG_DEBUG_FS
+void i915_gem_rewrite_scratch_ptes(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct drm_mm_node *n;
+	unsigned long start, end;
+	struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt;
+
+	drm_mm_for_each_hole(n, &dev_priv->mm.gtt_space, start, end) {
+		dev_priv->gtt.gtt_clear_range(dev,
+					      start >> PAGE_SHIFT,
+					      (end - start) >> PAGE_SHIFT);
+		if (ppgtt)
+			ppgtt->clear_range(ppgtt,
+					   start >> PAGE_SHIFT,
+					   (end - start) >> PAGE_SHIFT);
+	}
+}
+#endif
Signature · 1 line
1.8.3.1
#4 lore raw

PATCH 4/6 drm/i915: Synchronous execbuf for debug

In reply to Ben Widawsky

In order to be able to hit the previously introduced page faults, userspace may also want to make sure it's buffers are evicted, so that it doesn't get lucky on accident.

This debug feature will have a horrible performance implication (I measure roughly 3x in the negative direction for some basic tests).

---
 drivers/gpu/drm/i915/i915_drv.h            | 1 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 7 +++++--
 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
index b99087f8..c9e38a6 100644
@@ -1179,6 +1179,7 @@ typedef struct drm_i915_private {
 
 #define I915_DEBUG_NONE 0
 #define I915_SCRATCH_FAULTS (1<<0)
+#define I915_SYNC_EXECBUF (1<<1)
 	u64 debug_flags;
 } drm_i915_private_t;
 
drivers/gpu/drm/i915/i915_gem_execbuffer.c +5 -2
index 87a3227..fe3bb5f 100644
@@ -150,7 +150,8 @@ eb_get_object(struct eb_objects *eb, unsigned long handle)
 }
 
 static void
-eb_destroy(struct eb_objects *eb)
+eb_destroy(struct drm_i915_private *dev_priv,
+	   struct eb_objects *eb)
 {
 	while (!list_empty(&eb->objects)) {
 		struct drm_i915_gem_object *obj;
@@ -158,6 +159,8 @@ eb_destroy(struct eb_objects *eb)
 		obj = list_first_entry(&eb->objects,
 				       struct drm_i915_gem_object,
 				       exec_list);
+		if (unlikely(dev_priv->debug_flags & I915_SYNC_EXECBUF))
+			i915_gem_object_unbind(obj);
 		list_del_init(&obj->exec_list);
 		drm_gem_object_unreference(&obj->base);
 	}
@@ -1087,7 +1090,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	i915_gem_execbuffer_retire_commands(dev, file, ring, batch_obj);
 
 err:
-	eb_destroy(eb);
+	eb_destroy(dev_priv, eb);
 
 	mutex_unlock(&dev->struct_mutex);
 
Signature · 1 line
1.8.3.1
#5 lore raw

PATCH 5/6 drm/i915: Let userspace create a faultable pad page

In reply to Ben Widawsky

Whenever userspace allocates a BO, add one more page. On maps of this BO, make sure that last page will fault if accessed. We don't need to do this for the kernel, since our bugs should be fairly easy to catch already.

The code could be optimized to allocate one less page for this, but it requires a lot of work to make that happen. Instead, do everything as we normally would, and clear out the last page after it's all done.

NOTE: This does not convert previously allocated objects.

---
 drivers/gpu/drm/i915/i915_drv.h     |  2 ++
 drivers/gpu/drm/i915/i915_gem.c     |  7 +++++++
 drivers/gpu/drm/i915/i915_gem_gtt.c | 22 +++++++++++++++++++---
 3 files changed, 28 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index c9e38a6..3b2046b 100644
@@ -1180,6 +1180,7 @@ typedef struct drm_i915_private {
 #define I915_DEBUG_NONE 0
 #define I915_SCRATCH_FAULTS (1<<0)
 #define I915_SYNC_EXECBUF (1<<1)
+#define I915_PAD_PAGE (1<<2)
 	u64 debug_flags;
 } drm_i915_private_t;
 
@@ -1307,6 +1308,7 @@ struct drm_i915_gem_object {
 	unsigned int has_aliasing_ppgtt_mapping:1;
 	unsigned int has_global_gtt_mapping:1;
 	unsigned int has_dma_mapping:1;
+	unsigned int has_pad_page:1;
 
 	struct sg_table *pages;
 	int pages_pin_count;
drivers/gpu/drm/i915/i915_gem.c +7 -0
index c68b90f..f84aada 100644
@@ -205,6 +205,7 @@ i915_gem_create(struct drm_file *file,
 		uint64_t size,
 		uint32_t *handle_p)
 {
+	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj;
 	int ret;
 	u32 handle;
@@ -213,11 +214,17 @@ i915_gem_create(struct drm_file *file,
 	if (size == 0)
 		return -EINVAL;
 
+	if (dev_priv->debug_flags & I915_PAD_PAGE)
+		size += PAGE_SIZE;
+
 	/* Allocate the new object */
 	obj = i915_gem_alloc_object(dev, size);
 	if (obj == NULL)
 		return -ENOMEM;
 
+	if (dev_priv->debug_flags & I915_PAD_PAGE)
+		obj->has_pad_page = 1;
+
 	ret = drm_gem_handle_create(file, &obj->base, &handle);
 	if (ret) {
 		drm_gem_object_release(&obj->base);
drivers/gpu/drm/i915/i915_gem_gtt.c +19 -3
index f71636e..187738f 100644
@@ -43,6 +43,7 @@
 #define GEN6_PTE_ADDR_ENCODE(addr)	GEN6_GTT_ADDR_ENCODE(addr)
 /* Use a pattern to make debug a bit easier */
 #define GEN6_PTE_FAULT			0xbaddc0de
+#define GEN6_PAD_PTE_FAULT		0x0c0ffee
 
 static gen6_gtt_pte_t gen6_pte_encode(struct drm_device *dev,
 				      dma_addr_t addr,
@@ -200,6 +201,9 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 	if (unlikely(dev_priv->debug_flags & I915_SCRATCH_FAULTS))
 		scratch_pte = GEN6_PTE_FAULT;
 
+	if (unlikely(dev_priv->debug_flags & I915_PAD_PAGE))
+		scratch_pte = GEN6_PAD_PTE_FAULT;
+
 	while (num_entries) {
 		last_pte = first_pte + num_entries;
 		if (last_pte > I915_PPGTT_PT_ENTRIES)
@@ -385,9 +389,13 @@ void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    struct drm_i915_gem_object *obj,
 			    enum i915_cache_level cache_level)
 {
-	ppgtt->insert_entries(ppgtt, obj->pages,
-			      obj->gtt_space->start >> PAGE_SHIFT,
-			      cache_level);
+	unsigned int first = obj->gtt_space->start >> PAGE_SHIFT;
+
+	ppgtt->insert_entries(ppgtt, obj->pages, first, cache_level);
+	if (unlikely(obj->has_pad_page)) {
+		first += (obj->gtt_space->size >> PAGE_SHIFT) - 1;
+		ppgtt->clear_range(ppgtt, first, 1);
+	}
 }
 
 void i915_ppgtt_unbind_object(struct i915_hw_ppgtt *ppgtt,
@@ -529,6 +537,8 @@ static void gen6_ggtt_clear_range(struct drm_device *dev,
 					       I915_CACHE_LLC);
 	if (unlikely(dev_priv->debug_flags & I915_SCRATCH_FAULTS))
 		scratch_pte = GEN6_PTE_FAULT;
+	if (unlikely(dev_priv->debug_flags & I915_PAD_PAGE))
+		scratch_pte = GEN6_PAD_PTE_FAULT;
 	for (i = 0; i < num_entries; i++)
 		iowrite32(scratch_pte, &gtt_base[i]);
 	readl(gtt_base);
@@ -560,10 +570,15 @@ void i915_gem_gtt_bind_object(struct drm_i915_gem_object *obj,
 {
 	struct drm_device *dev = obj->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	unsigned int first = obj->gtt_space->start >> PAGE_SHIFT;
 
 	dev_priv->gtt.gtt_insert_entries(dev, obj->pages,
 					 obj->gtt_space->start >> PAGE_SHIFT,
 					 cache_level);
+	if (unlikely(obj->has_pad_page)) {
+		first += (obj->gtt_space->size >> PAGE_SHIFT) - 1;
+		dev_priv->gtt.gtt_clear_range(dev, first, 1);
+	}
 
 	obj->has_global_gtt_mapping = 1;
 }
@@ -632,6 +647,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	unsigned long hole_start, hole_end;
 
 	BUILD_BUG_ON(GEN6_PTE_FAULT & GEN6_PTE_VALID);
+	BUILD_BUG_ON(GEN6_PAD_PTE_FAULT & GEN6_PTE_VALID);
 	BUG_ON(mappable_end > end);
 
 	/* Subtract the guard page ... */
Signature · 1 line
1.8.3.1
#6 lore raw

PATCH 6/6 drm/i915: distinguish pad and fault pages

In reply to Ben Widawsky

For finer grained debug, make it possible to have both magic numbers be present for the PTEs if both faulting and pad pages options are enabled. Previously the padding magic number took precedence.

This change might not seem worthwhile to some. It helped me verify my code was correct, and I like it. It can be dropped without too much impact to the series though.

---
 drivers/gpu/drm/i915/i915_drv.h     |  6 ++++--
 drivers/gpu/drm/i915/i915_gem_gtt.c | 42 ++++++++++++++++++++++++-------------
 2 files changed, 31 insertions(+), 17 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +4 -2
index 3b2046b..aaf9554 100644
@@ -475,7 +475,8 @@ struct i915_gtt {
 	void (*gtt_remove)(struct drm_device *dev);
 	void (*gtt_clear_range)(struct drm_device *dev,
 				unsigned int first_entry,
-				unsigned int num_entries);
+				unsigned int num_entries,
+				bool pad);
 	void (*gtt_insert_entries)(struct drm_device *dev,
 				   struct sg_table *st,
 				   unsigned int pg_start,
@@ -499,7 +500,8 @@ struct i915_hw_ppgtt {
 	/* pte functions, mirroring the interface of the global gtt. */
 	void (*clear_range)(struct i915_hw_ppgtt *ppgtt,
 			    unsigned int first_entry,
-			    unsigned int num_entries);
+			    unsigned int num_entries,
+			    bool pad);
 	void (*insert_entries)(struct i915_hw_ppgtt *ppgtt,
 			       struct sg_table *st,
 			       unsigned int pg_start,
drivers/gpu/drm/i915/i915_gem_gtt.c +27 -15

138 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#7 lore raw
In reply to Ben Widawsky
Quoted 3 lines · Ben Widawsky

This series originated from the request from Paul, "can you enable page faults"? After some though and discussion, we came up with 3 debug features to implement:

The issue lies in that the CS and EU units like to prefetch 128 bytes and will cross page boundaries. Userspace is rather lax in providing the extra page (or preventing the read past the end of its bo) and so without adding a sentinel page behind every bo you quickly generate false positives. (Unless you also run a fixed userspace).

If you are prepared to fix userspace, tweaking the kernel not to install scratch pages everywhere is trivial.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#8 lore raw
Quoted 11 lines · Chris Wilson
Quoted 3 lines · Ben Widawsky

This series originated from the request from Paul, "can you enable page faults"? After some though and discussion, we came up with 3 debug features to implement:

The issue lies in that the CS and EU units like to prefetch 128 bytes and will cross page boundaries. Userspace is rather lax in providing the extra page (or preventing the read past the end of its bo) and so without adding a sentinel page behind every bo you quickly generate false positives. (Unless you also run a fixed userspace).

Quoted 6 lines

If you are prepared to fix userspace, tweaking the kernel not to install scratch pages everywhere is trivial.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre

I'll leave the further discussion of whether or not they still want all the features to Paul (I think the synchronous unbinding execbuf is useful regardless).

As for the trivial, "remove the scratch page" I didn't think that was the plan as generally we'd like to carry on as long as possible, and we know faults are fatal. I think keeping the scratch page, even if we didn't have to worry about prefetching is what we want.

Signature · 1 line
Ben Widawsky, Intel Open Source Technology Center
#9 lore raw
Quoted 15 lines · Ben Widawsky
Quoted 5 lines · Chris Wilson
Quoted 3 lines · Ben Widawsky

This series originated from the request from Paul, "can you enable page faults"? After some though and discussion, we came up with 3 debug features to implement:

Quoted 8 lines

The issue lies in that the CS and EU units like to prefetch 128 bytes and will cross page boundaries. Userspace is rather lax in providing the extra page (or preventing the read past the end of its bo) and so without adding a sentinel page behind every bo you quickly generate false positives. (Unless you also run a fixed userspace).

If you are prepared to fix userspace, tweaking the kernel not to install scratch pages everywhere is trivial.

Quoted 8 lines

I'll leave the further discussion of whether or not they still want all the features to Paul (I think the synchronous unbinding execbuf is useful regardless).

As for the trivial, "remove the scratch page" I didn't think that was the plan as generally we'd like to carry on as long as possible, and we know faults are fatal. I think keeping the scratch page, even if we didn't have to worry about prefetching is what we want.

No plans to remove the scratch page as it hides too many bugs, which generally are fatal enough when run for long enough. I was just offering up my experience on using page faults.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#10 lore raw
In reply to Chris Wilson
Quoted 6 lines · Chris Wilson
Quoted 2 lines · Ben Widawsky

This series originated from the request from Paul, "can you enable page faults"? After some though and discussion, we came up with 3 debug

features to

Quoted 1 line

implement:

Quoted 9 lines

The issue lies in that the CS and EU units like to prefetch 128 bytes and will cross page boundaries. Userspace is rather lax in providing the extra page (or preventing the read past the end of its bo) and so without adding a sentinel page behind every bo you quickly generate false positives. (Unless you also run a fixed userspace).

If you are prepared to fix userspace, tweaking the kernel not to install scratch pages everywhere is trivial.

Mesa already adds the necessary padding to EU programs to ensure that prefetch won't cause a page fault, and because of its "stack and heap" model for batch buffers, CS prefetching shouldn't cause a page fault either. I don't know whether the 2D drivers do something similar, so they might potentially need fixing.

Having tracked down a number of painful Mesa bugs over the last two years where we supplied the GPU with an incorrect pointer, or we sized a buffer incorrectly causing other buffers to be corrupted, I am fairly certain that being able to turn off scratch pages will be a benefit in debugging--these debug options should turn confusing, sporadic failures (and failures that don't get noticed for months) into easily reproducible failures that get noticed quickly. So these debug flags will definitely be useful in development.

If the 2D driver is lax about providing the necessary padding for prefetch, then that will make things more difficult, but the debug flags should still be useful. It just means that when using the debug flags, I'll have to run my piglit tests under gbm.

I'll try to give the patches a shot soon to make sure they work.

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#11 lore raw
Quoted 3 lines · Paul Berry
   On 28 June 2013 15:39, Chris Wilson <[1][email protected]>
   wrote:

[snip]

Quoted 1 line

I'll try to give the patches a shot soon to make sure they work.

By the way, if it wasn't clear, any combinations of the debug flags should work, ie. echo 7 > ....

Quoted 4 lines

References

1. mailto:[email protected]

Signature · 1 line
Ben Widawsky, Intel Open Source Technology Center
#12 lore raw
In reply to Paul Berry
Quoted 15 lines · Paul Berry

On 28 June 2013 15:39, Chris Wilson <[1][email protected]> wrote:

On Fri, Jun 28, 2013 at 03:23:31PM -0700, Ben Widawsky wrote:
> This series originated from the request from Paul, "can you enable page
> faults"? �After some though and discussion, we came up with 3 debug features to
> implement:

The issue lies in that the CS and EU units like to prefetch 128 bytes and will cross page boundaries. Userspace is rather lax in providing the extra page (or preventing the read past the end of its bo) and so without adding a sentinel page behind every bo you quickly generate false positives. (Unless you also run a fixed userspace).

Quoted 8 lines

If you are prepared to fix userspace, tweaking the kernel not to install scratch pages everywhere is trivial.

Mesa already adds the necessary padding to EU programs to ensure that prefetch won't cause a page fault, and because of its "stack and heap" model for batch buffers, CS prefetching shouldn't cause a page fault either.� I don't know whether the 2D drivers do something similar, so they might potentially need fixing.

You do not on the BLT ring, and have not done historically. The EU is no longer padded. We have been purposely lax in this area. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#13 lore raw
Quoted 28 lines · Chris Wilson
Quoted 1 line · Paul Berry
   On 28 June 2013 15:39, Chris Wilson <[1][email protected]>
Quoted 5 lines · wrote:

On Fri, Jun 28, 2013 at 03:23:31PM -0700, Ben Widawsky wrote:
> This series originated from the request from Paul, "can you enable page
> faults"? �After some though and discussion, we came up with 3

debug

Quoted 4 lines

features to
> implement:

The issue lies in that the CS and EU units like to prefetch 128

bytes

Quoted 1 line

and will cross page boundaries. Userspace is rather lax in

providing the

Quoted 5 lines

extra page (or preventing the read past the end of its bo) and so without adding a sentinel page behind every bo you quickly generate false positives. (Unless you also run a fixed userspace).

If you are prepared to fix userspace, tweaking the kernel not to

install

Quoted 6 lines

scratch pages everywhere is trivial.

Mesa already adds the necessary padding to EU programs to ensure that prefetch won't cause a page fault, and because of its "stack and heap" model for batch buffers, CS prefetching shouldn't cause a page fault either.� I don't know whether the 2D drivers do something similar, so

Quoted 7 lines

they

Quoted 1 line

might potentially need fixing.

You do not on the BLT ring, and have not done historically. The EU is no longer padded. We have been purposely lax in this area. -Chris

You're right--I didn't realize that we removed the EU padding, and I wasn't thinking about the BLT code.

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#14 lore raw

Can we get a collective status update? I'm touching some code at the moment which will have some uncomfortable conflicts with this series, and I'd like to have an idea on the plan.

Quoted 52 lines · Paul Berry

On 2 July 2013 12:03, Chris Wilson <[1][email protected]> wrote:

Quoted 1 line · Paul Berry

  On 28 June 2013 15:39, Chris Wilson

Quoted 3 lines · <[1][2][email protected]> wrote:
     On Fri, Jun 28, 2013 at 03:23:31PM -0700, Ben Widawsky wrote:
     > This series originated from the request from Paul, "can you

enable

Quoted 2 lines
     page
     > faults"? �After some though and discussion, we came up

with 3 debug

Quoted 4 lines
     features to
     > implement:

   The issue lies in that the CS and EU units like to prefetch

128 bytes

Quoted 1 line

   and will cross page boundaries. Userspace is rather lax in

providing the

Quoted 1 line

   extra page (or preventing the read past the end of its bo)

and so

Quoted 1 line

   without adding a sentinel page behind every bo you quickly

generate

Quoted 3 lines

   false positives. (Unless you also run a fixed userspace).

   If you are prepared to fix userspace, tweaking the kernel not

to install

Quoted 3 lines

   scratch pages everywhere is trivial.

  Mesa already adds the necessary padding to EU programs to ensure

that

Quoted 1 line

  prefetch won't cause a page fault, and because of its "stack and

heap"

Quoted 1 line

  model for batch buffers, CS prefetching shouldn't cause a page

fault

Quoted 1 line

  either.� I don't know whether the 2D drivers do something

similar, so they

Quoted 1 line

  might potentially need fixing.

You do not on the BLT ring, and have not done historically. The EU is no
longer padded. We have been purposely lax in this area.

-Chris

You're right--I didn't realize that we removed the EU padding, and I wasn't thinking about the BLT code.

References

1. mailto:[email protected]
2. mailto:[email protected]

Signature · 5 lines
Ben Widawsky, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#15 lore raw
In reply to Ben Widawsky
Quoted 4 lines · Ben Widawsky

2. echo 2 > /sys/kernel/debug/dri/0/i915_debug_flags
Unbind all buffers synchronously, so any batch which has bad code will fault every time and not get lucky by hitting a buffer it bound. This will not catch cases where it accidentally reads/writes someone else's buffer.

We already have this as the drop_caches debugfs interface. Chris used it to reproduce the 3.7 reloc-tree regression, the only downside is that no one ever got around to write the respective i-g-t ... :( -Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

#16 lore raw
Quoted 10 lines · Daniel Vetter
Quoted 4 lines · Ben Widawsky

2. echo 2 > /sys/kernel/debug/dri/0/i915_debug_flags
Unbind all buffers synchronously, so any batch which has bad code will fault every time and not get lucky by hitting a buffer it bound. This will not catch cases where it accidentally reads/writes someone else's buffer.

We already have this as the drop_caches debugfs interface. Chris used it to reproduce the 3.7 reloc-tree regression, the only downside is that no one ever got around to write the respective i-g-t ... :( -Daniel

In order to achieve the same behavior, one must instrument every GPU client to use that interface before, and after every batch (you can skip one if you're positive all clients have opted in). Also, every client must be run as root.

This mechanism I've introduced is much simpler to achieve the intended result.

Quoted 4 lines

--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Signature · 1 line
Ben Widawsky, Intel Open Source Technology Center
#17 lore raw
Quoted 12 lines · Ben Widawsky
Quoted 10 lines · Daniel Vetter
Quoted 4 lines · Ben Widawsky

2. echo 2 > /sys/kernel/debug/dri/0/i915_debug_flags
Unbind all buffers synchronously, so any batch which has bad code will fault every time and not get lucky by hitting a buffer it bound. This will not catch cases where it accidentally reads/writes someone else's buffer.

We already have this as the drop_caches debugfs interface. Chris used it to reproduce the 3.7 reloc-tree regression, the only downside is that no one ever got around to write the respective i-g-t ... :( -Daniel

Quoted 7 lines

In order to achieve the same behavior, one must instrument every GPU client to use that interface before, and after every batch (you can skip one if you're positive all clients have opted in). Also, every client must be run as root.

This mechanism I've introduced is much simpler to achieve the intended result.

Oh right, your patch is doing something else completely. One thing which would be neat on top of this is to randomize the locations of each bo a bit (varibale padding in front should be good enough). That way we can check whether the relocations are all handled properly. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch