Skip to contentSkip to conversation

drm/i915: Wait and retry if there is no space in the aperture mappable area

PATCH In review Siluvery, Arun 4 participants 11 messages Oct 22, 2013 – Dec 5, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/i915: Wait and retry if there is no space in the aperture mappable area

From: "Siluvery, Arun" <[email protected]>

When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in
signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.

---
 drivers/gpu/drm/i915/i915_gem.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +4 -0
index e7b39d7..927a27b 100644
@@ -3275,6 +3275,10 @@ search_free:
 					       obj->cache_level,
 					       map_and_fenceable,
 					       nonblocking);
+
+		if (ret == -ENOSPC)
+			ret = i915_gem_evict_everything(dev);
+
 		if (ret == 0)
 			goto search_free;
 
Signature · 1 line
1.8.4
#1 lore raw
Quoted 6 lines · Arun

From: "Siluvery, Arun" <[email protected]>

When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in
signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.

Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an evict_everything in here breaks reservations, so I think you are just papering over a bug.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#2 lore raw
Quoted 11 lines · Chris Wilson
Quoted 6 lines · Arun

From: "Siluvery, Arun" <[email protected]>

When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in
signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.

Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an evict_everything in here breaks reservations, so I think you are just papering over a bug.

If we want to fix this for real (i.e. allow userspace to reliably map stuff, mabye even bigger than the aperture) we need to fall back to suballocating tile-row aligned strides of the buffer (maybe pick the tile row multiply in between 1M-2M). Until that's done userspace can't rely on gtt mmaps relibly working for large buffers. The current heuristics we're using is half of the mappable space, but that's probably a bit too optimistic.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#3 lore raw
Quoted 13 lines · Chris Wilson
Quoted 11 lines · Arun
Quoted 6 lines

From: "Siluvery, Arun" <[email protected]>

When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.

Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an evict_everything in here breaks reservations, so I think you are just papering over a bug.

Quoted 7 lines

If we want to fix this for real (i.e. allow userspace to reliably map stuff, mabye even bigger than the aperture) we need to fall back to suballocating tile-row aligned strides of the buffer (maybe pick the tile row multiply in between 1M-2M). Until that's done userspace can't rely on gtt mmaps relibly working for large buffers. The current heuristics we're using is half of the mappable space, but that's probably a bit too optimistic. -Daniel

Is calling i915_gem_evict_everything() actually dangerous ? Despite its name, it appears to only evict unpinned buffers. Or am I missing something ?

What it does do is update the status of buffers which are no longer in use on the ring by calling i915_gem_retire_requests(). So from what I can tell (from a 10 minute trawl of the code admittedly) all this patch is doing is getting a more up to date view of the GTT demands so that we only fail with ENOSPC if there are no pinned buffers which can now be unpinned.

It doesn't address our underlying issue - userspace should still handle ENOSPC gracefully. However it certainly seems to be improving things considerably, so is beneficial if it really is a safe thing to do.

Jon

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

#4 lore raw
Quoted 5 lines · Jon

Is calling i915_gem_evict_everything() actually dangerous ? Despite its name, it appears to only evict unpinned buffers. Or am I missing something ?

What it does do is update the status of buffers which are no longer in use on the ring by calling i915_gem_retire_requests(). So from what I can tell (from a 10 minute trawl of the code admittedly) all this patch is doing is getting a more up to date view of the GTT demands so that we only fail with ENOSPC if there are no pinned buffers which can now be unpinned.

It doesn't address our underlying issue - userspace should still handle ENOSPC gracefully. However it certainly seems to be improving things considerably, so is beneficial if it really is a safe thing to do.

We use evict_everything in the execbuf code to defragment the gtt. But that's only beneficial because execbuffer needs to find space for multiple buffers at once. For just binding one single buffer the eviction code should scan all lists. Throwing in an additional evict_everything on top won't help at all.

Of course if you throw in copious amounts of evict_everything once you're in a bad spot the defragmentation this causes might help to get out of that corner. But it really shouldn't help with preventing SIGBUS in the first place. So I really wonder what exactly's going on here. Can you perhaps distill a testcase and share it (preferrably as an i-g-t patch ofc)?

Thanks, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#5 lore raw
In reply to Bloomfield, Jon
Quoted 21 lines · Jon
Quoted 13 lines · Chris Wilson
Quoted 11 lines · Arun
Quoted 6 lines

From: "Siluvery, Arun" <[email protected]>

When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.

Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an evict_everything in here breaks reservations, so I think you are just papering over a bug.

Quoted 7 lines

If we want to fix this for real (i.e. allow userspace to reliably map stuff, mabye even bigger than the aperture) we need to fall back to suballocating tile-row aligned strides of the buffer (maybe pick the tile row multiply in between 1M-2M). Until that's done userspace can't rely on gtt mmaps relibly working for large buffers. The current heuristics we're using is half of the mappable space, but that's probably a bit too optimistic. -Daniel

Quoted 1 line

Is calling i915_gem_evict_everything() actually dangerous ? Despite its name, it appears to only evict unpinned buffers. Or am I missing something ?

It breaks the expectations of the callers, which when they have multiple buffers to reserve at once have a better strategy than can be performed individually on a buffer. (That is when evict-everything becomes useful.)

Quoted 1 line

What it does do is update the status of buffers which are no longer in use on the ring by calling i915_gem_retire_requests(). So from what I can tell (from a 10 minute trawl of the code admittedly) all this patch is doing is getting a more up to date view of the GTT demands so that we only fail with ENOSPC if there are no pinned buffers which can now be unpinned.

No, evict-something also scans the requests to find the first available slot large enough to fit the buffer. (Except when evict-something is instructed not to block for pending retirement.)

There's a bug here if evict-everything finds something that evict-something doesn't.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#6 lore raw
In reply to Siluvery, Arun

Picking up this old thread, hopefully I haven't forgotten anyone from the cc list interested in the previous discussions ...

drivers/gpu/drm/i915/i915_gem.c search_free: · line 3281

Quoted 12 lines · Arun
drivers/gpu/drm/i915/i915_gem.c +4 -0
index e7b39d7..927a27b 100644
@@ -3275,6 +3275,10 @@ search_free:
                                               obj->cache_level,
                                               map_and_fenceable,
                                               nonblocking);
+
+               if (ret == -ENOSPC)
+                       ret = i915_gem_evict_everything(dev);
+

Despite my earlier claim this is indeed a real bugfix - compared to evict_something this will move away any hw context objects. So this isn't just purely duct-tape.

I think the right approach for this patch would be to create a i915_gem_evict_unpin_special_objects function or similar which atm just does the do_switch to the default context. We can then call this here and in evict_everything.

Can you please look into that and test whether that helps as well as your current hack?

I've noticed this while trying to create a testcase for the recently fought do_switch eviction related oops. I just couldn't and that got me thinking a bit ;-)

Cheers, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#7 lore raw
Quoted 24 lines · Daniel Vetter

drivers/gpu/drm/i915/i915_gem.c search_free: · line 3281

Quoted 12 lines · Arun
drivers/gpu/drm/i915/i915_gem.c +4 -0
index e7b39d7..927a27b 100644
@@ -3275,6 +3275,10 @@ search_free:
                                               obj->cache_level,
                                               map_and_fenceable,
                                               nonblocking);
+
+               if (ret == -ENOSPC)
+                       ret = i915_gem_evict_everything(dev);
+

Despite my earlier claim this is indeed a real bugfix - compared to evict_something this will move away any hw context objects. So this isn't just purely duct-tape.

I think the right approach for this patch would be to create a i915_gem_evict_unpin_special_objects function or similar which atm just does the do_switch to the default context. We can then call this here and in evict_everything.

Quoted 6 lines

Can you please look into that and test whether that helps as well as your current hack?

I've noticed this while trying to create a testcase for the recently fought do_switch eviction related oops. I just couldn't and that got me thinking a bit ;-)

Fyi I've just pushed out igt/gem_ctx_exec/eviction which exercises our current failure to evict the current contxt. It's going through execbuf though and doesn't test the gtt mmap path. Fixing up that is imo only worth it as part of the large plan I've outlined somewhere else in this thread. Atm the test fails, but it should work with proper context eviction. So this should be useful for a more minimal testcase.

Since the testcase starts with a too-big execbuf that should also exercise the endless loop you've outlined in the internal mail, but I think for safety we want to add one more execbuf call in the eviction test with a 3rd context and just 1 buffer too much in the reloc list. That should perfectly reproduce your failure case.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#8 lore raw

PATCH 1/1 drm/i915: When evicting something fails, try unpinning old contexts and framebuffers

In reply to Daniel Vetter

Before declaring that we cannot fit an object into the aperture, we should first check if we can clear the aperture of pinned objects that are now idle. These pinned objects include old hardware contexts and old scanout buffers.

---
 drivers/gpu/drm/i915/i915_gem.c       |  1 +
 drivers/gpu/drm/i915/i915_gem_evict.c | 11 ++++++++---
 drivers/gpu/drm/i915/intel_display.c  |  8 ++++++++
 drivers/gpu/drm/i915/intel_drv.h      |  1 +
 4 files changed, 18 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +1 -0
index a99aaff1656a..8756a44774a2 100644
@@ -2921,6 +2921,7 @@ int i915_gpu_idle(struct drm_device *dev)
 			return ret;
 	}
 
+	intel_wait_for_pending_flips(dev);
 	return 0;
 }
 
drivers/gpu/drm/i915/i915_gem_evict.c +8 -3
index b7376533633d..a66b82dac2f8 100644
@@ -80,6 +80,7 @@ i915_gem_evict_something(struct drm_device *dev, struct i915_address_space *vm,
 	 */
 
 	INIT_LIST_HEAD(&unwind_list);
+search_again:
 	if (mappable) {
 		BUG_ON(!i915_is_ggtt(vm));
 		drm_mm_init_scan_with_range(&vm->mm, min_size,
@@ -115,10 +116,14 @@ none:
 		list_del_init(&vma->exec_list);
 	}
 
-	/* We expect the caller to unpin, evict all and try again, or give up.
-	 * So calling i915_gem_evict_vm() is unnecessary.
+	/* Can we unpin some objects such as idle hw contents,
+	 * or pending flips?
 	 */
-	return -ENOSPC;
+	ret = nonblocking ? -ENOSPC : i915_gpu_idle(dev);
+	if (ret)
+		return ret;
+
+	goto search_again;
 
 found:
 	/* drm_mm doesn't allow any other other operations while
drivers/gpu/drm/i915/intel_display.c +8 -0
index fe4c5f055973..10d677a1ae2f 100644
@@ -2983,6 +2983,14 @@ static void intel_crtc_wait_for_pending_flips(struct drm_crtc *crtc)
 	mutex_unlock(&dev->struct_mutex);
 }
 
+void intel_wait_for_pending_flips(struct drm_device *dev)
+{
+	struct drm_crtc *crtc;
+
+	list_for_each_entry(crtc, &dev->mode_config.crtc_list, head)
+		intel_crtc_wait_for_pending_flips(crtc);
+}
+
 /* Program iCLKIP clock to the desired frequency */
 static void lpt_program_iclkip(struct drm_crtc *crtc)
 {
drivers/gpu/drm/i915/intel_drv.h +1 -0
index 2b5bcb617908..e8fd10aa07ea 100644
@@ -648,6 +648,7 @@ enum transcoder intel_pipe_to_cpu_transcoder(struct drm_i915_private *dev_priv,
 					     enum pipe pipe);
 void intel_wait_for_vblank(struct drm_device *dev, int pipe);
 void intel_wait_for_pipe_off(struct drm_device *dev, int pipe);
+void intel_wait_for_pending_flips(struct drm_device *dev);
 int ironlake_get_lanes_required(int target_clock, int link_bw, int bpp);
 void vlv_wait_port_ready(struct drm_i915_private *dev_priv,
 			 struct intel_digital_port *dport);
Signature · 1 line
1.8.5.1
#9 lore raw

Re: [PATCH] drm/i915: When evicting something fails, try unpinning old contexts and framebuffers

Quoted 5 lines · Chris Wilson
drivers/gpu/drm/i915/i915_gem_evict.c +0 -1
@@ -115,10 +116,14 @@ none:
                list_del_init(&vma->exec_list);
        }
 
-       /* We expect the caller to unpin, evict all and try again, or give up.

drivers/gpu/drm/i915/i915_gem_evict.c none: · line 126

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem_evict.c +7 -2
-        * So calling i915_gem_evict_vm() is unnecessary.
+       /* Can we unpin some objects such as idle hw contents,
+        * or pending flips?
         */
-       return -ENOSPC;
+       ret = nonblocking ? -ENOSPC : i915_gpu_idle(dev);
+       if (ret)
+               return ret;
+
+       goto search_again;

We need to make sure that we only try to do this once, for otherwise we spin forever. Jon Bloomfield just pointed out that issue in an internal mail with their evict_everything sledgehammer. Also can you please check whether this fixes the test I've just pushed out? Also please check that the test exercise the endless looping.

Finally we should also have a testcase for outstanding flips, and I think it's worth it to split the flip-related changes in gpu_idle out into a separate patch.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#10 lore raw

Re: [PATCH] drm/i915: When evicting something fails, try unpinning old contexts and framebuffers

Quoted 5 lines · Daniel Vetter
Quoted 4 lines · Chris Wilson
drivers/gpu/drm/i915/i915_gem_evict.c +0 -0
@@ -115,10 +116,14 @@ none:
                list_del_init(&vma->exec_list);
        }
 
Quoted 14 lines

drivers/gpu/drm/i915/i915_gem_evict.c none: · line 126

Quoted 11 lines
drivers/gpu/drm/i915/i915_gem_evict.c +7 -3
-       /* We expect the caller to unpin, evict all and try again, or give up.
-        * So calling i915_gem_evict_vm() is unnecessary.
+       /* Can we unpin some objects such as idle hw contents,
+        * or pending flips?
         */
-       return -ENOSPC;
+       ret = nonblocking ? -ENOSPC : i915_gpu_idle(dev);
+       if (ret)
+               return ret;
+
+       goto search_again;

We need to make sure that we only try to do this once, for otherwise we spin forever.

For sure. I had intended to do a if (pinned) do_idle(), but lost that track of thought when adding wait_for_flips. Too overexcited. Back to tracking damage.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre