drm/i915: Wait and retry if there is no space in the aperture mappable area
Subsystems (2)
PATCH 1/1 drm/i915: Wait and retry if there is no space in the aperture mappable area
From: "Siluvery, Arun" <[email protected]>
When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in
signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.
--- drivers/gpu/drm/i915/i915_gem.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +4 -0
@@ -3275,6 +3275,10 @@ search_free: obj->cache_level, map_and_fenceable, nonblocking); + +if (ret == -ENOSPC ) + ret =i915_gem_evict_everything (dev); +if (ret ==0 ) goto search_free;
Signature · 1 line
1.8.4
Quoted 6 lines · Arun
From: "Siluvery, Arun" <[email protected]>
When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in
signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.
Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an evict_everything in here breaks reservations, so I think you are just papering over a bug.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Quoted 11 lines · Chris Wilson
Quoted 6 lines · Arun
From: "Siluvery, Arun" <[email protected]>
When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in
signal 7(SIGBUS), code 2 (BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an
evict_everythingin here breaks reservations, so I think you are just papering over a bug.
If we want to fix this for real (i.e. allow userspace to reliably map stuff, mabye even bigger than the aperture) we need to fall back to suballocating tile-row aligned strides of the buffer (maybe pick the tile row multiply in between 1M-2M). Until that's done userspace can't rely on gtt mmaps relibly working for large buffers. The current heuristics we're using is half of the mappable space, but that's probably a bit too optimistic.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 13 lines · Chris Wilson
Quoted 11 lines · Arun
Quoted 6 lines
From: "Siluvery, Arun" <[email protected]>
When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in signal 7(SIGBUS), code 2 (
BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an
evict_everythingin here breaks reservations, so I think you are just papering over a bug.
Quoted 7 lines
If we want to fix this for real (i.e. allow userspace to reliably map stuff, mabye even bigger than the aperture) we need to fall back to suballocating tile-row aligned strides of the buffer (maybe pick the tile row multiply in between 1M-2M). Until that's done userspace can't rely on gtt mmaps relibly working for large buffers. The current heuristics we're using is half of the mappable space, but that's probably a bit too optimistic. -Daniel
Is calling i915_gem_evict_everything() actually dangerous ? Despite its name, it appears to only evict unpinned buffers. Or am I missing something ?
What it does do is update the status of buffers which are no longer in use on the ring by calling i915_gem_retire_requests(). So from what I can tell (from a 10 minute trawl of the code admittedly) all this patch is doing is getting a more up to date view of the GTT demands so that we only fail with ENOSPC if there are no pinned buffers which can now be unpinned.
It doesn't address our underlying issue - userspace should still handle ENOSPC gracefully. However it certainly seems to be improving things considerably, so is beneficial if it really is a safe thing to do.
Jon
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
Quoted 5 lines · Jon
Is calling
i915_gem_evict_everything()actually dangerous ? Despite its name, it appears to only evict unpinned buffers. Or am I missing something ?What it does do is update the status of buffers which are no longer in use on the ring by calling
i915_gem_retire_requests(). So from what I can tell (from a 10 minute trawl of the code admittedly) all this patch is doing is getting a more up to date view of the GTT demands so that we only fail with ENOSPC if there are no pinned buffers which can now be unpinned.It doesn't address our underlying issue - userspace should still handle ENOSPC gracefully. However it certainly seems to be improving things considerably, so is beneficial if it really is a safe thing to do.
We use evict_everything in the execbuf code to defragment the gtt. But that's only beneficial because execbuffer needs to find space for multiple buffers at once. For just binding one single buffer the eviction code should scan all lists. Throwing in an additional evict_everything on top won't help at all.
Of course if you throw in copious amounts of evict_everything once you're in a bad spot the defragmentation this causes might help to get out of that corner. But it really shouldn't help with preventing SIGBUS in the first place. So I really wonder what exactly's going on here. Can you perhaps distill a testcase and share it (preferrably as an i-g-t patch ofc)?
Thanks, Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 21 lines · Jon
Quoted 13 lines · Chris Wilson
Quoted 11 lines · Arun
Quoted 6 lines
From: "Siluvery, Arun" <[email protected]>
When a mapping is requested and if there is no space the mapping fails and the region is not physically backed. This results in signal 7(SIGBUS), code 2 (
BUS_ADRERR) when it is actually accessed. This patch handles this error, continues to wait and retries to find space.Eh, no. The line before will remove everything from the aperture that is unpinned. Throwing an
evict_everythingin here breaks reservations, so I think you are just papering over a bug.Quoted 7 lines
If we want to fix this for real (i.e. allow userspace to reliably map stuff, mabye even bigger than the aperture) we need to fall back to suballocating tile-row aligned strides of the buffer (maybe pick the tile row multiply in between 1M-2M). Until that's done userspace can't rely on gtt mmaps relibly working for large buffers. The current heuristics we're using is half of the mappable space, but that's probably a bit too optimistic. -Daniel
Quoted 1 line
Is calling
i915_gem_evict_everything()actually dangerous ? Despite its name, it appears to only evict unpinned buffers. Or am I missing something ?
It breaks the expectations of the callers, which when they have multiple buffers to reserve at once have a better strategy than can be performed individually on a buffer. (That is when evict-everything becomes useful.)
Quoted 1 line
What it does do is update the status of buffers which are no longer in use on the ring by calling
i915_gem_retire_requests(). So from what I can tell (from a 10 minute trawl of the code admittedly) all this patch is doing is getting a more up to date view of the GTT demands so that we only fail with ENOSPC if there are no pinned buffers which can now be unpinned.
No, evict-something also scans the requests to find the first available slot large enough to fit the buffer. (Except when evict-something is instructed not to block for pending retirement.)
There's a bug here if evict-everything finds something that evict-something doesn't.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Picking up this old thread, hopefully I haven't forgotten anyone from the cc list interested in the previous discussions ...
drivers/gpu/drm/i915/i915_gem.c search_free: · line 3281
Quoted 12 lines · Arun
drivers/gpu/drm/i915/i915_gem.c +4
-0@@ -3275,6 +3275,10 @@ search_free: obj->cache_level, map_and_fenceable, nonblocking); + +if (ret == -ENOSPC ) + ret =i915_gem_evict_everything (dev); +
Despite my earlier claim this is indeed a real bugfix - compared to evict_something this will move away any hw context objects. So this isn't just purely duct-tape.
I think the right approach for this patch would be to create a i915_gem_evict_unpin_special_objects function or similar which atm just does the do_switch to the default context. We can then call this here and in evict_everything.
Can you please look into that and test whether that helps as well as your current hack?
I've noticed this while trying to create a testcase for the recently fought do_switch eviction related oops. I just couldn't and that got me thinking a bit ;-)
Cheers, Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 24 lines · Daniel Vetter
drivers/gpu/drm/i915/i915_gem.c search_free: · line 3281
Quoted 12 lines · Arun
drivers/gpu/drm/i915/i915_gem.c +4
-0@@ -3275,6 +3275,10 @@ search_free: obj->cache_level, map_and_fenceable, nonblocking); + +if (ret == -ENOSPC ) + ret =i915_gem_evict_everything (dev); +Despite my earlier claim this is indeed a real bugfix - compared to
evict_somethingthis will move away any hw context objects. So this isn't just purely duct-tape.I think the right approach for this patch would be to create a
i915_gem_evict_unpin_special_objectsfunction or similar which atm just does thedo_switchto the default context. We can then call this here and inevict_everything.
Quoted 6 lines
Can you please look into that and test whether that helps as well as your current hack?
I've noticed this while trying to create a testcase for the recently fought
do_switcheviction related oops. I just couldn't and that got me thinking a bit ;-)
Fyi I've just pushed out igt/gem_ctx_exec/eviction which exercises our current failure to evict the current contxt. It's going through execbuf though and doesn't test the gtt mmap path. Fixing up that is imo only worth it as part of the large plan I've outlined somewhere else in this thread. Atm the test fails, but it should work with proper context eviction. So this should be useful for a more minimal testcase.
Since the testcase starts with a too-big execbuf that should also exercise the endless loop you've outlined in the internal mail, but I think for safety we want to add one more execbuf call in the eviction test with a 3rd context and just 1 buffer too much in the reloc list. That should perfectly reproduce your failure case.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
PATCH 1/1 drm/i915: When evicting something fails, try unpinning old contexts and framebuffers
In reply to Daniel VetterBefore declaring that we cannot fit an object into the aperture, we should first check if we can clear the aperture of pinned objects that are now idle. These pinned objects include old hardware contexts and old scanout buffers.
--- drivers/gpu/drm/i915/i915_gem.c | 1 + drivers/gpu/drm/i915/i915_gem_evict.c | 11 ++++++++--- drivers/gpu/drm/i915/intel_display.c | 8 ++++++++ drivers/gpu/drm/i915/intel_drv.h | 1 + 4 files changed, 18 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +1 -0
@@ -2921,6 +2921,7 @@ int i915_gpu_idle(struct drm_device *dev)return ret; } +intel_wait_for_pending_flips (dev);return 0 ; }
drivers/gpu/drm/i915/i915_gem_evict.c +8 -3
@@ -80,6 +80,7 @@ i915_gem_evict_something(struct drm_device *dev, struct i915_address_space *vm,*/ INIT_LIST_HEAD (&unwind_list); +search_again:if (mappable) {BUG_ON (!i915_is_ggtt (vm)); drm_mm_init_scan_with_range(&vm->mm, min_size,
@@ -115,10 +116,14 @@ none:list_del_init (&vma->exec_list); } -/* We expect the caller to unpin, evict all and try again, or give up. -* So calling i915_gem_evict_vm() is unnecessary. +/* Can we unpin some objects such as idle hw contents, +* or pending flips? */ -return -ENOSPC ; + ret = nonblocking ? -ENOSPC :i915_gpu_idle (dev); +if (ret) +return ret; + + goto search_again; found:/* drm_mm doesn't allow any other other operations while
drivers/gpu/drm/i915/intel_display.c +8 -0
@@ -2983,6 +2983,14 @@ static void intel_crtc_wait_for_pending_flips(struct drm_crtc *crtc)mutex_unlock (&dev->struct_mutex); } +void intel_wait_for_pending_flips (struct drm_device *dev) +{ +struct drm_crtc *crtc; + +list_for_each_entry (crtc, &dev->mode_config.crtc_list, head) +intel_crtc_wait_for_pending_flips (crtc); +} +/* Program iCLKIP clock to the desired frequency */ static void lpt_program_iclkip (struct drm_crtc *crtc) {
drivers/gpu/drm/i915/intel_drv.h +1 -0
@@ -648,6 +648,7 @@ enum transcoder intel_pipe_to_cpu_transcoder(struct drm_i915_private *dev_priv, enum pipe pipe);void intel_wait_for_vblank (struct drm_device *dev,int pipe);void intel_wait_for_pipe_off (struct drm_device *dev,int pipe); +void intel_wait_for_pending_flips (struct drm_device *dev);int ironlake_get_lanes_required (int target_clock,int link_bw,int bpp);void vlv_wait_port_ready (struct drm_i915_private *dev_priv,struct intel_digital_port *dport);
Signature · 1 line
1.8.5.1
Re: [PATCH] drm/i915: When evicting something fails, try unpinning old contexts and framebuffers
Quoted 5 lines · Chris Wilson
drivers/gpu/drm/i915/i915_gem_evict.c +0
-1@@ -115,10 +116,14 @@ none:list_del_init (&vma->exec_list); } -/* We expect the caller to unpin, evict all and try again, or give up.
drivers/gpu/drm/i915/i915_gem_evict.c none: · line 126
Quoted 10 lines
drivers/gpu/drm/i915/i915_gem_evict.c +7
-2-* So calling i915_gem_evict_vm() is unnecessary. +/* Can we unpin some objects such as idle hw contents, +* or pending flips? */ -return -ENOSPC ; + ret = nonblocking ? -ENOSPC :i915_gpu_idle (dev); +if (ret) +return ret; + + goto search_again;
We need to make sure that we only try to do this once, for otherwise we spin forever. Jon Bloomfield just pointed out that issue in an internal mail with their evict_everything sledgehammer. Also can you please check whether this fixes the test I've just pushed out? Also please check that the test exercise the endless looping.
Finally we should also have a testcase for outstanding flips, and I think it's worth it to split the flip-related changes in gpu_idle out into a separate patch.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH] drm/i915: When evicting something fails, try unpinning old contexts and framebuffers
Quoted 5 lines · Daniel Vetter
Quoted 4 lines · Chris Wilson
drivers/gpu/drm/i915/i915_gem_evict.c +0
-0@@ -115,10 +116,14 @@ none:list_del_init (&vma->exec_list); }
Quoted 14 lines
drivers/gpu/drm/i915/i915_gem_evict.c none: · line 126
Quoted 11 lines
drivers/gpu/drm/i915/i915_gem_evict.c +7
-3-/* We expect the caller to unpin, evict all and try again, or give up. -* So calling i915_gem_evict_vm() is unnecessary. +/* Can we unpin some objects such as idle hw contents, +* or pending flips? */ -return -ENOSPC ; + ret = nonblocking ? -ENOSPC :i915_gpu_idle (dev); +if (ret) +return ret; + + goto search_again;We need to make sure that we only try to do this once, for otherwise we spin forever.
For sure. I had intended to do a if (pinned) do_idle(), but lost that track of thought when adding wait_for_flips. Too overexcited. Back to tracking damage.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre