drm: Fix lock order reversal between mmap_sem and struct_mutex.
Subsystems (3)
PATCH 1/1 drm: Fix lock order reversal between mmap_sem and struct_mutex.
The basic problem wasmmap_sem (do_mmap()) -> struct_mutex (drm_gem_mmap(), i915_gem_fault()) struct_mutex (i915_gem_execbuffer()) -> mmap_sem (copy_from/to_user())
We have plenty of places where we want to hold device state the same (struct_mutex) while we move a non-trivial amount of data (copy_from/to_user()), such as i915_gem_pwrite(). Solve this by moving the easy things that needed struct_mutex with mmap_sem held to using a lock to cover just those data structures (offset hash and offset manager), and do trylock and reschedule in fault.
--- drivers/gpu/drm/drm_gem.c | 8 ++++---- drivers/gpu/drm/i915/i915_gem.c | 15 ++++++++++++++- include/drm/drmP.h | 1 + 3 files changed, 19 insertions(+), 5 deletions(-)
drivers/gpu/drm/drm_gem.c +4 -4
@@ -97,6 +97,7 @@ drm_gem_init(struct drm_device *dev) dev->mm_private = mm; +mutex_init (&mm->offset_mutex);if (drm_ht_create (&mm->offset_hash,19 )) {drm_free (mm,sizeof (struct drm_gem_mm ),DRM_MEM_MM );return -ENOMEM ;
@@ -508,10 +509,9 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)unsigned long prot;int ret =0 ; -mutex_lock (&dev->struct_mutex); - +mutex_lock (&mm->offset_mutex);if (drm_ht_find_item (&mm->offset_hash, vma->vm_pgoff, &hash)) { -mutex_unlock (&dev->struct_mutex); +mutex_unlock (&mm->offset_mutex);return drm_mmap (filp, vma); }
@@ -556,7 +556,7 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)drm_vm_open_locked (vma); out_unlock: -mutex_unlock (&dev->struct_mutex); +mutex_unlock (&mm->offset_mutex);return ret; }
drivers/gpu/drm/i915/i915_gem.c +14 -1
@@ -573,8 +573,16 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf) page_offset = ((unsigned long )vmf->virtual_address - vma->vm_start) >>PAGE_SHIFT ; +/* Get the struct mutex before accessing GEM data structures, but +* keep the struct_mutex -> mmap_sem lock ordering so that we don't +* need to mangle pwrite/pread to allow mmap_sem -> struct_mutex. +*/ +if (!mutex_trylock (&dev->struct_mutex)) { +need_resched (); +return VM_FAULT_NOPAGE ; + } +/* Now bind it into the GTT if needed */ -mutex_lock (&dev->struct_mutex);if (!obj_priv->gtt_space) { ret =i915_gem_object_bind_to_gtt (obj, obj_priv->gtt_alignment);if (ret) {
@@ -646,6 +654,7 @@ i915_gem_create_mmap_offset(struct drm_gem_object *obj) map->size = obj->size; map->handle = obj; +mutex_lock (&mm->offset_mutex);/* Get a DRM GEM mmap offset allocated... */ list->file_offset_node =drm_mm_search_free (&mm->offset_manager, obj->size /PAGE_SIZE ,0 ,0 );
@@ -671,12 +680,14 @@ i915_gem_create_mmap_offset(struct drm_gem_object *obj)/* By now we should be all set, any drm_mmap request on the offset * below will get to our mmap & fault handler */ obj_priv->mmap_offset = ((uint64_t ) list->hash.key) <<PAGE_SHIFT ; +mutex_unlock (&mm->offset_mutex);return 0 ; out_free_mm:drm_mm_put_block (list->file_offset_node); out_free_list: +mutex_unlock (&mm->offset_mutex);drm_free (list->map,sizeof (struct drm_map_list ),DRM_MEM_DRIVER );return ret;
@@ -690,6 +701,7 @@ i915_gem_free_mmap_offset(struct drm_gem_object *obj)struct drm_gem_mm *mm = dev->mm_private;struct drm_map_list *list; +mutex_lock (&mm->offset_mutex); list = &obj->map_list;drm_ht_remove_item (&mm->offset_hash, &list->hash);
@@ -704,6 +716,7 @@ i915_gem_free_mmap_offset(struct drm_gem_object *obj) } obj_priv->mmap_offset =0 ; +mutex_unlock (&mm->offset_mutex); }/**
include/drm/drmP.h +1 -0
@@ -570,6 +570,7 @@ struct drm_ati_pcigart_info {struct drm_gem_mm {struct drm_mm offset_manager;/**< Offset mgmt for buffer objects */ struct drm_open_hash offset_hash;/**< User token hash table for maps */ +struct mutex offset_mutex;/**< covers offset_manager and offset_hash */ };/**
Signature · 1 line
1.5.6.5
Eric Anholt said the following on 2009-2-18 8:59:
Quoted 11 lines
The basic problem was
mmap_sem(do_mmap()) ->struct_mutex(drm_gem_mmap(),i915_gem_fault())struct_mutex(i915_gem_execbuffer()) ->mmap_sem(copy_from/to_user())We have plenty of places where we want to hold device state the same (
struct_mutex) while we move a non-trivial amount of data (copy_from/to_user()), such asi915_gem_pwrite(). Solve this by moving the easy things that neededstruct_mutexwithmmap_semheld to using a lock to cover just those data structures (offset hash and offset manager), and do trylock and reschedule in fault.
Eric, I tested the patch.
But following bug still doesn't disappear.
http://bugzilla.kernel.org/show_bug.cgi?id=12419
PATCH 1/1 drm: Take mmap_sem up front to avoid lock order violations.
From: Kristian Høgsberg <[email protected]>
A number of GEM operations (and legacy drm ones) want to copy data to or from userspace while holding the struct_mutex lock. However, the fault handler calls us with the mmap_sem held and thus enforces the opposite locking order. This patch downs the mmap_sem up front for those operations that access userspace data under the struct_mutex lock to ensure the locking order is consistent.
---
Here's a different and simpler attempt to fix the locking order problem. We can just down_read() the mmap_sem pre-emptively up-front, and the locking order is respected. It's simpler than the mutex_trylock() game, avoids introducing a new mutex.
(forgot to add lkml, resending)
cheers,
Kristian
drivers/gpu/drm/i915/i915_dma.c | 6 +++++- drivers/gpu/drm/i915/i915_gem.c | 20 +++++++++++++------- 2 files changed, 18 insertions(+), 8 deletions(-)
drivers/gpu/drm/i915/i915_dma.c +5 -1
@@ -642,9 +642,11 @@ static int i915_batchbuffer(struct drm_device *dev, void *data, sizeof(struct drm_clip_rect)))return -EFAULT ; +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_dispatch_batchbuffer (dev, batch);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);if (sarea_priv) sarea_priv->last_dispatch =READ_BREADCRUMB (dev_priv);
@@ -674,14 +676,16 @@ static int i915_cmdbuffer(struct drm_device *dev, void *data,return -EFAULT ; } +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_dispatch_cmdbuffer (dev, cmdbuf);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem); +if (ret) {DRM_ERROR ("i915_dispatch_cmdbuffer failed\n" );return ret; } -if (sarea_priv) sarea_priv->last_dispatch =READ_BREADCRUMB (dev_priv);return 0 ;
drivers/gpu/drm/i915/i915_gem.c +13 -7
@@ -171,6 +171,7 @@ i915_gem_pread_ioctl(struct drm_device *dev, void *data,return -EINVAL ; } +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret = i915_gem_object_set_cpu_read_domain_range(obj, args->offset,
@@ -196,6 +197,7 @@ i915_gem_pread_ioctl(struct drm_device *dev, void *data,drm_gem_object_unreference (obj);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);return 0 ; }
@@ -264,7 +266,7 @@ i915_gem_gtt_pwrite(struct drm_device *dev, struct drm_gem_object *obj,if (!access_ok (VERIFY_READ , user_data, remain))return -EFAULT ; - +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_gem_object_pin (obj,0 );if (ret) {
@@ -315,6 +317,7 @@ i915_gem_gtt_pwrite(struct drm_device *dev, struct drm_gem_object *obj, fail:i915_gem_object_unpin (obj);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);return ret; }
@@ -328,6 +331,7 @@ i915_gem_shmem_pwrite(struct drm_device *dev, struct drm_gem_object *obj,loff_t offset;ssize_t written; +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_gem_object_set_to_cpu_domain (obj,1 );
@@ -350,6 +354,7 @@ i915_gem_shmem_pwrite(struct drm_device *dev, struct drm_gem_object *obj, }mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);return 0 ; }
@@ -2473,22 +2478,21 @@ i915_gem_execbuffer(struct drm_device *dev, void *data, goto pre_mutex_err; } +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex);i915_verify_inactive (dev, __FILE__, __LINE__);if (dev_priv->mm.wedged) {DRM_ERROR ("Execbuf while wedged\n" ); -mutex_unlock (&dev->struct_mutex); ret = -EIO ; - goto pre_mutex_err; + goto mutex_err; }if (dev_priv->mm.suspended) {DRM_ERROR ("Execbuf while VT-switched.\n" ); -mutex_unlock (&dev->struct_mutex); ret = -EBUSY ; - goto pre_mutex_err; + goto mutex_err; }/* Look up object handles */
@@ -2641,8 +2645,6 @@ err:for (i =0 ; i < args->buffer_count; i++)drm_gem_object_unreference (object_list[i]); -mutex_unlock (&dev->struct_mutex); -if (!ret) {/* Copy the new buffer offsets back to the user's exec list. */ ret = copy_to_user((struct drm_i915_relocation_entry __user *)
@@ -2655,6 +2657,10 @@ err: args->buffer_count, ret); } +mutex_err: +mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem); + pre_mutex_err:drm_free (object_list,sizeof (*object_list) * args->buffer_count,DRM_MEM_DRIVER );
Signature · 1 line
1.6.1.3
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
Quoted 9 lines · On Wed, 2009-02-18 at 11:38 -0500, [email protected] wrote:
From: Kristian Høgsberg <[email protected]>
A number of GEM operations (and legacy drm ones) want to copy data to or from userspace while holding the
struct_mutexlock. However, the fault handler calls us with themmap_semheld and thus enforces the opposite locking order. This patch downs themmap_semup front for those operations that access userspace data under thestruct_mutexlock to ensure the locking order is consistent.
Quoted 8 lines
Signed-off-by Kristian Høgsberg <[email protected]>---
Here's a different and simpler attempt to fix the locking order problem. We can just
down_read()themmap_sempre-emptively up-front, and the locking order is respected. It's simpler than themutex_trylock()game, avoids introducing a new mutex.
Hell no!
for one, mmap_sem is not a recursive lock, so a pagefault will utterly fail with this in place.
Secondly, holding mmap_sem for no good reason just sucks.
Quoted 137 lines
drivers/gpu/drm/i915/i915_dma.c | 6 +++++- drivers/gpu/drm/i915/i915_gem.c | 20 +++++++++++++------- 2 files changed, 18 insertions(+), 8 deletions(-)drivers/gpu/drm/i915/i915_dma.c +5
-1@@ -642,9 +642,11 @@ static int i915_batchbuffer(struct drm_device *dev, void *data, sizeof(struct drm_clip_rect)))return -EFAULT ; +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_dispatch_batchbuffer (dev, batch);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);if (sarea_priv) sarea_priv->last_dispatch =READ_BREADCRUMB (dev_priv);@@ -674,14 +676,16 @@ static int i915_cmdbuffer(struct drm_device *dev, void *data,return -EFAULT ; } +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_dispatch_cmdbuffer (dev, cmdbuf);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem); +if (ret) {DRM_ERROR ("i915_dispatch_cmdbuffer failed\n" );return ret; } -if (sarea_priv) sarea_priv->last_dispatch =READ_BREADCRUMB (dev_priv);return 0 ;drivers/gpu/drm/i915/i915_gem.c +13
-7@@ -171,6 +171,7 @@ i915_gem_pread_ioctl(struct drm_device *dev, void *data,return -EINVAL ; } +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret = i915_gem_object_set_cpu_read_domain_range(obj, args->offset,@@ -196,6 +197,7 @@ i915_gem_pread_ioctl(struct drm_device *dev, void *data,drm_gem_object_unreference (obj);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);return 0 ; }@@ -264,7 +266,7 @@ i915_gem_gtt_pwrite(struct drm_device *dev, struct drm_gem_object *obj,if (!access_ok (VERIFY_READ , user_data, remain))return -EFAULT ; - +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_gem_object_pin (obj,0 );if (ret) {@@ -315,6 +317,7 @@ i915_gem_gtt_pwrite(struct drm_device *dev, struct drm_gem_object *obj, fail:i915_gem_object_unpin (obj);mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);return ret; }@@ -328,6 +331,7 @@ i915_gem_shmem_pwrite(struct drm_device *dev, struct drm_gem_object *obj,loff_t offset;ssize_t written; +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex); ret =i915_gem_object_set_to_cpu_domain (obj,1 );@@ -350,6 +354,7 @@ i915_gem_shmem_pwrite(struct drm_device *dev, struct drm_gem_object *obj, }mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem);return 0 ; }@@ -2473,22 +2478,21 @@ i915_gem_execbuffer(struct drm_device *dev, void *data, goto pre_mutex_err; } +down_read (¤t->mm->mmap_sem);mutex_lock (&dev->struct_mutex);i915_verify_inactive (dev, __FILE__, __LINE__);if (dev_priv->mm.wedged) {DRM_ERROR ("Execbuf while wedged\n" ); -mutex_unlock (&dev->struct_mutex); ret = -EIO ; - goto pre_mutex_err; + goto mutex_err; }if (dev_priv->mm.suspended) {DRM_ERROR ("Execbuf while VT-switched.\n" ); -mutex_unlock (&dev->struct_mutex); ret = -EBUSY ; - goto pre_mutex_err; + goto mutex_err; }/* Look up object handles */ @@ -2641,8 +2645,6 @@ err:for (i =0 ; i < args->buffer_count; i++)drm_gem_object_unreference (object_list[i]); -mutex_unlock (&dev->struct_mutex); -if (!ret) {/* Copy the new buffer offsets back to the user's exec list. */ ret = copy_to_user((struct drm_i915_relocation_entry __user *)@@ -2655,6 +2657,10 @@ err: args->buffer_count, ret); } +mutex_err: +mutex_unlock (&dev->struct_mutex); +up_read (¤t->mm->mmap_sem); + pre_mutex_err:drm_free (object_list,sizeof (*object_list) * args->buffer_count,DRM_MEM_DRIVER );
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · On Wed, 2009-02-18 at 11:38 -0500, [email protected] wrote:
From: Kristian Høgsberg <[email protected]>
A number of GEM operations (and legacy drm ones) want to copy data to or from userspace while holding the
struct_mutexlock. However, the fault handler calls us with themmap_semheld and thus enforces the opposite locking order. This patch downs themmap_semup front for those operations that access userspace data under thestruct_mutexlock to ensure the locking order is consistent.
Quoted 8 lines
Quoted 8 lines
Signed-off-by Kristian Høgsberg <[email protected]>---
Here's a different and simpler attempt to fix the locking order problem. We can just
down_read()themmap_sempre-emptively up-front, and the locking order is respected. It's simpler than themutex_trylock()game, avoids introducing a new mutex.
OK let me try that again -- my initial response was a tad curt :/
While I appreciate your efforts in fixing GEM (I too have an interest in seeing it done), I cannot support your patch.
Firstly, you're using mmap_sem well outside its problem domain, this is bad form. Furthermore, holding it for extended durations for no good reason affects all other users.
Secondly, mmap_sem is not a recursive lock (very few kernel locks are, and we generally frown upon recursive locking schemes), this means that the fault handler still cannot function properly.
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · On Wed, 2009-02-18 at 11:38 -0500, [email protected] wrote:
From: Kristian Høgsberg <[email protected]>
A number of GEM operations (and legacy drm ones) want to copy data to or from userspace while holding the
struct_mutexlock. However, the fault handler calls us with themmap_semheld and thus enforces the opposite locking order. This patch downs themmap_semup front for those operations that access userspace data under thestruct_mutexlock to ensure the locking order is consistent.Signed-off-by Kristian Høgsberg <[email protected]>---
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Here's a different and simpler attempt to fix the locking order problem. We can just
down_read()themmap_sempre-emptively up-front, and the locking order is respected. It's simpler than themutex_trylock()game, avoids introducing a new mutex.OK let me try that again -- my initial response was a tad curt :/
No that's fair, I was aware that the patch was probably borderline and got the feedback I was looking for ;)
Quoted 6 lines
While I appreciate your efforts in fixing GEM (I too have an interest in seeing it done), I cannot support your patch.
Firstly, you're using
mmap_semwell outside its problem domain, this is bad form. Furthermore, holding it for extended durations for no good reason affects all other users.
Yup, agree.
Quoted 3 lines
Secondly,
mmap_semis not a recursive lock (very few kernel locks are, and we generally frown upon recursive locking schemes), this means that the fault handler still cannot function properly.
I understand, but we take it twice only as a read lock, so that should work, right? We prevent the deadlock the lockdep validator warned about and as far as I can see, the patch doesn't introduce a new one. But other than that I agree with the frowning on recursive locking, it's too often used to paper over badly thought out locking.
cheers,
Kristian
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
Quoted 10 lines · Kristian Høgsberg
Quoted 4 lines
Secondly,
mmap_semis not a recursive lock (very few kernel locks are, and we generally frown upon recursive locking schemes), this means that the fault handler still cannot function properly.I understand, but we take it twice only as a read lock, so that should work, right? We prevent the deadlock the lockdep validator warned about and as far as I can see, the patch doesn't introduce a new one. But other than that I agree with the frowning on recursive locking, it's too often used to paper over badly thought out locking.
It doesn't work. rwsems are fair (otherwise there is terrible starvation properties), so if another process does an interleaved down_write, then the 2nd down_read will block until the down_write is serviced.
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
Quoted 12 lines · Nick Piggin
Quoted 10 lines · Kristian Høgsberg
Quoted 4 lines
Secondly,
mmap_semis not a recursive lock (very few kernel locks are, and we generally frown upon recursive locking schemes), this means that the fault handler still cannot function properly.I understand, but we take it twice only as a read lock, so that should work, right? We prevent the deadlock the lockdep validator warned about and as far as I can see, the patch doesn't introduce a new one. But other than that I agree with the frowning on recursive locking, it's too often used to paper over badly thought out locking.
Quoted 3 lines
It doesn't work. rwsems are fair (otherwise there is terrible starvation properties), so if another process does an interleaved
down_write, then the 2nddown_readwill block until thedown_writeis serviced.
Ooh, right, yes of course, ouch.
thanks,
Kristian
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
In reply to Peter ZijlstraQuoted 10 lines · Peter Zijlstra
Quoted 9 lines · On Wed, 2009-02-18 at 11:38 -0500, [email protected] wrote:
From: Kristian Høgsberg <[email protected]>
A number of GEM operations (and legacy drm ones) want to copy data to or from userspace while holding the
struct_mutexlock. However, the fault handler calls us with themmap_semheld and thus enforces the opposite locking order. This patch downs themmap_semup front for those operations that access userspace data under thestruct_mutexlock to ensure the locking order is consistent.
Quoted 7 lines
Quoted 7 lines
Signed-off-by Kristian Høgsberg <[email protected]>---
Here's a different and simpler attempt to fix the locking order problem. We can just
down_read()themmap_sempre-emptively up-front, and the locking order is respected. It's simpler than themutex_trylock()game, avoids introducing a new mutex.
The "simple" way to fix this is to just allocate a temporary buffer to copy a snapshot of the data going to/from userspace. Then do the real usercopy to/from that buffer outside the locks.
You don't have any performance critical bulk copies (ie. that will blow the L1 cache), do you?
Re: [PATCH] drm: Take mmap_sem up front to avoid lock order violations.
Quoted 19 lines · Nick Piggin
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · On Wed, 2009-02-18 at 11:38 -0500, [email protected] wrote:
From: Kristian Høgsberg <[email protected]>
A number of GEM operations (and legacy drm ones) want to copy data to or from userspace while holding the
struct_mutexlock. However, the fault handler calls us with themmap_semheld and thus enforces the opposite locking order. This patch downs themmap_semup front for those operations that access userspace data under thestruct_mutexlock to ensure the locking order is consistent.Quoted 7 lines
Quoted 7 lines
Signed-off-by Kristian Høgsberg <[email protected]>---
Here's a different and simpler attempt to fix the locking order problem. We can just
down_read()themmap_sempre-emptively up-front, and the locking order is respected. It's simpler than themutex_trylock()game, avoids introducing a new mutex.
Quoted 6 lines
The "simple" way to fix this is to just allocate a temporary buffer to copy a snapshot of the data going to/from userspace. Then do the real usercopy to/from that buffer outside the locks.
You don't have any performance critical bulk copies (ie. that will blow the L1 cache), do you?
16kb is the most common size (batchbuffers). 32k is popular on 915 (vertex), and varying between 0-128k on 965 (vertex). The pwrite path generally represents 10-30% of CPU consumption in CPU-bound apps.
Signature · 2 lines
Eric Anholt [email protected] [email protected]
Quoted 3 lines · Eric Anholt
The basic problem was
mmap_sem(do_mmap()) ->struct_mutex(drm_gem_mmap(),i915_gem_fault())struct_mutex(i915_gem_execbuffer()) ->mmap_sem(copy_from/to_user())
That's not the only problem, there's also:
dup_mmap()
down_write(mmap_sem)
vm_ops->open() -> drm_vm_open()
mutex_lock(struct_mutex);
Quoted 6 lines
We have plenty of places where we want to hold device state the same (
struct_mutex) while we move a non-trivial amount of data (copy_from/to_user()), such asi915_gem_pwrite(). Solve this by moving the easy things that neededstruct_mutexwithmmap_semheld to using a lock to cover just those data structures (offset hash and offset manager), and do trylock and reschedule in fault.
So we establish,
mmap_sem
offset_mutex
i915_gem_mmap_gtt_ioctl()
mutex_lock(struct_mutex)
i915_gem_create_mmap_offset()
mutex_lock(offset_mutex)However we still have
struct_mutex
mmap_sem
in basically every copy_*_user() case
But you cannot seem to switch ->fault() to use offset_mutex, which would work out the inversion because you then have:
struct_mutex
mmap_sem
offset_mutex
So why bother with the offset_mutex? Instead you make your ->fault() fail randomly.
I'm not sure what Wang Chen sees after this patch, but I should not be the exact same splat, still it would not at all surprise me if there's plenty left.
The locking looks very fragile and I don't think this patch is helping anything, sorry.
Quoted 47 lines
Signed-off-by Eric Anholt <[email protected]>--- drivers/gpu/drm/drm_gem.c | 8 ++++---- drivers/gpu/drm/i915/i915_gem.c | 15 ++++++++++++++- include/drm/drmP.h | 1 + 3 files changed, 19 insertions(+), 5 deletions(-)drivers/gpu/drm/drm_gem.c +4
-4@@ -97,6 +97,7 @@ drm_gem_init(struct drm_device *dev) dev->mm_private = mm; +mutex_init (&mm->offset_mutex);if (drm_ht_create (&mm->offset_hash,19 )) {drm_free (mm,sizeof (struct drm_gem_mm ),DRM_MEM_MM );return -ENOMEM ;@@ -508,10 +509,9 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)unsigned long prot;int ret =0 ; -mutex_lock (&dev->struct_mutex); - +mutex_lock (&mm->offset_mutex);if (drm_ht_find_item (&mm->offset_hash, vma->vm_pgoff, &hash)) { -mutex_unlock (&dev->struct_mutex); +mutex_unlock (&mm->offset_mutex);return drm_mmap (filp, vma); }@@ -556,7 +556,7 @@ int drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)drm_vm_open_locked (vma); out_unlock: -mutex_unlock (&dev->struct_mutex); +mutex_unlock (&mm->offset_mutex);return ret; }drivers/gpu/drm/i915/i915_gem.c +0
-0@@ -573,8 +573,16 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf) page_offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >>
drivers/gpu/drm/i915/i915_gem.c int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf) · line 583
Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +8
-0PAGE_SHIFT ; +/* Get the struct mutex before accessing GEM data structures, but +* keep the struct_mutex -> mmap_sem lock ordering so that we don't +* need to mangle pwrite/pread to allow mmap_sem -> struct_mutex. +*/ +if (!mutex_trylock (&dev->struct_mutex)) { +need_resched (); +return VM_FAULT_NOPAGE ; + }
So we just fail the fault if someone happens to hold the struct_mutex? Seems rather fragile, could be another thread doing an ioctl.
Quoted 58 lines
drivers/gpu/drm/i915/i915_gem.c +5
-1/* Now bind it into the GTT if needed */ -mutex_lock (&dev->struct_mutex);if (!obj_priv->gtt_space) { ret =i915_gem_object_bind_to_gtt (obj, obj_priv->gtt_alignment);if (ret) {@@ -646,6 +654,7 @@ i915_gem_create_mmap_offset(struct drm_gem_object *obj) map->size = obj->size; map->handle = obj; +mutex_lock (&mm->offset_mutex);/* Get a DRM GEM mmap offset allocated... */ list->file_offset_node =drm_mm_search_free (&mm->offset_manager, obj->size /PAGE_SIZE ,0 ,0 );@@ -671,12 +680,14 @@ i915_gem_create_mmap_offset(struct drm_gem_object *obj)/* By now we should be all set, any drm_mmap request on the offset * below will get to our mmap & fault handler */ obj_priv->mmap_offset = ((uint64_t ) list->hash.key) <<PAGE_SHIFT ; +mutex_unlock (&mm->offset_mutex);return 0 ; out_free_mm:drm_mm_put_block (list->file_offset_node); out_free_list: +mutex_unlock (&mm->offset_mutex);drm_free (list->map,sizeof (struct drm_map_list ),DRM_MEM_DRIVER );return ret;@@ -690,6 +701,7 @@ i915_gem_free_mmap_offset(struct drm_gem_object *obj)struct drm_gem_mm *mm = dev->mm_private;struct drm_map_list *list; +mutex_lock (&mm->offset_mutex); list = &obj->map_list;drm_ht_remove_item (&mm->offset_hash, &list->hash);@@ -704,6 +716,7 @@ i915_gem_free_mmap_offset(struct drm_gem_object *obj) } obj_priv->mmap_offset =0 ; +mutex_unlock (&mm->offset_mutex); }/** include/drm/drmP.h +1
-0@@ -570,6 +570,7 @@ struct drm_ati_pcigart_info {struct drm_gem_mm {struct drm_mm offset_manager;/**< Offset mgmt for buffer objects */ struct drm_open_hash offset_hash;/**< User token hash table for maps */ +struct mutex offset_mutex;/**< covers offset_manager and offset_hash */ };/**
Quoted 50 lines · Peter Zijlstra
Quoted 4 lines · Eric Anholt
The basic problem was
mmap_sem(do_mmap()) ->struct_mutex(drm_gem_mmap(),i915_gem_fault())struct_mutex(i915_gem_execbuffer()) ->mmap_sem(copy_from/to_user())That's not the only problem, there's also:
dup_mmap()
down_write(mmap_sem)
vm_ops->open()->drm_vm_open()mutex_lock(struct_mutex);Quoted 7 lines
We have plenty of places where we want to hold device state the same (
struct_mutex) while we move a non-trivial amount of data (copy_from/to_user()), such asi915_gem_pwrite(). Solve this by moving the easy things that neededstruct_mutexwithmmap_semheld to using a lock to cover just those data structures (offset hash and offset manager), and do trylock and reschedule in fault.So we establish,
mmap_sem
offset_mutex
i915_gem_mmap_gtt_ioctl()mutex_lock(struct_mutex) i915_gem_create_mmap_offset() mutex_lock(offset_mutex)However we still have
struct_mutex
mmap_semin basically every copy_*_user() case
But you cannot seem to switch ->fault() to use
offset_mutex, which would work out the inversion because you then have:
struct_mutex
mmap_sem
offset_mutexSo why bother with the
offset_mutex? Instead you make your ->fault() fail randomly.
Quoted 8 lines
I'm not sure what Wang Chen sees after this patch, but I should not be the exact same splat, still it would not at all surprise me if there's plenty left.
The locking looks very fragile and I don't think this patch is helping anything, sorry.
It looks to me like the driver preferred locking order is
object_mutex (which happens to be the device global struct_mutex)
mmap_sem
offset_mutex.
So if one could avoid using the struct_mutex for object bookkeeping (A separate lock) then vm_open() and vm_close() would adhere to that locking order as well, simply by not taking the struct_mutex at all.
So only fault() remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note that fault() doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding the struct_mutex time to release it.
/Thomas
Quoted 12 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.
Quoted 7 lines
So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.
It doesn't do the reschedule -- need_resched() will check if the current task was marked to be scheduled away, furthermore yield based locking sucks chunks.
What's so very difficult about pulling the copy_*_user() out from under the locks?
Quoted 21 lines · Peter Zijlstra
Quoted 12 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.Quoted 7 lines
So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.
Quoted 6 lines
It doesn't do the reschedule --
need_resched()will check if the current task was marked to be scheduled away, furthermore yield based locking sucks chunks.What's so very difficult about pulling the copy_*_user() out from under the locks?
That we're expecting the data movement to occur while holding device state in place. For example, we write data through the GTT most of the time so we:
lock struct_mutex
pin the object to the GTT
flushing caches as neededcopy_from_user
unpin object
unlock struct_mutex
If I'm to pull the copy_from_user out, that means I have to:
alloc temporary storage
for each block of temp storage size:
copy_from_user
lock struct_mutex
pin the object to the GTT
flush caches as needed
memcpy
unpin object
unlock struct_mutex
At this point of introducing our third copy of the user's data in our hottest path, we should probably ditch the pwrite path entirely and go to user mapping of the objects for performance. Requiring user mapping (which has significant overhead) cuts the likelihood of moving from user-space object caching to kernel object caching in the future, which has the potential of saving steaming piles of memory.
Signature · 2 lines
Eric Anholt [email protected] [email protected]
Quoted 22 lines · Eric Anholt
Quoted 21 lines · Peter Zijlstra
Quoted 19 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.
Quoted 3 lines
Quoted 3 lines
It doesn't do the reschedule --
need_resched()will check if the current task was marked to be scheduled away, furthermore yield based locking sucks chunks.
Imagine what would happen if your faulting task was the highest RT prio task in the system, you'd end up with a life-lock.
Quoted 7 lines
Quoted 2 lines
What's so very difficult about pulling the copy_*_user() out from under the locks?
That we're expecting the data movement to occur while holding device state in place. For example, we write data through the GTT most of the time so we:
Quoted 6 lines
lock
struct_mutex
pin the object to the GTT
flushing caches as neededcopy_from_user
unpin object
unlockstruct_mutex
So you cannot drop the lock once you've pinned the dst object?
Quoted 12 lines
If I'm to pull the
copy_from_userout, that means I have to:alloc temporary storage
for each block of temp storage size:
copy_from_user
lockstruct_mutex
pin the object to the GTT
flush caches as needed
memcpy
unpin object
unlockstruct_mutex
Quoted 6 lines
At this point of introducing our third copy of the user's data in our hottest path, we should probably ditch the pwrite path entirely and go to user mapping of the objects for performance. Requiring user mapping (which has significant overhead) cuts the likelihood of moving from user-space object caching to kernel object caching in the future, which has the potential of saving steaming piles of memory.
Or you could get_user_pages() to fault the user pages and pin them, and then do pagefault_disable() and use copy_from_user_inatomic or such, and release the pages again.
Quoted 65 lines · Peter Zijlstra
Quoted 22 lines · Eric Anholt
Quoted 21 lines · Peter Zijlstra
Quoted 19 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.Quoted 3 lines
Quoted 3 lines
It doesn't do the reschedule --
need_resched()will check if the current task was marked to be scheduled away, furthermore yield based locking sucks chunks.Imagine what would happen if your faulting task was the highest RT prio task in the system, you'd end up with a life-lock.
Quoted 7 lines
Quoted 2 lines
What's so very difficult about pulling the copy_*_user() out from under the locks?
That we're expecting the data movement to occur while holding device state in place. For example, we write data through the GTT most of the time so we:
Quoted 6 lines
lock
struct_mutex
pin the object to the GTT
flushing caches as neededcopy_from_user
unpin object
unlockstruct_mutexSo you cannot drop the lock once you've pinned the dst object?
Quoted 12 lines
If I'm to pull the
copy_from_userout, that means I have to:alloc temporary storage
for each block of temp storage size:
copy_from_user
lockstruct_mutex
pin the object to the GTT
flush caches as needed
memcpy
unpin object
unlockstruct_mutexQuoted 6 lines
At this point of introducing our third copy of the user's data in our hottest path, we should probably ditch the pwrite path entirely and go to user mapping of the objects for performance. Requiring user mapping (which has significant overhead) cuts the likelihood of moving from user-space object caching to kernel object caching in the future, which has the potential of saving steaming piles of memory.
Quoted 3 lines
Or you could
get_user_pages()to fault the user pages and pin them, and then dopagefault_disable()and usecopy_from_user_inatomicor such, and release the pages again.
I started poking at this today, since the get_user_pages sounded like the solution. Only then I noticed: when we unbind an existing object, we have to unmap_mapping_range to clear the clients' mappings to it in the GTT, which needs to happen while the struct lock (protecting the gtt structure and the gtt to object mappings) is held. So for fault we have mmap_sem held to struct mutex taken for poking at the gtt structure, and for unbind we have struct mutex held to mmap_sem taken to clear mappings.
Signature · 2 lines
Eric Anholt [email protected] [email protected]
Quoted 69 lines · Eric Anholt
Quoted 62 lines · Peter Zijlstra
Quoted 14 lines · Eric Anholt
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.It doesn't do the reschedule --
need_resched()will check if the current task was marked to be scheduled away, furthermore yield based locking sucks chunks.Imagine what would happen if your faulting task was the highest RT prio task in the system, you'd end up with a life-lock.
Quoted 7 lines
Quoted 3 lines
What's so very difficult about pulling the copy_*_user() out from under the locks?
That we're expecting the data movement to occur while holding device state in place. For example, we write data through the GTT most of the time so we:
Quoted 7 lines
lock
struct_mutex
pin the object to the GTT
flushing caches as neededcopy_from_user
unpin object
unlockstruct_mutexSo you cannot drop the lock once you've pinned the dst object?
Quoted 12 lines
If I'm to pull the
copy_from_userout, that means I have to:alloc temporary storage
for each block of temp storage size:
copy_from_user
lockstruct_mutex
pin the object to the GTT
flush caches as needed
memcpy
unpin object
unlockstruct_mutexQuoted 7 lines
At this point of introducing our third copy of the user's data in our hottest path, we should probably ditch the pwrite path entirely and go to user mapping of the objects for performance. Requiring user mapping (which has significant overhead) cuts the likelihood of moving from user-space object caching to kernel object caching in the future, which has the potential of saving steaming piles of memory.
Quoted 4 lines
Or you could
get_user_pages()to fault the user pages and pin them, and then dopagefault_disable()and usecopy_from_user_inatomicor such, and release the pages again.
Quoted 10 lines
I started poking at this today, since the
get_user_pagessounded like the solution. Only then I noticed: when we unbind an existing object, we have tounmap_mapping_rangeto clear the clients' mappings to it in the GTT, which needs to happen while the struct lock (protecting the gtt structure and the gtt to object mappings) is held. So for fault we havemmap_semheld to struct mutex taken for poking at the gtt structure, and for unbind we have struct mutex held tommap_semtaken to clear mappings.
I don't think the mmap_sem is taken during unmap_mapping_rage() ?
/Thomas
Quoted 4 lines · Eric Anholt
Quoted 3 lines
Or you could
get_user_pages()to fault the user pages and pin them, and then dopagefault_disable()and usecopy_from_user_inatomicor such, and release the pages again.
Quoted 8 lines
I started poking at this today, since the
get_user_pagessounded like the solution. Only then I noticed: when we unbind an existing object, we have tounmap_mapping_rangeto clear the clients' mappings to it in the GTT, which needs to happen while the struct lock (protecting the gtt structure and the gtt to object mappings) is held. So for fault we havemmap_semheld to struct mutex taken for poking at the gtt structure, and for unbind we have struct mutex held tommap_semtaken to clear mappings.
So it again comes down to the fact that you cannot pin a gtt object without also holding this struct_mutex?
Normally such things are done by elevating a refcount so that both regular frees and reclaim gets delayed until you're done with the object.
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.
Quoted 8 lines
Quoted 5 lines
So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.It doesn't do the reschedule --
need_resched()will check if the current task was marked to be scheduled away,
Yes. my mistake. set_tsk_need_resched() would be the proper call. If I'm correctly informed, that would kick in the scheduler _after_ the mmap_sem() is released, just before returning to user-space.
Quoted 3 lines
furthermore yield based locking
sucks chunks.
Yes, but AFAICT in this situation it is the only way to reverse locking order in a deadlock safe manner. If there is a lot of contention it will eat cpu. Unfortunately since the struct_mutex is such a wide lock there will probably be contention in some situations.
BTW isn't this quite common in distributed resource management, when you can't ensure that all requestors will request resources in the same order?
Try to grab all resources you need for an operation. If you fail to get one, release the resources you already have, sleep waiting for the failing one to be available and then retry.
In this case we fail be cause we _may_ have a deadlock. Since we cannot release the mmap_sem and wait, we do the second best thing and tell the kernel to reschedule when the mmap_sem is released.
Quoted 4 lines
What's so very difficult about pulling the copy_*_user() out from under the locks?
Given Eric's comment this is a GEM performance-critical path, so even from a CPU-usage perspecive, the trylock solution may be preferred.
From a make-the-code-easy-to-understand perspective, I agree pulling out copy_*_user() is a better solution. It might even be that the trylock solution doesn't kill the warnings from the lock dependency tracker.
/Thomas
Quoted 24 lines · Thomas Hellstrom
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Thomas Hellstrom
It looks to me like the driver preferred locking order is
object_mutex(which happens to be the device globalstruct_mutex)
mmap_sem
offset_mutex.So if one could avoid using the
struct_mutexfor object bookkeeping (A separate lock) thenvm_open()andvm_close()would adhere to that locking order as well, simply by not taking thestruct_mutexat all.So only
fault()remains, in which that locking order is reversed. Personally I think the trylock ->reschedule->retry method with proper commenting is a good solution. It will be the _only_ place where locking order is reversed and it is done in a deadlock-safe manner. Note thatfault()doesn't really fail, but requests a retry from user-space with rescheduling to give the process holding thestruct_mutextime to release it.
Quoted 6 lines
Quoted 2 lines
It doesn't do the reschedule --
need_resched()will check if the current task was marked to be scheduled away,Yes. my mistake.
set_tsk_need_resched()would be the proper call. If I'm correctly informed, that would kick in the scheduler _after_ themmap_sem()is released, just before returning to user-space.
Yes, but it would still life-lock in the RT example given in the other email.
Quoted 7 lines
Quoted 3 lines
furthermore yield based locking
sucks chunks.Yes, but AFAICT in this situation it is the only way to reverse locking order in a deadlock safe manner. If there is a lot of contention it will eat cpu. Unfortunately since the
struct_mutexis such a wide lock there will probably be contention in some situations.
I'd be surprised if this were the only solution. Maybe its the easiest, but not one I'll support.
Quoted 5 lines
BTW isn't this quite common in distributed resource management, when you can't ensure that all requestors will request resources in the same order? Try to grab all resources you need for an operation. If you fail to get one, release the resources you already have, sleep waiting for the failing one to be available and then retry.
Not if you're building deterministic systems. Such constructs are highly non-deterministic.
Furthermore, this isn't really a distributed system is it?