Consolidate the mmu notifier interval_tree and locking
Subsystems (15)
v2 cover Consolidate the mmu notifier interval_tree and locking
From: Jason Gunthorpe <[email protected]>
8 of the mmu_notifier using drivers (i915_gem, radeon_mn, umem_odp, hfi1, scif_dma, vhost, gntdev, hmm) drivers are using a common pattern where they only use invalidate_range_start/end and immediately check the invalidating range against some driver data structure to tell if the driver is interested. Half of them use an interval_tree, the others are simple linear search lists.
Of the ones I checked they largely seem to have various kinds of races, bugs and poor implementation. This is a result of the complexity in how the notifier interacts with get_user_pages(). It is extremely difficult to use it correctly.
Consolidate all of this code together into the core mmu_notifier and provide a locking scheme similar to hmm_mirror that allows the user to safely use get_user_pages() and reliably know if the page list still matches the mm.
This new arrangment plays nicely with the !blockable mode for OOM. Scanning the interval tree is done such that the intersection test will always succeed, and since there is no invalidate_range_end exposed to drivers the scheme safely allows multiple drivers to be subscribed.
Four places are converted as an example of how the new API is used. Four are left for future patches:
- i915_gem has complex locking around destruction of a registration, needs more study
- hfi1 (2nd user) needs access to the rbtree
- scif_dma has a complicated logic flow
- vhost's mmu notifiers are already being rewritten
This series, and the other code it depends on is available on my github:
https://github.com/jgunthorpe/linux/commits/mmu_notifier
v2 changes:
- Add mmu_range_set_seq() to set the mrn sequence number under the driver lock and make the locking more understandable
- Add some additional comments around locking/READ_ONCe
- Make the WARN_ON flow in mn_itree_invalidate a bit easier to follow
- Fix wrong WARN_ON
Jason Gunthorpe (15):
mm/mmu_notifier: define the header pre-processor parts even if
disabled
mm/mmu_notifier: add an interval tree notifier mm/hmm: allow hmm_range to be used with a mmu_range_notifier or
hmm_mirror
mm/hmm: define the pre-processor related parts of hmm.h even if
disabled
RDMA/odp: Use mmu_range_notifier_insert()
RDMA/hfi1: Use mmu_range_notifier_inset for user_exp_rcv drm/radeon: use mmu_range_notifier_insert
xen/gntdev: Use select for DMA_SHARED_BUFFER xen/gntdev: use mmu_range_notifier_insert
nouveau: use mmu_notifier directly for invalidate_range_start
nouveau: use mmu_range_notifier instead of hmm_mirror drm/amdgpu: Call find_vma under mmap_sem
drm/amdgpu: Use mmu_range_insert instead of hmm_mirror drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror mm/hmm: remove hmm_mirror and related
Documentation/vm/hmm.rst | 105 +--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 9 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 457 +++------------ drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 53 -- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 111 ++-- drivers/gpu/drm/nouveau/nouveau_svm.c | 231 +++++--- drivers/gpu/drm/radeon/radeon.h | 9 +- drivers/gpu/drm/radeon/radeon_mn.c | 219 ++----- drivers/infiniband/core/device.c | 1 - drivers/infiniband/core/umem_odp.c | 288 +-------- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/hfi1/hfi.h | 2 +- drivers/infiniband/hw/hfi1/user_exp_rcv.c | 146 ++--- drivers/infiniband/hw/hfi1/user_exp_rcv.h | 3 +- drivers/infiniband/hw/mlx5/mlx5_ib.h | 7 +- drivers/infiniband/hw/mlx5/mr.c | 3 +- drivers/infiniband/hw/mlx5/odp.c | 50 +- drivers/xen/Kconfig | 3 +- drivers/xen/gntdev-common.h | 8 +- drivers/xen/gntdev.c | 180 ++---- include/linux/hmm.h | 195 +------ include/linux/mmu_notifier.h | 144 ++++- include/rdma/ib_umem_odp.h | 65 +-- include/rdma/ib_verbs.h | 2 - kernel/fork.c | 1 - mm/Kconfig | 2 +- mm/hmm.c | 275 +-------- mm/mmu_notifier.c | 546 +++++++++++++++++- 32 files changed, 1225 insertions(+), 1922 deletions(-)
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 1/15 mm/mmu_notifier: define the header pre-processor parts even if disabled
From: Jason Gunthorpe <[email protected]>
Now that we have KERNEL_HEADER_TEST all headers are generally compile tested, so relying on makefile tricks to avoid compiling code that depends on CONFIG_MMU_NOTIFIER is more annoying.
Instead follow the usual pattern and provide most of the header with only the functions stubbed out when CONFIG_MMU_NOTIFIER is disabled. This ensures code compiles no matter what the config setting is.
While here, struct mmu_notifier_mm is private to mmu_notifier.c, move it.
--- include/linux/mmu_notifier.h | 46 +++++++++++++----------------------- mm/mmu_notifier.c | 13 ++++++++++ 2 files changed, 30 insertions(+), 29 deletions(-)
include/linux/mmu_notifier.h +17 -29
@@ -7,8 +7,9 @@#include <linux/mm_types.h> #include <linux/srcu.h> +struct mmu_notifier_mm ;struct mmu_notifier ; -struct mmu_notifier_ops ; +struct mmu_notifier_range ;/** * enum mmu_notifier_event - reason for the mmu notifier callback
@@ -40,36 +41,8 @@ enum mmu_notifier_event { MMU_NOTIFY_SOFT_DIRTY, }; -#ifdef CONFIG_MMU_NOTIFIER - -#ifdef CONFIG_LOCKDEP -extern struct lockdep_map __mmu_notifier_invalidate_range_start_map; -#endif - -/* -* The mmu notifier_mm structure is allocated and installed in -* mm->mmu_notifier_mm inside the mm_take_all_locks() protected -* critical section and it's released only when mm_count reaches zero -* in mmdrop(). -*/ -struct mmu_notifier_mm { -/* all mmu notifiers registerd in this mm are queued in this list */ -struct hlist_head list; -/* to serialize the list modifications and hlist_unhashed */ -spinlock_t lock; -}; -#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) -struct mmu_notifier_range { -struct vm_area_struct *vma; -struct mm_struct *mm; -unsigned long start; -unsigned long end; -unsigned flags; -enum mmu_notifier_event event; -}; -struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is
@@ -249,6 +222,21 @@ struct mmu_notifier {unsigned int users; }; +#ifdef CONFIG_MMU_NOTIFIER + +#ifdef CONFIG_LOCKDEP +extern struct lockdep_map __mmu_notifier_invalidate_range_start_map; +#endif + +struct mmu_notifier_range { +struct vm_area_struct *vma; +struct mm_struct *mm; +unsigned long start; +unsigned long end; +unsigned flags; +enum mmu_notifier_event event; +}; +static inline int mm_has_notifiers (struct mm_struct *mm) {return unlikely (mm->mmu_notifier_mm);
mm/mmu_notifier.c +13 -0
@@ -27,6 +27,19 @@ struct lockdep_map __mmu_notifier_invalidate_range_start_map = { };#endif +/* +* The mmu notifier_mm structure is allocated and installed in +* mm->mmu_notifier_mm inside the mm_take_all_locks() protected +* critical section and it's released only when mm_count reaches zero +* in mmdrop(). +*/ +struct mmu_notifier_mm { +/* all mmu notifiers registered in this mm are queued in this list */ +struct hlist_head list; +/* to serialize the list modifications and hlist_unhashed */ +spinlock_t lock; +}; +/* * This function can't run concurrently against mmu_notifier_register * because mm->mm_users > 0 during mmu_notifier_register and exit_mmap
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 01/15] mm/mmu_notifier: define the header pre-processor parts even if disabled
Quoted 11 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
Now that we have
KERNEL_HEADER_TESTall headers are generally compile tested, so relying on makefile tricks to avoid compiling code that depends onCONFIG_MMU_NOTIFIERis more annoying.Instead follow the usual pattern and provide most of the header with only the functions stubbed out when
CONFIG_MMU_NOTIFIERis disabled. This ensures code compiles no matter what the config setting is.While here, struct
mmu_notifier_mmis private tommu_notifier.c, move it.
and correct a minor spelling error in a comment. Good. :)
Quoted 8 lines
Reviewed-by Jérôme Glisse <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>--- include/linux/mmu_notifier.h | 46 +++++++++++++----------------------- mm/mmu_notifier.c | 13 ++++++++++ 2 files changed, 30 insertions(+), 29 deletions(-)
Because this is correct as-is, you can add:
...whether or not you take the following recommendation, which is:
you've only done part of the job of making struct mmu_notifier_mm private to mmu_notifier.c. There's more:
* struct mmu_notifier_mm is referred to in two places now: mm_types.h and (still) mmu_notifier.h. Therefore:
a) Move the last two traces of it out of mmu_notifier.h, and
b) Put a forward declaration in mm_types.h, which is where it belongs because that's where it's referred to.
So if you apply this incremental patch on top, I think it's where you want to be:
include/linux/mm_types.h +1 -0
@@ -25,6 +25,7 @@struct address_space ;struct mem_cgroup ; +struct mmu_notifier_mm ;/* * Each physical page in the system has a struct page associated with
include/linux/mmu_notifier.h +2 -9
@@ -8,7 +8,6 @@#include <linux/srcu.h> #include <linux/interval_tree.h> -struct mmu_notifier_mm ;struct mmu_notifier ;struct mmu_notifier_range ;struct mmu_range_notifier ;
@@ -263,10 +262,7 @@ struct mmu_notifier_range {enum mmu_notifier_event event; }; -static inline int mm_has_notifiers (struct mm_struct *mm) -{ -return unlikely (mm->mmu_notifier_mm); -} +int mm_has_notifiers (struct mm_struct *mm);struct mmu_notifier *mmu_notifier_get_locked (const struct mmu_notifier_ops *ops,struct mm_struct *mm);
@@ -477,10 +473,7 @@ static inline void mmu_notifier_invalidate_range(struct mm_struct *mm,__mmu_notifier_invalidate_range (mm, start, end); } -static inline void mmu_notifier_mm_init (struct mm_struct *mm) -{ - mm->mmu_notifier_mm =NULL ; -} +void mmu_notifier_mm_init (struct mm_struct *mm);static inline void mmu_notifier_mm_destroy (struct mm_struct *mm) {
mm/mmu_notifier.c +10 -0
@@ -47,6 +47,16 @@ struct mmu_notifier_mm {struct hlist_head deferred_list; }; +int mm_has_notifiers (struct mm_struct *mm) +{ +return unlikely (mm->mmu_notifier_mm); +} + +void mmu_notifier_mm_init (struct mm_struct *mm) +{ + mm->mmu_notifier_mm =NULL ; +} +/* * This is a collision-retry read-side/write-side 'lock', a lot like a * seqcount, however this allows multiple write-sides to hold it at
thanks,
John Hubbard
NVIDIA
Quoted 89 lines
include/linux/mmu_notifier.h +17
-29@@ -7,8 +7,9 @@#include <linux/mm_types.h> #include <linux/srcu.h> +struct mmu_notifier_mm ;struct mmu_notifier ; -struct mmu_notifier_ops ; +struct mmu_notifier_range ;/** * enum mmu_notifier_event - reason for the mmu notifier callback @@ -40,36 +41,8 @@ enum mmu_notifier_event { MMU_NOTIFY_SOFT_DIRTY, }; -#ifdef CONFIG_MMU_NOTIFIER - -#ifdef CONFIG_LOCKDEP -extern struct lockdep_map __mmu_notifier_invalidate_range_start_map; -#endif - -/* -* The mmu notifier_mm structure is allocated and installed in -* mm->mmu_notifier_mm inside the mm_take_all_locks() protected -* critical section and it's released only when mm_count reaches zero -* in mmdrop(). -*/ -struct mmu_notifier_mm { -/* all mmu notifiers registerd in this mm are queued in this list */ -struct hlist_head list; -/* to serialize the list modifications and hlist_unhashed */ -spinlock_t lock; -}; -#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) -struct mmu_notifier_range { -struct vm_area_struct *vma; -struct mm_struct *mm; -unsigned long start; -unsigned long end; -unsigned flags; -enum mmu_notifier_event event; -}; -struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is @@ -249,6 +222,21 @@ struct mmu_notifier {unsigned int users; }; +#ifdef CONFIG_MMU_NOTIFIER + +#ifdef CONFIG_LOCKDEP +extern struct lockdep_map __mmu_notifier_invalidate_range_start_map; +#endif + +struct mmu_notifier_range { +struct vm_area_struct *vma; +struct mm_struct *mm; +unsigned long start; +unsigned long end; +unsigned flags; +enum mmu_notifier_event event; +}; +static inline int mm_has_notifiers (struct mm_struct *mm) {return unlikely (mm->mmu_notifier_mm);mm/mmu_notifier.c +6
-0@@ -27,6 +27,19 @@ struct lockdep_map __mmu_notifier_invalidate_range_start_map = { };#endif +/* +* The mmu notifier_mm structure is allocated and installed in +* mm->mmu_notifier_mm inside the mm_take_all_locks() protected +* critical section and it's released only when mm_count reaches zero +* in mmdrop(). +*/
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 45
Quoted 10 lines
mm/mmu_notifier.c +7
-0+struct mmu_notifier_mm { +/* all mmu notifiers registered in this mm are queued in this list */ +struct hlist_head list; +/* to serialize the list modifications and hlist_unhashed */ +spinlock_t lock; +}; +/* * This function can't run concurrently against mmu_notifier_register * because mm->mm_users > 0 during mmu_notifier_register and exit_mmap
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 01/15] mm/mmu_notifier: define the header pre-processor parts even if disabled
Quoted 23 lines · John Hubbard
Quoted 11 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
Now that we have
KERNEL_HEADER_TESTall headers are generally compile tested, so relying on makefile tricks to avoid compiling code that depends onCONFIG_MMU_NOTIFIERis more annoying.Instead follow the usual pattern and provide most of the header with only the functions stubbed out when
CONFIG_MMU_NOTIFIERis disabled. This ensures code compiles no matter what the config setting is.While here, struct
mmu_notifier_mmis private tommu_notifier.c, move it.and correct a minor spelling error in a comment. Good. :)
Quoted 7 lines
Reviewed-by Jérôme Glisse <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>include/linux/mmu_notifier.h | 46 +++++++++++++----------------------- mm/mmu_notifier.c | 13 ++++++++++ 2 files changed, 30 insertions(+), 29 deletions(-)
Quoted 4 lines
Because this is correct as-is, you can add:
Reviewed-by John Hubbard <[email protected]>
Thanks
mm/mmu_notifier.c struct mmu_notifier_mm { · line 53
Quoted 11 lines
mm/mmu_notifier.c +5
-0+++ b/mm/mmu_notifier.c@@ -47,6 +47,16 @@ struct mmu_notifier_mm {struct hlist_head deferred_list; }; +int mm_has_notifiers (struct mm_struct *mm) +{ +return unlikely (mm->mmu_notifier_mm); +}
This inline is performance sensitive, it needs to stay inlined..
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 2/15 mm/mmu_notifier: add an interval tree notifier
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
Of the 13 users of mmu_notifiers, 8 of them use only invalidate_range_start/end() and immediately intersect the mmu_notifier_range with some kind of internal list of VAs. 4 use an interval tree (i915_gem, radeon_mn, umem_odp, hfi1). 4 use a linked list of some kind (scif_dma, vhost, gntdev, hmm)
And the remaining 5 either don't use invalidate_range_start() or do some special thing with it.
It turns out that building a correct scheme with an interval tree is pretty complicated, particularly if the use case is synchronizing against another thread doing get_user_pages(). Many of these implementations have various subtle and difficult to fix races.
This approach puts the interval tree as common code at the top of the mmu notifier call tree and implements a shareable locking scheme.
It includes:
- An interval tree tracking VA ranges, with per-range callbacks
- A read/write locking scheme for the interval tree that avoids sleeping in the notifier path (for OOM killer)
- A sequence counter based collision-retry locking scheme to tell device page fault that a VA range is being concurrently invalidated.
This is based on various ideas:
- hmm accumulates invalidated VA ranges and releases them when all invalidates are done, via active_invalidate_ranges count. This approach avoids having to intersect the interval tree twice (as umem_odp does) at the potential cost of a longer device page fault.
- kvm/umem_odp use a sequence counter to drive the collision retry, via invalidate_seq
- a deferred work todo list on unlock scheme like RTNL, via deferred_list. This makes adding/removing interval tree members more deterministic
- seqlock, except this version makes the seqlock idea multi-holder on the write side by protecting it with active_invalidate_ranges and a spinlock
To minimize MM overhead when only the interval tree is being used, the entire SRCU and hlist overheads are dropped using some simple branches. Similarly the interval tree overhead is dropped when in hlist mode.
The overhead from the mandatory spinlock is broadly the same as most of existing users which already had a lock (or two) of some sort on the invalidation path.
--- include/linux/mmu_notifier.h | 98 +++++++ mm/Kconfig | 1 + mm/mmu_notifier.c | 533 +++++++++++++++++++++++++++++++++-- 3 files changed, 607 insertions(+), 25 deletions(-)
include/linux/mmu_notifier.h +98 -0
@@ -6,10 +6,12 @@#include <linux/spinlock.h> #include <linux/mm_types.h> #include <linux/srcu.h> +#include <linux/interval_tree.h> struct mmu_notifier_mm ;struct mmu_notifier ;struct mmu_notifier_range ; +struct mmu_range_notifier ;/** * enum mmu_notifier_event - reason for the mmu notifier callback
@@ -32,6 +34,9 @@ struct mmu_notifier_range;* access flags). User should soft dirty the page in the end callback to make * sure that anyone relying on soft dirtyness catch pages that might be written * through non CPU mappings. +* +* @MMU_NOTIFY_RELEASE: used during mmu_range_notifier invalidate to signal that +* the mm refcount is zero and the range is no longer accessible. */ enum mmu_notifier_event {MMU_NOTIFY_UNMAP =0 ,
@@ -39,6 +44,7 @@ enum mmu_notifier_event {MMU_NOTIFY_PROTECTION_VMA ,MMU_NOTIFY_PROTECTION_PAGE ,MMU_NOTIFY_SOFT_DIRTY , + MMU_NOTIFY_RELEASE, };#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)
@@ -222,6 +228,26 @@ struct mmu_notifier {unsigned int users; }; +/** +* struct mmu_range_notifier_ops +* @invalidate: Upon return the caller must stop using any SPTEs within this +* range, this function can sleep. Return false if blocking was +* required but range is non-blocking +*/ +struct mmu_range_notifier_ops { +bool (*invalidate)(struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range, +unsigned long cur_seq); +}; + +struct mmu_range_notifier { +struct interval_tree_node interval_tree; +const struct mmu_range_notifier_ops *ops; +struct hlist_node deferred_item; +unsigned long invalidate_seq; +struct mm_struct *mm; +}; +#ifdef CONFIG_MMU_NOTIFIER #ifdef CONFIG_LOCKDEP
@@ -263,6 +289,78 @@ extern int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm);extern void mmu_notifier_unregister (struct mmu_notifier *mn,struct mm_struct *mm); + +unsigned long mmu_range_read_begin (struct mmu_range_notifier *mrn); +int mmu_range_notifier_insert (struct mmu_range_notifier *mrn, +unsigned long start,unsigned long length, +struct mm_struct *mm); +int mmu_range_notifier_insert_locked (struct mmu_range_notifier *mrn, +unsigned long start,unsigned long length, +struct mm_struct *mm); +void mmu_range_notifier_remove (struct mmu_range_notifier *mrn); + +/** +* mmu_range_set_seq - Save the invalidation sequence +* @mrn - The mrn passed to invalidate +* @cur_seq - The cur_seq passed to invalidate +* +* This must be called unconditionally from the invalidate callback of a +* struct mmu_range_notifier_ops under the same lock that is used to call +* mmu_range_read_retry(). It updates the sequence number for later use by +* mmu_range_read_retry(). +* +* If the user does not call mmu_range_read_begin() or mmu_range_read_retry() +* then this call is not required. +*/ +static inline void mmu_range_set_seq (struct mmu_range_notifier *mrn, +unsigned long cur_seq) +{ +WRITE_ONCE (mrn->invalidate_seq, cur_seq); +} + +/** +* mmu_range_read_retry - End a read side critical section against a VA range +* mrn: The range under lock +* seq: The return of the paired mmu_range_read_begin() +* +* This MUST be called under a user provided lock that is also held +* unconditionally by op->invalidate() when it calls mmu_range_set_seq(). +* +* Each call should be paired with a single mmu_range_read_begin() and +* should be used to conclude the read side. +* +* Returns true if an invalidation collided with this critical section, and +* the caller should retry. +*/ +static inline bool mmu_range_read_retry (struct mmu_range_notifier *mrn, +unsigned long seq) +{ +return mrn->invalidate_seq != seq; +} + +/** +* mmu_range_check_retry - Test if a collision has occurred +* mrn: The range under lock +* seq: The return of the matching mmu_range_read_begin() +* +* This can be used in the critical section between mmu_range_read_begin() and +* mmu_range_read_retry(). A return of true indicates an invalidation has +* collided with this lock and a future mmu_range_read_retry() will return +* true. +* +* False is not reliable and only suggests a collision has not happened. It +* can be called many times and does not have to hold the user provided lock. +* +* This call can be used as part of loops and other expensive operations to +* expedite a retry. +*/ +static inline bool mmu_range_check_retry (struct mmu_range_notifier *mrn, +unsigned long seq) +{ +/* Pairs with the WRITE_ONCE in mmu_range_set_seq() */ +return READ_ONCE (mrn->invalidate_seq) != seq; +} +extern void __mmu_notifier_mm_destroy (struct mm_struct *mm);extern void __mmu_notifier_release (struct mm_struct *mm); extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm,
mm/Kconfig +1 -0
@@ -284,6 +284,7 @@ config VIRT_TO_BUS config MMU_NOTIFIER bool select SRCU + select INTERVAL_TREE config KSM bool "Enable KSM for page merging"
mm/mmu_notifier.c +508 -25
655 lines not on this page · raw message
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Kuehling, Felix
maintainerRe: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
I haven't had enough time to fully understand the deferred logic in this change. I spotted one problem, see comments inline.
Quoted 60 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
Of the 13 users of
mmu_notifiers, 8 of them use only invalidate_range_start/end() and immediately intersect themmu_notifier_rangewith some kind of internal list of VAs. 4 use an interval tree (i915_gem,radeon_mn,umem_odp, hfi1). 4 use a linked list of some kind (scif_dma, vhost, gntdev, hmm)And the remaining 5 either don't use
invalidate_range_start()or do some special thing with it.It turns out that building a correct scheme with an interval tree is pretty complicated, particularly if the use case is synchronizing against another thread doing
get_user_pages(). Many of these implementations have various subtle and difficult to fix races.This approach puts the interval tree as common code at the top of the mmu notifier call tree and implements a shareable locking scheme.
It includes:
- An interval tree tracking VA ranges, with per-range callbacks
- A read/write locking scheme for the interval tree that avoids sleeping in the notifier path (for OOM killer)
- A sequence counter based collision-retry locking scheme to tell device page fault that a VA range is being concurrently invalidated.This is based on various ideas:
- hmm accumulates invalidated VA ranges and releases them when all invalidates are done, viaactive_invalidate_rangescount. This approach avoids having to intersect the interval tree twice (asumem_odpdoes) at the potential cost of a longer device page fault.- kvm/umem_odp use a sequence counter to drive the collision retry, via
invalidate_seq- a deferred work todo list on unlock scheme like RTNL, via
deferred_list. This makes adding/removing interval tree members more deterministic- seqlock, except this version makes the seqlock idea multi-holder on the write side by protecting it with
active_invalidate_rangesand a spinlockTo minimize MM overhead when only the interval tree is being used, the entire SRCU and hlist overheads are dropped using some simple branches. Similarly the interval tree overhead is dropped when in hlist mode.
The overhead from the mandatory spinlock is broadly the same as most of existing users which already had a lock (or two) of some sort on the invalidation path.
Cc Andrea Arcangeli <[email protected]>Cc Michal Hocko <[email protected]>Acked-by Christian König <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>--- include/linux/mmu_notifier.h | 98 +++++++ mm/Kconfig | 1 + mm/mmu_notifier.c | 533 +++++++++++++++++++++++++++++++++-- 3 files changed, 607 insertions(+), 25 deletions(-)
[snip]
Quoted 4 lines
mm/mmu_notifier.c +0
-0
[snip]
Quoted 7 lines
* because
mm->mm_users> 0 duringmmu_notifier_registerandexit_mmapmm/mmu_notifier.c +1
-1@@ -52,17 +286,24 @@ struct mmu_notifier_mm {* can't go away from under us as exit_mmap holds an mm_count pin * itself. */ -void __mmu_notifier_release (struct mm_struct *mm) +static void mn_hlist_release(struct mmu_notifier_mm *mmn_mm,
mm/mmu_notifier.c struct mmu_notifier_mm { · line 299
Quoted 10 lines
mm/mmu_notifier.c +6
-0+ struct mm_struct *mm) {struct mmu_notifier *mn;int id; +if (mmn_mm->has_interval) +mn_itree_release (mmn_mm, mm); + +if (hlist_empty (&mmn_mm->list)) +return ;
This seems to duplicate the conditions in __mmu_notifier_release. See my comments below, I think one of them is wrong. I suspect this one, because __mmu_notifier_release follows the same pattern as the other notifiers.
Quoted 34 lines
mm/mmu_notifier.c +6
-5+/* * SRCU here will block mmu_notifier_unregister until * ->release returns. */ id =srcu_read_lock (&srcu); -hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) +hlist_for_each_entry_rcu (mn, &mmn_mm->list, hlist)/* * If ->release runs before mmu_notifier_unregister it must be * handled, as it's the only way for the driver to flush all @@ -72,9 +313,9 @@ void __mmu_notifier_release(struct mm_struct *mm)if (mn->ops->release) mn->ops->release (mn, mm); -spin_lock (&mm->mmu_notifier_mm->lock); -while (unlikely (!hlist_empty (&mm->mmu_notifier_mm->list))) { - mn =hlist_entry (mm->mmu_notifier_mm->list.first, +spin_lock (&mmn_mm->lock); +while (unlikely (!hlist_empty (&mmn_mm->list))) { + mn =hlist_entry (mmn_mm->list.first, struct mmu_notifier, hlist);/* @@ -85,7 +326,7 @@ void __mmu_notifier_release(struct mm_struct *mm)*/ hlist_del_init_rcu (&mn->hlist); } -spin_unlock (&mm->mmu_notifier_mm->lock); +spin_unlock (&mmn_mm->lock);srcu_read_unlock (&srcu, id);/*
mm/mmu_notifier.c void __mmu_notifier_release(struct mm_struct *mm) · line 349
Quoted 9 lines
mm/mmu_notifier.c +6
-0@@ -100,6 +341,17 @@ void __mmu_notifier_release(struct mm_struct *mm)synchronize_srcu (&srcu); } +void __mmu_notifier_release (struct mm_struct *mm) +{ +struct mmu_notifier_mm *mmn_mm = mm->mmu_notifier_mm; + +if (mmn_mm->has_interval) +mn_itree_release (mmn_mm, mm);
If mmn_mm->list is not empty, this will be done twice because mn_hlist_release duplicates this.
mm/mmu_notifier.c void __mmu_notifier_release(struct mm_struct *mm) · line 352
Quoted 3 lines
mm/mmu_notifier.c +3
-0+ +if (!hlist_empty (&mmn_mm->list)) +mn_hlist_release (mmn_mm, mm);
mn_hlist_release checks the same condition itself.
mm/mmu_notifier.c void __mmu_notifier_release(struct mm_struct *mm) · line 357
Quoted 5 lines
mm/mmu_notifier.c +2
-0+} +/* * If no young bitflag is supported by the hardware, ->clear_flush_young can * unmap the address and return 1 or 0 depending if the mapping previously
[snip]
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 12 lines · Felix
Quoted 12 lines
* because
mm->mm_users> 0 duringmmu_notifier_registerandexit_mmapmm/mmu_notifier.c +2
-1@@ -52,17 +286,24 @@ struct mmu_notifier_mm {* can't go away from under us as exit_mmap holds an mm_count pin * itself. */ -void __mmu_notifier_release (struct mm_struct *mm) +static void mn_hlist_release (struct mmu_notifier_mm *mmn_mm, +struct mm_struct *mm) {struct mmu_notifier *mn;int id;
Quoted 10 lines
mm/mmu_notifier.c struct mmu_notifier_mm { · line 299
Quoted 5 lines
mm/mmu_notifier.c +5
-0+if (mmn_mm->has_interval) +mn_itree_release (mmn_mm, mm); + +if (hlist_empty (&mmn_mm->list)) +return ;This seems to duplicate the conditions in
__mmu_notifier_release. See my comments below, I think one of them is wrong. I suspect this one, because__mmu_notifier_releasefollows the same pattern as the other notifiers.
Yep, this is a rebasing error from a earlier version, the above two lines should be deleted.
I think it is harmless so it should not impact any testing.
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
In reply to Jason GunthorpeOn 10/28/19 1:10 PM, Jason Gunthorpe wrote:
...
Quoted 10 lines
include/linux/mmu_notifier.h | 98 +++++++ mm/Kconfig | 1 + mm/mmu_notifier.c | 533 +++++++++++++++++++++++++++++++++-- 3 files changed, 607 insertions(+), 25 deletions(-)include/linux/mmu_notifier.h +0
-0
include/linux/mmu_notifier.h line 14
Quoted 9 lines
include/linux/mmu_notifier.h +2
-0@@ -6,10 +6,12 @@#include <linux/spinlock.h> #include <linux/mm_types.h> #include <linux/srcu.h> +#include <linux/interval_tree.h> struct mmu_notifier_mm ;struct mmu_notifier ;struct mmu_notifier_range ; +struct mmu_range_notifier ;
Hi Jason,
Nice design, I love the seq foundation! So far, I'm not able to spot anything actually wrong with the implementation, sorry about that.
Generally my reaction is: given that the design is complex, try to mitigate that with documentation and naming. So the comments are in these areas:
1. There is a rather severe naming overlap (not technically a naming conflict, but still) with existing mmn work, which already has, for example:
struct mmu_notifier_range
...and you're adding:
struct mmu_range_notifier
...so I'll try to help sort that out.
2. I'm also seeing a couple of things that are really hard for the reader verify are correct (abuse and battery of the low bit in .invalidate_seq, for example, haha), so I have some recommendations there.
3. Documentation improvements, which easy to apply, with perhaps one exception. (Here, because this a complicated area, documentation does make a difference, so it's worth a little extra fuss.)
4. Other nits that don't matter too much, but just help polish things up as usual.
Quoted 14 lines
include/linux/mmu_notifier.h +3
-0/** * enum mmu_notifier_event - reason for the mmu notifier callback @@ -32,6 +34,9 @@ struct mmu_notifier_range;* access flags). User should soft dirty the page in the end callback to make * sure that anyone relying on soft dirtyness catch pages that might be written * through non CPU mappings. +* +* @MMU_NOTIFY_RELEASE: used during mmu_range_notifier invalidate to signal that +* the mm refcount is zero and the range is no longer accessible. */ enum mmu_notifier_event {MMU_NOTIFY_UNMAP =0 ,
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 48
Quoted 5 lines
include/linux/mmu_notifier.h +1
-0@@ -39,6 +44,7 @@ enum mmu_notifier_event {MMU_NOTIFY_PROTECTION_VMA ,MMU_NOTIFY_PROTECTION_PAGE ,MMU_NOTIFY_SOFT_DIRTY , + MMU_NOTIFY_RELEASE, };
OK, let the naming debates begin! ha. Anyway, after careful study of the overall patch, and some browsing of the larger patchset, it's clear that:
* The new "MMU range notifier" that you've created is, approximately, a new object. It uses classic mmu notifiers inside, as an implementation detail, and it does *similar* things (notifications) as mmn's. But it's certainly not the same as mmn's, as shown later when you say the need to an entirely new ops struct, and data struct too.
Therefore, you need a separate events enum as well. This is important. MMN's won't be issuing MMN_NOTIFY_RELEASE events, nor will MNR's be issuing the first four prexisting MMU_NOTIFY_* items. So it would be a design mistake to glom them together, unless you ultimately decided to merge these MMN and MNR objects (which I don't really see any intention of, and that's fine).
So this should read:
enum mmu_range_notifier_event {
MMU_NOTIFY_RELEASE,
};
...assuming that we stay with "mmu_range_notifier" as a core name for this whole thing.
Also, it is best moved down to be next to the new MNR structs, so that all the MNR stuff is in one group.
Extra credit: IMHO, this clearly deserves to all be in a new mmu_range_notifier.h header file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.
include/linux/mmu_notifier.h struct mmu_notifier { · line 230
Quoted 6 lines
include/linux/mmu_notifier.h +0
-0#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)@@ -222,6 +228,26 @@ struct mmu_notifier {unsigned int users; };
That should also be moved down, next to the new structs.
A little bit above these next items, just above "struct mmu_notifier" (not shown here, it's outside the diff area), there is some documentation about classic MMNs. It would be nice if it were clearer that that documentation is not relevant to MNRs. Actually, this is another reason that a separate header file would be nice.
include/linux/mmu_notifier.h struct mmu_notifier { · line 236
Quoted 6 lines
include/linux/mmu_notifier.h +6
-0+/** +* struct mmu_range_notifier_ops +* @invalidate: Upon return the caller must stop using any SPTEs within this +* range, this function can sleep. Return false if blocking was +* required but range is non-blocking +*/
How about this (I'm not sure I fully understand the return value, though):
/**
* struct mmu_range_notifier_ops
* @invalidate: Upon return the caller must stop using any SPTEs within this
* range.
*
* This function is permitted to sleep.
*
* @Return: false if blocking was required, but @range is
* non-blocking.
*
*/
Quoted 4 lines
include/linux/mmu_notifier.h +4
-0+struct mmu_range_notifier_ops { +bool (*invalidate)(struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range, +unsigned long cur_seq);
include/linux/mmu_notifier.h struct mmu_notifier { · line 250
Quoted 10 lines
include/linux/mmu_notifier.h +10
-0+}; + +struct mmu_range_notifier { +struct interval_tree_node interval_tree; +const struct mmu_range_notifier_ops *ops; +struct hlist_node deferred_item; +unsigned long invalidate_seq; +struct mm_struct *mm; +}; +
Again, now we have the new struct mmu_range_notifier, and the old struct mmu_notifier_range, and it's not good.
Ideas:
a) Live with it.
b) (Discarded, too many callers): rename old one. Nope.
c) Rename new one. Ideas:
struct mmu_interval_notifier
struct mmu_range_intersection
...other ideas?Quoted 9 lines
include/linux/mmu_notifier.h +2
-0#ifdef CONFIG_MMU_NOTIFIER #ifdef CONFIG_LOCKDEP @@ -263,6 +289,78 @@ extern int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm);extern void mmu_notifier_unregister (struct mmu_notifier *mn,struct mm_struct *mm); + +unsigned long mmu_range_read_begin (struct mmu_range_notifier *mrn);
include/linux/mmu_notifier.h extern int __mmu_notifier_register(struct mmu_notifier *mn, · line 303
Quoted 10 lines
include/linux/mmu_notifier.h +10
-0+int mmu_range_notifier_insert (struct mmu_range_notifier *mrn, +unsigned long start,unsigned long length, +struct mm_struct *mm); +int mmu_range_notifier_insert_locked (struct mmu_range_notifier *mrn, +unsigned long start,unsigned long length, +struct mm_struct *mm); +void mmu_range_notifier_remove (struct mmu_range_notifier *mrn); + +/** +* mmu_range_set_seq - Save the invalidation sequence
How about:
* mmu_range_set_seq - Set the .invalidate_seq to a new value.
include/linux/mmu_notifier.h extern int __mmu_notifier_register(struct mmu_notifier *mn, · line 312
Quoted 9 lines
include/linux/mmu_notifier.h +9
-0+* @mrn - The mrn passed to invalidate +* @cur_seq - The cur_seq passed to invalidate +* +* This must be called unconditionally from the invalidate callback of a +* struct mmu_range_notifier_ops under the same lock that is used to call +* mmu_range_read_retry(). It updates the sequence number for later use by +* mmu_range_read_retry(). +* +* If the user does not call mmu_range_read_begin() or mmu_range_read_retry()
nit: "caller" is better than "user", when referring to...well, callers. "user" most often refers to user space, whereas a call stack and function calling is clearly what you're referring to here (and in other places, especially "user lock").
Quoted 29 lines
include/linux/mmu_notifier.h +29
-0+* then this call is not required. +*/ +static inline void mmu_range_set_seq (struct mmu_range_notifier *mrn, +unsigned long cur_seq) +{ +WRITE_ONCE (mrn->invalidate_seq, cur_seq); +} + +/** +* mmu_range_read_retry - End a read side critical section against a VA range +* mrn: The range under lock +* seq: The return of the paired mmu_range_read_begin() +* +* This MUST be called under a user provided lock that is also held +* unconditionally by op->invalidate() when it calls mmu_range_set_seq(). +* +* Each call should be paired with a single mmu_range_read_begin() and +* should be used to conclude the read side. +* +* Returns true if an invalidation collided with this critical section, and +* the caller should retry. +*/ +static inline bool mmu_range_read_retry (struct mmu_range_notifier *mrn, +unsigned long seq) +{ +return mrn->invalidate_seq != seq; +} + +/**
include/linux/mmu_notifier.h extern int __mmu_notifier_register(struct mmu_notifier *mn, · line 351
Quoted 10 lines
include/linux/mmu_notifier.h +10
-0+* mmu_range_check_retry - Test if a collision has occurred +* mrn: The range under lock +* seq: The return of the matching mmu_range_read_begin() +* +* This can be used in the critical section between mmu_range_read_begin() and +* mmu_range_read_retry(). A return of true indicates an invalidation has +* collided with this lock and a future mmu_range_read_retry() will return +* true. +* +* False is not reliable and only suggests a collision has not happened. It
let's say "suggests that a collision *may* not have occurred."
...
Quoted 21 lines
mm/mmu_notifier.c +4
-0@@ -12,6 +12,7 @@#include <linux/export.h> #include <linux/mm.h> #include <linux/err.h> +#include <linux/interval_tree.h> #include <linux/srcu.h> #include <linux/rcupdate.h> #include <linux/sched.h> @@ -36,10 +37,243 @@ struct lockdep_map __mmu_notifier_invalidate_range_start_map = {struct mmu_notifier_mm {/* all mmu notifiers registered in this mm are queued in this list */ struct hlist_head list; +bool has_interval;/* to serialize the list modifications and hlist_unhashed */ spinlock_t lock; +unsigned long invalidate_seq; +unsigned long active_invalidate_ranges;
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 54
Quoted 10 lines
mm/mmu_notifier.c +8
-0+struct rb_root_cached itree; +wait_queue_head_t wq; +struct hlist_head deferred_list; }; +/* +* This is a collision-retry read-side/write-side 'lock', a lot like a +* seqcount, however this allows multiple write-sides to hold it at +* once. Conceptually the write side is protecting the values of the PTEs in +* this mm, such that PTES cannot be read into SPTEs while any writer exists.
Just to be kind, can we say "SPTEs (shadow PTEs)", just this once? :)
Quoted 8 lines
mm/mmu_notifier.c +8
-0+* +* Note that the core mm creates nested invalidate_range_start()/end() regions +* within the same thread, and runs invalidate_range_start()/end() in parallel +* on multiple CPUs. This is designed to not reduce concurrency or block +* progress on the mm side. +* +* As a secondary function, holding the full write side also serves to prevent +* writers for the itree, this is an optimization to avoid extra locking
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 72
Quoted 10 lines
mm/mmu_notifier.c +10
-0+* during invalidate_range_start/end notifiers. +* +* The write side has two states, fully excluded: +* - mm->active_invalidate_ranges != 0 +* - mnn->invalidate_seq & 1 == True +* - some range on the mm_struct is being invalidated +* - the itree is not allowed to change +* +* And partially excluded: +* - mm->active_invalidate_ranges != 0
I assume this implies mnn->invalidate_seq & 1 == False in this case? If so, let's say so. I'm probably getting that wrong, too.
Quoted 16 lines
mm/mmu_notifier.c +16
-0+* - some range on the mm_struct is being invalidated +* - the itree is allowed to change +* +* The later state avoids some expensive work on inv_end in the common case of +* no mrn monitoring the VA. +*/ +static bool mn_itree_is_invalidating (struct mmu_notifier_mm *mmn_mm) +{ +lockdep_assert_held (&mmn_mm->lock); +return mmn_mm->invalidate_seq &1 ; +} + +static struct mmu_range_notifier * +mn_itree_inv_start_range (struct mmu_notifier_mm *mmn_mm, +const struct mmu_notifier_range *range, +unsigned long *seq)
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 98
Quoted 10 lines
mm/mmu_notifier.c +10
-0+{ +struct interval_tree_node *node; +struct mmu_range_notifier *res =NULL ; + +spin_lock (&mmn_mm->lock); + mmn_mm->active_invalidate_ranges++; + node =interval_tree_iter_first (&mmn_mm->itree, range->start, + range->end -1 ); +if (node) { + mmn_mm->invalidate_seq |=1 ;
OK, this either needs more documentation and assertions, or a different approach. Because I see addition, subtraction, AND, OR and booleans all being applied to this field, and it's darn near hopeless to figure out whether or not it really is even or odd at the right times.
Different approach: why not just add a mmn_mm->is_invalidating member variable? It's not like you're short of space in that struct.
Quoted 26 lines
mm/mmu_notifier.c +26
-0+ res = container_of(node, struct mmu_range_notifier, + interval_tree); + } + + *seq = mmn_mm->invalidate_seq; +spin_unlock (&mmn_mm->lock); +return res; +} + +static struct mmu_range_notifier * +mn_itree_inv_next (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) +{ +struct interval_tree_node *node; + + node =interval_tree_iter_next (&mrn->interval_tree, range->start, + range->end -1 ); +if (!node) +return NULL ; + return container_of(node, struct mmu_range_notifier, interval_tree); +} + +static void mn_itree_inv_end (struct mmu_notifier_mm *mmn_mm) +{ +struct mmu_range_notifier *mrn; +struct hlist_node *next;
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 134
Quoted 10 lines
mm/mmu_notifier.c +10
-0+bool need_wake = false; + +spin_lock (&mmn_mm->lock); +if (--mmn_mm->active_invalidate_ranges || + !mn_itree_is_invalidating (mmn_mm)) { +spin_unlock (&mmn_mm->lock); +return ; + } + + mmn_mm->invalidate_seq++;
Is this the right place for an assertion that this is now an even value?
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 139
Quoted 5 lines
mm/mmu_notifier.c +5
-0+ need_wake = true; + +/* +* The inv_end incorporates a deferred mechanism like +* rtnl_lock(). Adds and removes are queued until the final inv_end
Let me point out that rtnl_lock() itself is a one-liner that calls mutex_lock(). But I suppose if one studies that file closely there is more. :)
...
Quoted 42 lines
mm/mmu_notifier.c +42
-0+unsigned long mmu_range_read_begin (struct mmu_range_notifier *mrn) +{ +struct mmu_notifier_mm *mmn_mm = mrn->mm->mmu_notifier_mm; +unsigned long seq; +bool is_invalidating; + +/* +* If the mrn has a different seq value under the user_lock than we +* started with then it has collided. +* +* If the mrn currently has the same seq value as the mmn_mm seq, then +* it is currently between invalidate_start/end and is colliding. +* +* The locking looks broadly like this: +* mn_tree_invalidate_start(): mmu_range_read_begin(): +* spin_lock +* seq = READ_ONCE(mrn->invalidate_seq); +* seq == mmn_mm->invalidate_seq +* spin_unlock +* spin_lock +* seq = ++mmn_mm->invalidate_seq +* spin_unlock +* op->invalidate_range(): +* user_lock +* mmu_range_set_seq() +* mrn->invalidate_seq = seq +* user_unlock +* +* [Required: mmu_range_read_retry() == true] +* +* mn_itree_inv_end(): +* spin_lock +* seq = ++mmn_mm->invalidate_seq +* spin_unlock +* +* user_lock +* mmu_range_read_retry(): +* mrn->invalidate_seq != seq +* user_unlock +* +* Barriers are not needed here as any races here are closed by an +* eventual mmu_range_read_retry(), which provides a barrier via the
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 233
Quoted 10 lines
mm/mmu_notifier.c +10
-0+* user_lock. +*/ +spin_lock (&mmn_mm->lock); +/* Pairs with the WRITE_ONCE in mmu_range_set_seq() */ + seq =READ_ONCE (mrn->invalidate_seq); + is_invalidating = seq == mmn_mm->invalidate_seq; +spin_unlock (&mmn_mm->lock); + +/* +* mrn->invalidate_seq is always set to an odd value. This ensures
This claim just looks wrong the first N times one reads the code, given that there is mmu_range_set_seq() to set it to an arbitrary value! Maybe you mean
"is always set to an odd value when invalidating"??
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 237
Quoted 4 lines
mm/mmu_notifier.c +4
-0+* that if seq does wrap we will always clear the below sleep in some +* reasonable time as mmn_mm->invalidate_seq is even in the idle +* state. +*/
Let's move that comment higher up. The code that follows it has nothing to do with it, so it's confusing here.
...
Quoted 6 lines
mm/mmu_notifier.c +2
-0@@ -529,6 +852,166 @@ void mmu_notifier_put(struct mmu_notifier *mn) }EXPORT_SYMBOL_GPL (mmu_notifier_put); +static int __mmu_range_notifier_insert(struct mmu_range_notifier *mrn, + unsigned long start,
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 866
Quoted 10 lines
mm/mmu_notifier.c +10
-0+unsigned long length, + struct mmu_notifier_mm *mmn_mm, + struct mm_struct *mm) +{ + mrn->mm = mm; +RB_CLEAR_NODE (&mrn->interval_tree.rb); + mrn->interval_tree.start = start; +/* +* Note that the representation of the intervals in the interval tree +* considers the ending point as contained in the interval.
Thanks for that comment!
Quoted 25 lines
mm/mmu_notifier.c +25
-0+*/ +if (length ==0 || +check_add_overflow (start, length -1 , &mrn->interval_tree.last)) +return -EOVERFLOW ; + +/* pairs with mmdrop in mmu_range_notifier_remove() */ +mmgrab (mm); + +/* +* If some invalidate_range_start/end region is going on in parallel +* we don't know what VA ranges are affected, so we must assume this +* new range is included. +* +* If the itree is invalidating then we are not allowed to change +* it. Retrying until invalidation is done is tricky due to the +* possibility for live lock, instead defer the add to the unlock so +* this algorithm is deterministic. +* +* In all cases the value for the mrn->mr_invalidate_seq should be +* odd, see mmu_range_read_begin() +*/ +spin_lock (&mmn_mm->lock); +if (mmn_mm->active_invalidate_ranges) { +if (mn_itree_is_invalidating (mmn_mm)) + hlist_add_head(&mrn->deferred_item,
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 901
Quoted 10 lines
mm/mmu_notifier.c +10
-0+ &mmn_mm->deferred_list); + else { + mmn_mm->invalidate_seq |=1 ; +interval_tree_insert (&mrn->interval_tree, + &mmn_mm->itree); + } + mrn->invalidate_seq = mmn_mm->invalidate_seq; + }else { +WARN_ON (mn_itree_is_invalidating (mmn_mm)); + mrn->invalidate_seq = mmn_mm->invalidate_seq -1 ;
Ohhh, checkmate. I lose. Why is *subtracting* the right thing to do for seq numbers here? I'm acutely unhappy trying to figure this out. I suspect it's another unfortunate side effect of trying to use the lower bit of the seq number (even/odd) for something else.
Quoted 17 lines
mm/mmu_notifier.c +17
-0+interval_tree_insert (&mrn->interval_tree, &mmn_mm->itree); + } +spin_unlock (&mmn_mm->lock); +return 0 ; +} + +/** +* mmu_range_notifier_insert - Insert a range notifier +* @mrn: Range notifier to register +* @start: Starting virtual address to monitor +* @length: Length of the range to monitor +* @mm : mm_struct to attach to +* +* This function subscribes the range notifier for notifications from the mm. +* Upon return the ops related to mmu_range_notifier will be called whenever +* an event that intersects with the given range occurs. +*
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 928
Quoted 10 lines
mm/mmu_notifier.c +10
-0+* Upon return the range_notifier may not be present in the interval tree yet. +* The caller must use the normal range notifier locking flow via +* mmu_range_read_begin() to establish SPTEs for this range. +*/ +int mmu_range_notifier_insert (struct mmu_range_notifier *mrn, +unsigned long start,unsigned long length, +struct mm_struct *mm) +{ +struct mmu_notifier_mm *mmn_mm; +int ret;
Hmmm, I think a later patch improperly changes the above to "int ret = 0;". I'll check on that. It's correct here, though.
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 932
Quoted 4 lines
mm/mmu_notifier.c +4
-0+ +might_lock (&mm->mmap_sem); + + mmn_mm =smp_load_acquire (&mm->mmu_notifier_mm);
What does the above pair with? Should have a comment that specifies that.
thanks,
John Hubbard
NVIDIA
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 1 line · John Hubbard
On 10/28/19 1:10 PM, Jason Gunthorpe wrote:
[...]
Quoted 44 lines
Quoted 13 lines
/**
* enummmu_notifier_event- reason for the mmu notifier callbackinclude/linux/mmu_notifier.h +3
-0@@ -32,6 +34,9 @@ struct mmu_notifier_range;* access flags). User should soft dirty the page in the end callback to make * sure that anyone relying on soft dirtyness catch pages that might be written * through non CPU mappings. +* +* @MMU_NOTIFY_RELEASE: used during mmu_range_notifier invalidate to signal that +* the mm refcount is zero and the range is no longer accessible. */ enum mmu_notifier_event {MMU_NOTIFY_UNMAP =0 ,include/linux/mmu_notifier.h enum mmu_notifier_event { · line 48
Quoted 5 lines
include/linux/mmu_notifier.h +1
-0@@ -39,6 +44,7 @@ enum mmu_notifier_event {MMU_NOTIFY_PROTECTION_VMA ,MMU_NOTIFY_PROTECTION_PAGE ,MMU_NOTIFY_SOFT_DIRTY , + MMU_NOTIFY_RELEASE, };OK, let the naming debates begin! ha. Anyway, after careful study of the overall patch, and some browsing of the larger patchset, it's clear that:
* The new "MMU range notifier" that you've created is, approximately, a new object. It uses classic mmu notifiers inside, as an implementation detail, and it does *similar* things (notifications) as mmn's. But it's certainly not the same as mmn's, as shown later when you say the need to an entirely new ops struct, and data struct too.
Therefore, you need a separate events enum as well. This is important. MMN's won't be issuing
MMN_NOTIFY_RELEASEevents, nor will MNR's be issuing the first four prexisting MMU_NOTIFY_* items. So it would be a design mistake to glom them together, unless you ultimately decided to merge these MMN and MNR objects (which I don't really see any intention of, and that's fine).So this should read:
enum
mmu_range_notifier_event{
MMU_NOTIFY_RELEASE,
};...assuming that we stay with "
mmu_range_notifier" as a core name for this whole thing.
Quoted 6 lines
Also, it is best moved down to be next to the new MNR structs, so that all the MNR stuff is in one group.
Extra credit: IMHO, this clearly deserves to all be in a new
mmu_range_notifier.hheader file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.
The range notifier should get the event too, it would be a waste, i think it is an oversight here. The release event is fine so NAK to you separate event. Event is really an helper for notifier i had a set of patch for nouveau to leverage this i need to resucite them. So no need to split thing, i would just forward the event ie add event to mmu_range_notifier_ops.invalidate() i failed to catch that in v1 sorry.
[...]
Quoted 22 lines
Quoted 4 lines
include/linux/mmu_notifier.h +4
-0+struct mmu_range_notifier_ops { +bool (*invalidate)(struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range, +unsigned long cur_seq);include/linux/mmu_notifier.h struct mmu_notifier { · line 250
Quoted 10 lines
include/linux/mmu_notifier.h +10
-0+}; + +struct mmu_range_notifier { +struct interval_tree_node interval_tree; +const struct mmu_range_notifier_ops *ops; +struct hlist_node deferred_item; +unsigned long invalidate_seq; +struct mm_struct *mm; +}; +Again, now we have the new struct
mmu_range_notifier, and the old structmmu_notifier_range, and it's not good.Ideas:
a) Live with it.
Quoted 7 lines
b) (Discarded, too many callers): rename old one. Nope.
c) Rename new one. Ideas:
struct mmu_interval_notifier struct mmu_range_intersection ...other ideas?
I vote for interval_notifier we do want notifier in name but i am also fine with current name.
[...]
Quoted 19 lines
Quoted 11 lines
+ *
+ * Note that the core mm creates nested invalidate_range_start()/end() regions
+ * within the same thread, and runs invalidate_range_start()/end() in parallel
+ * on multiple CPUs. This is designed to not reduce concurrency or block
+ * progress on the mm side.
+ *
+ * As a secondary function, holding the full write side also serves to prevent
+ * writers for the itree, this is an optimization to avoid extra locking
+ * during invalidate_range_start/end notifiers.
+ *
+ * The write side has two states, fully excluded:Quoted 7 lines
+ * - mm->active_invalidate_ranges != 0 + * - mnn->invalidate_seq & 1 == True + * - some range on the mm_struct is being invalidated + * - the itree is not allowed to change+ *
+ * And partially excluded:
+ * -mm->active_invalidate_ranges!= 0
Quoted 2 lines
I assume this implies
mnn->invalidate_seq& 1 == False in this case? If so, let's say so. I'm probably getting that wrong, too.
Yes (mnn->invalidate_seq & 1) == 0
Quoted 34 lines
Quoted 17 lines
+ * - some range on the
mm_structis being invalidated
+ * - the itree is allowed to change
+ *
+ * The later state avoids some expensive work oninv_endin the common case of
+ * no mrn monitoring the VA.
+ */mm/mmu_notifier.c +10
-0+static bool mn_itree_is_invalidating (struct mmu_notifier_mm *mmn_mm) +{ +lockdep_assert_held (&mmn_mm->lock); +return mmn_mm->invalidate_seq &1 ; +} + +static struct mmu_range_notifier * +mn_itree_inv_start_range (struct mmu_notifier_mm *mmn_mm, +const struct mmu_notifier_range *range, +unsigned long *seq)mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 98
Quoted 10 lines
mm/mmu_notifier.c +10
-0+{ +struct interval_tree_node *node; +struct mmu_range_notifier *res =NULL ; + +spin_lock (&mmn_mm->lock); + mmn_mm->active_invalidate_ranges++; + node =interval_tree_iter_first (&mmn_mm->itree, range->start, + range->end -1 ); +if (node) { + mmn_mm->invalidate_seq |=1 ;OK, this either needs more documentation and assertions, or a different approach. Because I see addition, subtraction, AND, OR and booleans all being applied to this field, and it's darn near hopeless to figure out whether or not it really is even or odd at the right times.
Quoted 2 lines
Different approach: why not just add a
mmn_mm->is_invalidatingmember variable? It's not like you're short of space in that struct.
The invalidate_seq scheme looks fine to me, maybe it can use more comments.
Quoted 32 lines
Quoted 32 lines
mm/mmu_notifier.c +30
-0+ res = container_of(node, struct mmu_range_notifier, + interval_tree); + } + + *seq = mmn_mm->invalidate_seq; +spin_unlock (&mmn_mm->lock); +return res; +} + +static struct mmu_range_notifier * +mn_itree_inv_next (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) +{ +struct interval_tree_node *node; + + node =interval_tree_iter_next (&mrn->interval_tree, range->start, + range->end -1 ); +if (!node) +return NULL ; + return container_of(node, struct mmu_range_notifier, interval_tree); +} + +static void mn_itree_inv_end (struct mmu_notifier_mm *mmn_mm) +{ +struct mmu_range_notifier *mrn; +struct hlist_node *next; +bool need_wake = false; + +spin_lock (&mmn_mm->lock); + if (--mmn_mm->active_invalidate_ranges ||
Quoted 8 lines
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 134
Quoted 6 lines
mm/mmu_notifier.c +6
-0+ !mn_itree_is_invalidating(mmn_mm)) { +spin_unlock (&mmn_mm->lock); +return ; + } + + mmn_mm->invalidate_seq++;Is this the right place for an assertion that this is now an even value?
Yes at that point it should be even ie mmn_mm->active_invalidate_ranges == 0 and we are holding the lock thus nothing can set the lower bit of invalidate_seq and ++ should lead to even number.
Quoted 9 lines
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 139
Quoted 6 lines
mm/mmu_notifier.c +5
-0+ need_wake = true; + +/* +* The inv_end incorporates a deferred mechanism like +* rtnl_lock(). Adds and removes are queued until the final inv_end Let me point out that
rtnl_lock()itself is a one-liner that callsmutex_lock(). But I suppose if one studies that file closely there is more. :)
I think i commented in v1 about rtnl_lock() being something network people only might be familiar, i think i saw it documented somewhere, maybe a lwn article. But if you are familiar with network it is a think well understood ... for any reasonable network scholar ;)
Quoted 58 lines
...
Quoted 42 lines
mm/mmu_notifier.c +42
-0+unsigned long mmu_range_read_begin (struct mmu_range_notifier *mrn) +{ +struct mmu_notifier_mm *mmn_mm = mrn->mm->mmu_notifier_mm; +unsigned long seq; +bool is_invalidating; + +/* +* If the mrn has a different seq value under the user_lock than we +* started with then it has collided. +* +* If the mrn currently has the same seq value as the mmn_mm seq, then +* it is currently between invalidate_start/end and is colliding. +* +* The locking looks broadly like this: +* mn_tree_invalidate_start(): mmu_range_read_begin(): +* spin_lock +* seq = READ_ONCE(mrn->invalidate_seq); +* seq == mmn_mm->invalidate_seq +* spin_unlock +* spin_lock +* seq = ++mmn_mm->invalidate_seq +* spin_unlock +* op->invalidate_range(): +* user_lock +* mmu_range_set_seq() +* mrn->invalidate_seq = seq +* user_unlock +* +* [Required: mmu_range_read_retry() == true] +* +* mn_itree_inv_end(): +* spin_lock +* seq = ++mmn_mm->invalidate_seq +* spin_unlock +* +* user_lock +* mmu_range_read_retry(): +* mrn->invalidate_seq != seq +* user_unlock +* +* Barriers are not needed here as any races here are closed by an +* eventual mmu_range_read_retry(), which provides a barrier via the mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 233
Quoted 10 lines
mm/mmu_notifier.c +10
-0+* user_lock. +*/ +spin_lock (&mmn_mm->lock); +/* Pairs with the WRITE_ONCE in mmu_range_set_seq() */ + seq =READ_ONCE (mrn->invalidate_seq); + is_invalidating = seq == mmn_mm->invalidate_seq; +spin_unlock (&mmn_mm->lock); + +/* +* mrn->invalidate_seq is always set to an odd value. This ensures This claim just looks wrong the first N times one reads the code, given that there is
mmu_range_set_seq()to set it to an arbitrary value! Maybe you mean
Quoted 1 line
"is always set to an odd value when invalidating"??
No it is always odd, you must call mmu_range_set_seq() only from the op->invalidate_range() callback at which point the seq is odd. As well when mrn is added and its seq first set it is set to an odd value always. Maybe the comment, should read:
* mrn->invalidate_seq is always, yes always, set to an odd value. This ensures
To stress that it is not an error.
Quoted 8 lines
mm/mmu_notifier.c struct lockdep_map __mmu_notifier_invalidate_range_start_map = { · line 237
Quoted 5 lines
mm/mmu_notifier.c +4
-0+* that if seq does wrap we will always clear the below sleep in some +* reasonable time as mmn_mm->invalidate_seq is even in the idle +* state. +*/ Let's move that comment higher up. The code that follows it has nothing to do with it, so it's confusing here.
No the comment is in the right place, the fact that it is odd and that idle state is even explains why the wait() will never last forever. Already had a discussion on this in v1.
[...]
Quoted 24 lines
Quoted 24 lines
mm/mmu_notifier.c +24
-0+/* +* If some invalidate_range_start/end region is going on in parallel +* we don't know what VA ranges are affected, so we must assume this +* new range is included. +* +* If the itree is invalidating then we are not allowed to change +* it. Retrying until invalidation is done is tricky due to the +* possibility for live lock, instead defer the add to the unlock so +* this algorithm is deterministic. +* +* In all cases the value for the mrn->mr_invalidate_seq should be +* odd, see mmu_range_read_begin() +*/ +spin_lock (&mmn_mm->lock); +if (mmn_mm->active_invalidate_ranges) { +if (mn_itree_is_invalidating (mmn_mm)) +hlist_add_head (&mrn->deferred_item, + &mmn_mm->deferred_list); +else { + mmn_mm->invalidate_seq |=1 ; +interval_tree_insert (&mrn->interval_tree, + &mmn_mm->itree); + } + mrn->invalidate_seq = mmn_mm->invalidate_seq;
Quoted 8 lines
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 901
Quoted 3 lines
mm/mmu_notifier.c +3
-0+ }else { +WARN_ON (mn_itree_is_invalidating (mmn_mm)); + mrn->invalidate_seq = mmn_mm->invalidate_seq -1 ;Ohhh, checkmate. I lose. Why is *subtracting* the right thing to do for seq numbers here? I'm acutely unhappy trying to figure this out. I suspect it's another unfortunate side effect of trying to use the lower bit of the seq number (even/odd) for something else.
If there is no mmn_mm->active_invalidate_ranges then it means that mmn_mm->invalidate_seq is even and thus mmn_mm->invalidate_seq - 1 is an odd number which means that mrn->invalidate_seq is initialized to odd value and if you follow the rule for calling mmu_range_set_seq() then it will _always_ be an odd number and this close the loop with the above comments :)
Quoted 32 lines
Quoted 18 lines
mm/mmu_notifier.c +17
-0+interval_tree_insert (&mrn->interval_tree, &mmn_mm->itree); + } +spin_unlock (&mmn_mm->lock); +return 0 ; +} + +/** +* mmu_range_notifier_insert - Insert a range notifier +* @mrn: Range notifier to register +* @start: Starting virtual address to monitor +* @length: Length of the range to monitor +* @mm : mm_struct to attach to +* +* This function subscribes the range notifier for notifications from the mm. +* Upon return the ops related to mmu_range_notifier will be called whenever +* an event that intersects with the given range occurs. +* mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 928
Quoted 10 lines
mm/mmu_notifier.c +10
-0+* Upon return the range_notifier may not be present in the interval tree yet. +* The caller must use the normal range notifier locking flow via +* mmu_range_read_begin() to establish SPTEs for this range. +*/ +int mmu_range_notifier_insert (struct mmu_range_notifier *mrn, +unsigned long start,unsigned long length, +struct mm_struct *mm) +{ +struct mmu_notifier_mm *mmn_mm; +int ret;Hmmm, I think a later patch improperly changes the above to "int ret = 0;". I'll check on that. It's correct here, though.
Quoted 6 lines
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 932
Quoted 4 lines
mm/mmu_notifier.c +4
-0+ +might_lock (&mm->mmap_sem); + + mmn_mm =smp_load_acquire (&mm->mmu_notifier_mm);What does the above pair with? Should have a comment that specifies that.
It was discussed in v1 but maybe a comment of what was said back then would be helpful. Something like:
/*
* We need to insure that all writes to mm->mmu_notifier_mm are visible before
* any checks we do on mmn_mm below as otherwise CPU might re-order write done
* by another CPU core to mm->mmu_notifier_mm structure fields after the read
* belows.
*/
Cheers,
Jérôme
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 11 lines · Jerome Glisse
Quoted 4 lines
Extra credit: IMHO, this clearly deserves to all be in a new
mmu_range_notifier.hheader file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.The range notifier should get the event too, it would be a waste, i think it is an oversight here. The release event is fine so NAK to you separate event. Event is really an helper for notifier i had a set of patch for nouveau to leverage this i need to resucite them. So no need to split thing, i would just forward the event ie add event to
mmu_range_notifier_ops.invalidate()i failed to catch that in v1 sorry.
I think what you mean is already done?
struct mmu_range_notifier_ops {
bool (*invalidate)(struct mmu_range_notifier *mrn,
const struct mmu_notifier_range *range, unsigned long cur_seq);
Quoted 8 lines
No it is always odd, you must call
mmu_range_set_seq()only from theop->invalidate_range()callback at which point the seq is odd. As well when mrn is added and its seq first set it is set to an odd value always. Maybe the comment, should read:*
mrn->invalidate_seqis always, yes always, set to an odd value. This ensuresTo stress that it is not an error.
I went with this:
/*
* mrn->invalidate_seq must always be set to an odd value via
* mmu_range_set_seq() using the provided cur_seq from
* mn_itree_inv_start_range(). This ensures that if seq does wrap we
* will always clear the below sleep in some reasonable time as
* mmn_mm->invalidate_seq is even in the idle state.
*/
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
mm/mmu_notifier.c +11
-0+spin_lock (&mmn_mm->lock); +if (mmn_mm->active_invalidate_ranges) { +if (mn_itree_is_invalidating (mmn_mm)) +hlist_add_head (&mrn->deferred_item, + &mmn_mm->deferred_list); +else { + mmn_mm->invalidate_seq |=1 ; +interval_tree_insert (&mrn->interval_tree, + &mmn_mm->itree); + } + mrn->invalidate_seq = mmn_mm->invalidate_seq;Quoted 8 lines
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 901
Quoted 3 lines
mm/mmu_notifier.c +3
-0+ }else { +WARN_ON (mn_itree_is_invalidating (mmn_mm)); + mrn->invalidate_seq = mmn_mm->invalidate_seq -1 ;Ohhh, checkmate. I lose. Why is *subtracting* the right thing to do for seq numbers here? I'm acutely unhappy trying to figure this out. I suspect it's another unfortunate side effect of trying to use the lower bit of the seq number (even/odd) for something else.
Quoted 6 lines
If there is no
mmn_mm->active_invalidate_rangesthen it means thatmmn_mm->invalidate_seqis even and thusmmn_mm->invalidate_seq- 1 is an odd number which means thatmrn->invalidate_seqis initialized to odd value and if you follow the rule for callingmmu_range_set_seq()then it will _always_ be an odd number and this close the loop with the above comments :)
The key thing is that it is an odd value that will take a long time before mmn_mm->invalidate seq reaches it
Quoted 9 lines
Quoted 5 lines
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 932
Quoted 3 lines
mm/mmu_notifier.c +3
-0+might_lock (&mm->mmap_sem); + + mmn_mm =smp_load_acquire (&mm->mmu_notifier_mm);What does the above pair with? Should have a comment that specifies that.
It was discussed in v1 but maybe a comment of what was said back then would be helpful. Something like:
Quoted 6 lines
/*
* We need to insure that all writes tomm->mmu_notifier_mmare visible before
* any checks we do onmmn_mmbelow as otherwise CPU might re-order write done
* by another CPU core tomm->mmu_notifier_mmstructure fields after the read
* belows.
*/
This comment made it, just at the store side:
/*
* Serialize the update against mmu_notifier_unregister. A
* side note: mmu_notifier_release can't run concurrently with
* us because we hold the mm_users pin (either implicitly as
* current->mm or explicitly with get_task_mm() or similar).
* We can't race against any other mmu notifier method either
* thanks to mm_take_all_locks().
*
* release semantics on the initialization of the mmu_notifier_mm's
* contents are provided for unlocked readers. acquire can only be
* used while holding the mmgrab or mmget, and is safe because once
* created the mmu_notififer_mm is not freed until the mm is
* destroyed. As above, users holding the mmap_sem or one of the
* mm_take_all_locks() do not need to use acquire semantics. */
if (mmu_notifier_mm)
smp_store_release(&mm->mmu_notifier_mm, mmu_notifier_mm);
Which I think is really overly belaboring the typical smp store/release pattern, but people do seem unfamiliar with them...
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 14 lines · Jason Gunthorpe
Quoted 11 lines · Jerome Glisse
Quoted 4 lines
Extra credit: IMHO, this clearly deserves to all be in a new
mmu_range_notifier.hheader file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.The range notifier should get the event too, it would be a waste, i think it is an oversight here. The release event is fine so NAK to you separate event. Event is really an helper for notifier i had a set of patch for nouveau to leverage this i need to resucite them. So no need to split thing, i would just forward the event ie add event to
mmu_range_notifier_ops.invalidate()i failed to catch that in v1 sorry.
Quoted 6 lines
I think what you mean is already done?
struct
mmu_range_notifier_ops{
bool (*invalidate)(structmmu_range_notifier*mrn,
const structmmu_notifier_range*range, unsigned longcur_seq);
Yes it is sorry, i got confuse with mmu_range_notifier and mmu_notifier_range :) It is almost a palyndrome structure ;)
Quoted 12 lines
Quoted 9 lines
No it is always odd, you must call
mmu_range_set_seq()only from theop->invalidate_range()callback at which point the seq is odd. As well when mrn is added and its seq first set it is set to an odd value always. Maybe the comment, should read:*
mrn->invalidate_seqis always, yes always, set to an odd value. This ensuresTo stress that it is not an error.
I went with this:
Quoted 7 lines
/*
*mrn->invalidate_seqmust always be set to an odd value via
*mmu_range_set_seq()using the providedcur_seqfrom
*mn_itree_inv_start_range(). This ensures that if seq does wrap we
* will always clear the below sleep in some reasonable time as
*mmn_mm->invalidate_seqis even in the idle state.
*/
Yes fine with me.
[...]
Quoted 36 lines
Quoted 9 lines
Quoted 5 lines
mm/mmu_notifier.c void mmu_notifier_put(struct mmu_notifier *mn) · line 932
Quoted 3 lines
mm/mmu_notifier.c +3
-0+might_lock (&mm->mmap_sem); + + mmn_mm =smp_load_acquire (&mm->mmu_notifier_mm);What does the above pair with? Should have a comment that specifies that.
It was discussed in v1 but maybe a comment of what was said back then would be helpful. Something like:
Quoted 6 lines
/*
* We need to insure that all writes tomm->mmu_notifier_mmare visible before
* any checks we do onmmn_mmbelow as otherwise CPU might re-order write done
* by another CPU core tomm->mmu_notifier_mmstructure fields after the read
* belows.
*/This comment made it, just at the store side:
/*
* Serialize the update againstmmu_notifier_unregister. A
* side note:mmu_notifier_releasecan't run concurrently with
* us because we hold themm_userspin (either implicitly as
*current->mmor explicitly withget_task_mm()or similar).
* We can't race against any other mmu notifier method either
* thanks tomm_take_all_locks().
*
* release semantics on the initialization of the mmu_notifier_mm's
* contents are provided for unlocked readers. acquire can only be
* used while holding the mmgrab or mmget, and is safe because once
* created themmu_notififer_mmis not freed until the mm is
* destroyed. As above, users holding themmap_semor one of the
*mm_take_all_locks()do not need to use acquire semantics. */
if (mmu_notifier_mm)
smp_store_release(&mm->mmu_notifier_mm, mmu_notifier_mm);
Quoted 2 lines
Which I think is really overly belaboring the typical smp store/release pattern, but people do seem unfamiliar with them...
Perfect with me. I think also sometimes you forgot what memory model is and thus store/release pattern do, i know i do and i need to refresh my mind.
Cheers,
Jérôme
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 17 lines · Jerome Glisse
Quoted 16 lines · Jason Gunthorpe
Quoted 11 lines · Jerome Glisse
Quoted 4 lines
Extra credit: IMHO, this clearly deserves to all be in a new
mmu_range_notifier.hheader file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.The range notifier should get the event too, it would be a waste, i think it is an oversight here. The release event is fine so NAK to you separate event. Event is really an helper for notifier i had a set of patch for nouveau to leverage this i need to resucite them. So no need to split thing, i would just forward the event ie add event to
mmu_range_notifier_ops.invalidate()i failed to catch that in v1 sorry.I think what you mean is already done?
Quoted 7 lines
Quoted 4 lines
struct
mmu_range_notifier_ops{
bool (*invalidate)(structmmu_range_notifier*mrn,
const structmmu_notifier_range*range, unsigned longcur_seq);Yes it is sorry, i got confuse with
mmu_range_notifierandmmu_notifier_range:) It is almost a palyndrome structure ;)
Lets change the name then, this is clearly not working. I'll reflow everything tomorrow
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 23 lines · Jason Gunthorpe
Quoted 22 lines · Jerome Glisse
Quoted 20 lines · Jason Gunthorpe
Quoted 11 lines · Jerome Glisse
Quoted 4 lines
Extra credit: IMHO, this clearly deserves to all be in a new
mmu_range_notifier.hheader file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.The range notifier should get the event too, it would be a waste, i think it is an oversight here. The release event is fine so NAK to you separate event. Event is really an helper for notifier i had a set of patch for nouveau to leverage this i need to resucite them. So no need to split thing, i would just forward the event ie add event to
mmu_range_notifier_ops.invalidate()i failed to catch that in v1 sorry.I think what you mean is already done?
struct
mmu_range_notifier_ops{
bool (*invalidate)(structmmu_range_notifier*mrn,
const structmmu_notifier_range*range, unsigned longcur_seq);
Quoted 5 lines
Quoted 2 lines
Yes it is sorry, i got confuse with
mmu_range_notifierandmmu_notifier_range:) It is almost a palyndrome structure ;)Lets change the name then, this is clearly not working. I'll reflow everything tomorrow
Semantic patch to do that run from your linux kernel directory with your patch applied (you can run it one patch after the other and the git commit -a --fixup HEAD)
spatch --sp-file name-of-the-file-below --dir . --all-includes --in-place
%< ------------------------------------------------------------------ @@
@@
struct
-mmu_range_notifier
+mmu_interval_notifier
@@
@@
struct
-mmu_range_notifier
+mmu_interval_notifier
{...};
// Change mrn name to mmu_in
@@
struct mmu_interval_notifier *mrn;
@@
-mrn
+mmu_in
@@
identifier fn;
@@
fn(...,
-struct mmu_interval_notifier *mrn,
+struct mmu_interval_notifier *mmu_in,
...) {...} ------------------------------------------------------------------ >%
You need coccinelle (which provides spatch). It is untested but it should work also i could not come up with a nice name to update mrn as min is way too confusing. If you have better name feel free to use it.
Oh and coccinelle is pretty clever about code formating so it should do a good jobs at keeping things nicely formated and align.
Cheers,
Jérôme
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 64 lines · Jerome Glisse
Quoted 23 lines · Jason Gunthorpe
Quoted 22 lines · Jerome Glisse
Quoted 20 lines · Jason Gunthorpe
Quoted 11 lines · Jerome Glisse
Quoted 4 lines
Extra credit: IMHO, this clearly deserves to all be in a new
mmu_range_notifier.hheader file, but I know that's extra work. Maybe later as a follow-up patch, if anyone has the time.The range notifier should get the event too, it would be a waste, i think it is an oversight here. The release event is fine so NAK to you separate event. Event is really an helper for notifier i had a set of patch for nouveau to leverage this i need to resucite them. So no need to split thing, i would just forward the event ie add event to
mmu_range_notifier_ops.invalidate()i failed to catch that in v1 sorry.I think what you mean is already done?
struct
mmu_range_notifier_ops{
bool (*invalidate)(structmmu_range_notifier*mrn,
const structmmu_notifier_range*range, unsigned longcur_seq);Quoted 5 lines
Quoted 2 lines
Yes it is sorry, i got confuse with
mmu_range_notifierandmmu_notifier_range:) It is almost a palyndrome structure ;)Lets change the name then, this is clearly not working. I'll reflow everything tomorrow
Semantic patch to do that run from your linux kernel directory with your patch applied (you can run it one patch after the other and the git commit -a --fixup HEAD)
spatch --sp-file name-of-the-file-below --dir . --all-includes --in-place
%< ------------------------------------------------------------------ @@
@@
struct(quoted diff) +1
-1-mmu_range_notifier +mmu_interval_notifier@@
@@
struct(quoted diff) +1
-1-mmu_range_notifier +mmu_interval_notifier{...};
// Change mrn name to
mmu_in
@@
structmmu_interval_notifier*mrn;
@@(quoted diff) +1
-1-mrn +mmu_in@@
identifier fn;
@@
fn(...,(quoted diff) +1
-1-struct mmu_interval_notifier *mrn, +struct mmu_interval_notifier *mmu_in,...) {...}
Quoted 3 lines
You need coccinelle (which provides spatch). It is untested but it should work also i could not come up with a nice name to update mrn as min is way too confusing. If you have better name feel free to use it.
I used 'mni' as we already use 'mn' to refer to the notifier, and 'mmu_in' looks like some input parameter or something
It mostly worked, lots of comments to fix manually though:
https://github.com/jgunthorpe/linux/commits/mmu_notifier
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
In reply to John HubbardQuoted 2 lines · John Hubbard
Nice design, I love the seq foundation! So far, I'm not able to spot anything actually wrong with the implementation, sorry about that.
Alas :( I feel there must be a bug in here still, but onwards!
One of the main sad points was it didn't make sense to use the existing seqlock/seqcount primitives as they have both the wrong write concurrancy model and extra barriers that are not needed when it is always manipulated under a spinlock
Quoted 10 lines
1. There is a rather severe naming overlap (not technically a naming conflict, but still) with existing mmn work, which already has, for example:
struct
mmu_notifier_range...and you're adding:
struct
mmu_range_notifier...so I'll try to help sort that out.
Yes, I've been sad about this too.
Quoted 11 lines
So this should read:
enum
mmu_range_notifier_event{
MMU_NOTIFY_RELEASE,
};...assuming that we stay with "
mmu_range_notifier" as a core name for this whole thing.Also, it is best moved down to be next to the new MNR structs, so that all the MNR stuff is in one group.
I agree with Jerome, this enum is part of the 'struct mmu_notifier_range' (ie the description of the invalidation) and it doesn't really matter that only these new notifiers can be called with this type, it is still part of the mmu_notifier_range.
The comment already says it only applies to the mmu_range_notifier scheme..
Quoted 7 lines
Review struct mmu_notifier { · line 230
Quoted 5 lines
#define
MMU_NOTIFIER_RANGE_BLOCKABLE(1 << 0)(quoted diff) +0
-0@@ -222,6 +228,26 @@ struct mmu_notifier { unsigned int users; };That should also be moved down, next to the new structs.
Which this?
Quoted 9 lines
Review struct mmu_notifier { · line 236
Quoted 6 lines
(quoted diff) +6
-0+/** + * struct mmu_range_notifier_ops + * @invalidate: Upon return the caller must stop using any SPTEs within this + * range, this function can sleep. Return false if blocking was + * required but range is non-blocking + */How about this (I'm not sure I fully understand the return value, though):
Quoted 11 lines
/**
* structmmu_range_notifier_ops
* @invalidate: Upon return the caller must stop using any SPTEs within this
* range.
*
* This function is permitted to sleep.
*
* @Return: false if blocking was required, but @range is
* non-blocking.
*
*/
Is this kdoc format for function pointers?
Quoted 23 lines
Quoted 5 lines
(quoted diff) +4
-0+struct mmu_range_notifier_ops { + bool (*invalidate)(struct mmu_range_notifier *mrn, + const struct mmu_notifier_range *range, + unsigned long cur_seq);Review struct mmu_notifier { · line 250
Quoted 10 lines
(quoted diff) +10
-0+}; + +struct mmu_range_notifier { + struct interval_tree_node interval_tree; + const struct mmu_range_notifier_ops *ops; + struct hlist_node deferred_item; + unsigned long invalidate_seq; + struct mm_struct *mm; +}; +Again, now we have the new struct
mmu_range_notifier, and the old structmmu_notifier_range, and it's not good.Ideas:
a) Live with it.
Quoted 7 lines
b) (Discarded, too many callers): rename old one. Nope.
c) Rename new one. Ideas:
struct mmu_interval_notifier struct mmu_range_intersection ...other ideas?
This odd duality has already cause some confusion, but names here are hard. mmu_interval_notifier is the best alternative I've heard.
Changing this name is a lot of work - are we happy 'mmu_interval_notifier' is the right choice?
Quoted 6 lines
Review struct mmu_notifier { · line 252
Quoted 2 lines
(quoted diff) +2
-0+/** + * mmu_range_set_seq - Save the invalidation sequenceHow about:
*
mmu_range_set_seq- Set the .invalidate_seq to a new value.
It is not a 'new value', it is a value that is provided to the invalidate callback
Quoted 11 lines
Review struct mmu_notifier {
Quoted 10 lines
(quoted diff) +9
-0+ * @mrn - The mrn passed to invalidate + * @cur_seq - The cur_seq passed to invalidate + * + * This must be called unconditionally from the invalidate callback of a + * struct mmu_range_notifier_ops under the same lock that is used to call + * mmu_range_read_retry(). It updates the sequence number for later use by + * mmu_range_read_retry(). + * + * If the user does not call mmu_range_read_begin() or mmu_range_read_retry()
Quoted 3 lines
nit: "caller" is better than "user", when referring to...well, callers. "user" most often refers to user space, whereas a call stack and function calling is clearly what you're referring to here (and in other places, especially "user lock").
Done
Quoted 12 lines
Quoted 11 lines
+/**
+ *mmu_range_check_retry- Test if a collision has occurred
+ * mrn: The range under lock
+ * seq: The return of the matchingmmu_range_read_begin()
+ *
+ * This can be used in the critical section betweenmmu_range_read_begin()and
+ *mmu_range_read_retry(). A return of true indicates an invalidation has
+ * collided with this lock and a futuremmu_range_read_retry()will return
+ * true.
+ *
+ * False is not reliable and only suggests a collision has not happened. It
Quoted 1 line
let's say "suggests that a collision *may* not have occurred."
Sure
Quoted 7 lines
Quoted 5 lines
+/*
+ * This is a collision-retry read-side/write-side 'lock', a lot like a
+ * seqcount, however this allows multiple write-sides to hold it at
+ * once. Conceptually the write side is protecting the values of the PTEs in
+ * this mm, such that PTES cannot be read into SPTEs while any writer exists.Just to be kind, can we say "SPTEs (shadow PTEs)", just this once? :)
Haha, sure, why not
Quoted 11 lines
Quoted 8 lines
+ * The write side has two states, fully excluded:
+ * - mm->active_invalidate_ranges != 0 + * - mnn->invalidate_seq & 1 == True + * - some range on the mm_struct is being invalidated + * - the itree is not allowed to change+ *
+ * And partially excluded:
+ * -mm->active_invalidate_ranges!= 0I assume this implies
mnn->invalidate_seq& 1 == False in this case? If so, let's say so. I'm probably getting that wrong, too.
Yes that is right, done
Quoted 25 lines
Quoted 25 lines
+ * - some range on the
mm_structis being invalidated
+ * - the itree is allowed to change
+ *
+ * The later state avoids some expensive work oninv_endin the common case of
+ * no mrn monitoring the VA.
+ */(quoted diff) +18
-0+static bool mn_itree_is_invalidating(struct mmu_notifier_mm *mmn_mm) +{ + lockdep_assert_held(&mmn_mm->lock); + return mmn_mm->invalidate_seq & 1; +} + +static struct mmu_range_notifier * +mn_itree_inv_start_range(struct mmu_notifier_mm *mmn_mm, + const struct mmu_notifier_range *range, + unsigned long *seq) +{ + struct interval_tree_node *node; + struct mmu_range_notifier *res = NULL; + + spin_lock(&mmn_mm->lock); + mmn_mm->active_invalidate_ranges++; + node = interval_tree_iter_first(&mmn_mm->itree, range->start, + range->end - 1);
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (node) { + mmn_mm->invalidate_seq |= 1;OK, this either needs more documentation and assertions, or a different approach. Because I see addition, subtraction, AND, OR and booleans all being applied to this field, and it's darn near hopeless to figure out whether or not it really is even or odd at the right times.
This is a standard design for a seqlock scheme and follows the existing design of the linux seq lock.
The lower bit indicates the lock'd state and the upper bits indicate the generation of the lock
The operations on the lock itself are then:
seq |= 1 # Take the lock seq++ # Release an acquired lock seq & 1 # True if locked
Which is how this is written
Quoted 2 lines
Different approach: why not just add a
mmn_mm->is_invalidatingmember variable? It's not like you're short of space in that struct.
Splitting it makes alot of stuff more complex and unnatural.
The ops above could be put in inline wrappers, but they only occur only in functions already called mn_itree_inv_start_range() and mn_itree_inv_end() and mn_itree_is_invalidating().
There is the one 'take the lock' outlier in__mmu_range_notifier_insert() though
Quoted 8 lines
Quoted 8 lines
(quoted diff) +8
-0+static void mn_itree_inv_end(struct mmu_notifier_mm *mmn_mm) +{ + struct mmu_range_notifier *mrn; + struct hlist_node *next; + bool need_wake = false; + + spin_lock(&mmn_mm->lock); + if (--mmn_mm->active_invalidate_ranges ||
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ !mn_itree_is_invalidating(mmn_mm)) { + spin_unlock(&mmn_mm->lock); + return; + } + + mmn_mm->invalidate_seq++;Is this the right place for an assertion that this is now an even value?
Yes, but I'm reluctant to add such a runtime check on this fastish path.. How about a comment?
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ need_wake = true; + + /* + * The inv_end incorporates a deferred mechanism like + * rtnl_lock(). Adds and removes are queued until the final inv_endLet me point out that
rtnl_lock()itself is a one-liner that callsmutex_lock(). But I suppose if one studies that file closely there is more. :)
Lets change that to rtnl_unlock() then
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4
-0+ spin_lock(&mmn_mm->lock); + /* Pairs with the WRITE_ONCE in mmu_range_set_seq() */ + seq = READ_ONCE(mrn->invalidate_seq); + is_invalidating = seq == mmn_mm->invalidate_seq;
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ spin_unlock(&mmn_mm->lock); + + /* + * mrn->invalidate_seq is always set to an odd value. This ensuresThis claim just looks wrong the first N times one reads the code, given that there is
mmu_range_set_seq()to set it to an arbitrary value! Maybe you mean
mmu_range_set_seq() is NOT to be used to set to an arbitary value, it must only be used to set to the value provided in the invalidate() callback and that value is always odd. Lets make this super clear:
/*
* mrn->invalidate_seq must always be set to an odd value via
* mmu_range_set_seq() using the provided cur_seq from
* mn_itree_inv_start_range(). This ensures that if seq does wrap we
* will always clear the below sleep in some reasonable time as
* mmn_mm->invalidate_seq is even in the idle state.
*/
The invarient is that the 'struct mmu_range_notifier' always has an odd 'seq'
Quoted 7 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ * that if seq does wrap we will always clear the below sleep in some + * reasonable time as mmn_mm->invalidate_seq is even in the idle + * state. + */Let's move that comment higher up. The code that follows it has nothing to do with it, so it's confusing here.
The comment is explaining why the wait_event is safe, even if we wrap the sequence number, which is a significant and very subtle corner case. This is really why we have the even/odd thing at all.
Quoted 11 lines
Quoted 11 lines
(quoted diff) +11
-0+ spin_lock(&mmn_mm->lock); + if (mmn_mm->active_invalidate_ranges) { + if (mn_itree_is_invalidating(mmn_mm)) + hlist_add_head(&mrn->deferred_item, + &mmn_mm->deferred_list); + else { + mmn_mm->invalidate_seq |= 1; + interval_tree_insert(&mrn->interval_tree, + &mmn_mm->itree); + } + mrn->invalidate_seq = mmn_mm->invalidate_seq;
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ } else { + WARN_ON(mn_itree_is_invalidating(mmn_mm)); + mrn->invalidate_seq = mmn_mm->invalidate_seq - 1;Ohhh, checkmate. I lose. Why is *subtracting* the right thing to do for seq numbers here? I'm acutely unhappy trying to figure this out. I suspect it's another unfortunate side effect of trying to use the lower bit of the seq number (even/odd) for something else.
No, this is actually done for the seq number itself. We need to generate a seq number that is != the current invalidate_seq as this new mrn is not invalidating.
The best seq to use is one that the invalidate_seq will not reach for a long time, ie 'invalidate_seq + MAX' which is expressed as -1
The even/odd thing just takes care of itself naturally here as invalidate_seq is guarenteed even and -1 creates both an odd mrn value and a good seq number.
The algorithm would actually work correctly if this was 'mrn->invalidate_seq = 1', but occasionally things would block when they don't need to block.
Lets add a comment:
/*
* The starting seq for a mrn not under invalidation should be
* odd, not equal to the current invalidate_seq and
* invalidate_seq should not 'wrap' to the new seq any time
* soon.
*/
Quoted 9 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+int mmu_range_notifier_insert(struct mmu_range_notifier *mrn, + unsigned long start, unsigned long length, + struct mm_struct *mm) +{ + struct mmu_notifier_mm *mmn_mm; + int ret;Hmmm, I think a later patch improperly changes the above to "int ret = 0;". I'll check on that. It's correct here, though.
Looks OK in my tree?
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ might_lock(&mm->mmap_sem); + + mmn_mm = smp_load_acquire(&mm->mmu_notifier_mm);What does the above pair with? Should have a comment that specifies that.
smp_load_acquire() always pairs with smp_store_release() to the same memory, there is only one store, is a comment really needed?
Below are the comment updates I made, thanks!
Jason
include/linux/mmu_notifier.h +7 -6
@@ -302,15 +302,15 @@ void mmu_range_notifier_remove(struct mmu_range_notifier *mrn);/** * mmu_range_set_seq - Save the invalidation sequence * @mrn - The mrn passed to invalidate -* @cur_seq - The cur_seq passed to invalidate +* @cur_seq - The cur_seq passed to the invalidate() callback * * This must be called unconditionally from the invalidate callback of a * struct mmu_range_notifier_ops under the same lock that is used to call * mmu_range_read_retry(). It updates the sequence number for later use by -* mmu_range_read_retry(). +* mmu_range_read_retry(). The provided cur_seq will always be odd. * -* If the user does not call mmu_range_read_begin() or mmu_range_read_retry() -* then this call is not required. +* If the caller does not call mmu_range_read_begin() or +* mmu_range_read_retry() then this call is not required. */ static inline void mmu_range_set_seq (struct mmu_range_notifier *mrn,unsigned long cur_seq)
@@ -348,8 +348,9 @@ static inline bool mmu_range_read_retry(struct mmu_range_notifier *mrn,* collided with this lock and a future mmu_range_read_retry() will return * true. * -* False is not reliable and only suggests a collision has not happened. It -* can be called many times and does not have to hold the user provided lock. +* False is not reliable and only suggests a collision may not have +* occured. It can be called many times and does not have to hold the user +* provided lock. * * This call can be used as part of loops and other expensive operations to * expedite a retry.
mm/mmu_notifier.c +17 -7
@@ -51,7 +51,8 @@ struct mmu_notifier_mm {* This is a collision-retry read-side/write-side 'lock', a lot like a * seqcount, however this allows multiple write-sides to hold it at * once. Conceptually the write side is protecting the values of the PTEs in -* this mm, such that PTES cannot be read into SPTEs while any writer exists. +* this mm, such that PTES cannot be read into SPTEs (shadow PTEs) while any +* writer exists. * * Note that the core mm creates nested invalidate_range_start()/end() regions * within the same thread, and runs invalidate_range_start()/end() in parallel
@@ -64,12 +65,13 @@ struct mmu_notifier_mm {* * The write side has two states, fully excluded: * - mm->active_invalidate_ranges != 0 -* - mnn->invalidate_seq & 1 == True +* - mnn->invalidate_seq & 1 == True (odd) * - some range on the mm_struct is being invalidated * - the itree is not allowed to change * * And partially excluded: * - mm->active_invalidate_ranges != 0 +* - mnn->invalidate_seq & 1 == False (even) * - some range on the mm_struct is being invalidated * - the itree is allowed to change *
@@ -131,12 +133,13 @@ static void mn_itree_inv_end(struct mmu_notifier_mm *mmn_mm)return ; } +/* Make invalidate_seq even */ mmn_mm->invalidate_seq++; need_wake = true;/* * The inv_end incorporates a deferred mechanism like -* rtnl_lock(). Adds and removes are queued until the final inv_end +* rtnl_unlock(). Adds and removes are queued until the final inv_end * happens then they are progressed. This arrangement for tree updates * is used to avoid using a blocking lock during * invalidate_range_start.
@@ -230,10 +233,11 @@ unsigned long mmu_range_read_begin(struct mmu_range_notifier *mrn)spin_unlock (&mmn_mm->lock);/* -* mrn->invalidate_seq is always set to an odd value. This ensures -* that if seq does wrap we will always clear the below sleep in some -* reasonable time as mmn_mm->invalidate_seq is even in the idle -* state. +* mrn->invalidate_seq must always be set to an odd value via +* mmu_range_set_seq() using the provided cur_seq from +* mn_itree_inv_start_range(). This ensures that if seq does wrap we +* will always clear the below sleep in some reasonable time as +* mmn_mm->invalidate_seq is even in the idle state. */ lock_map_acquire (&__mmu_notifier_invalidate_range_start_map);lock_map_release (&__mmu_notifier_invalidate_range_start_map);
@@ -892,6 +896,12 @@ static int __mmu_range_notifier_insert(struct mmu_range_notifier *mrn, mrn->invalidate_seq = mmn_mm->invalidate_seq; }else {WARN_ON (mn_itree_is_invalidating (mmn_mm)); +/* +* The starting seq for a mrn not under invalidation should be +* odd, not equal to the current invalidate_seq and +* invalidate_seq should not 'wrap' to the new seq any time +* soon. +*/ mrn->invalidate_seq = mmn_mm->invalidate_seq -1 ;interval_tree_insert (&mrn->interval_tree, &mmn_mm->itree); }
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
John Hubbard
Reviewed-byRe: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
On 11/7/19 12:06 PM, Jason Gunthorpe wrote:
...
Quoted 9 lines
Quoted 3 lines
Also, it is best moved down to be next to the new MNR structs, so that all the MNR stuff is in one group.
I agree with Jerome, this enum is part of the 'struct
mmu_notifier_range' (ie the description of the invalidation) and it doesn't really matter that only these new notifiers can be called with this type, it is still part of themmu_notifier_range.
OK.
Quoted 11 lines
The comment already says it only applies to the
mmu_range_notifierscheme..Quoted 7 lines
Review struct mmu_notifier { · line 230
Quoted 5 lines
#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)(quoted diff) +0
-0@@ -222,6 +228,26 @@ struct mmu_notifier { unsigned int users; };That should also be moved down, next to the new structs.
Quoted 1 line
Which this?
I was referring to MMU_NOTIFIER_RANGE_BLOCKABLE, above. Trying to put all the new range notifier stuff in one place. But maybe not, if these are really not as separate as I thought.
Quoted 22 lines
Quoted 10 lines
Review struct mmu_notifier { · line 236
Quoted 7 lines
(quoted diff) +6
-0+/** + * struct mmu_range_notifier_ops + * @invalidate: Upon return the caller must stop using any SPTEs within this + * range, this function can sleep. Return false if blocking was + * required but range is non-blocking + */How about this (I'm not sure I fully understand the return value, though):
Quoted 11 lines
/**
* structmmu_range_notifier_ops
* @invalidate: Upon return the caller must stop using any SPTEs within this
* range.
*
* This function is permitted to sleep.
*
* @Return: false if blocking was required, but @range is
* non-blocking.
*
*/
Quoted 1 line
Is this kdoc format for function pointers?
heh, I'm sort of winging it, I'm not sure how function pointers are supposed to be documented in kdoc. Actually the only key take-away here is to write
"This function can sleep"
as a separate sentence..
...
Quoted 11 lines
Quoted 5 lines
c) Rename new one. Ideas:
struct mmu_interval_notifier struct mmu_range_intersection ...other ideas?This odd duality has already cause some confusion, but names here are hard.
mmu_interval_notifieris the best alternative I've heard.Changing this name is a lot of work - are we happy '
mmu_interval_notifier' is the right choice?
Yes, it's my favorite too. I'd vote for going with that.
...
Quoted 13 lines
Quoted 6 lines
OK, this either needs more documentation and assertions, or a different approach. Because I see addition, subtraction, AND, OR and booleans all being applied to this field, and it's darn near hopeless to figure out whether or not it really is even or odd at the right times.
This is a standard design for a seqlock scheme and follows the existing design of the linux seq lock.
The lower bit indicates the lock'd state and the upper bits indicate the generation of the lock
Quoted 6 lines
The operations on the lock itself are then:
seq |= 1 # Take the lock seq++ # Release an acquired lock seq & 1 # True if lockedWhich is how this is written
Very nice, would you be open to putting that into (any) one of the comment headers? That's an unusually clear and concise description:
/*
* This is a standard design for a seqlock scheme and follows the
* existing design of the linux seq lock.
*
* The lower bit indicates the lock'd state and the upper bits indicate
* the generation of the lock
*
* The operations on the lock itself are then:
* seq |= 1 # Take the lock * seq++ # Release an acquired lock * seq & 1 # True if locked
*/
Quoted 6 lines
Quoted 3 lines
Different approach: why not just add a
mmn_mm->is_invalidatingmember variable? It's not like you're short of space in that struct.Splitting it makes alot of stuff more complex and unnatural.
OK, agreed.
Quoted 24 lines
The ops above could be put in inline wrappers, but they only occur only in functions already called
mn_itree_inv_start_range()andmn_itree_inv_end()andmn_itree_is_invalidating().There is the one 'take the lock' outlier in
__mmu_range_notifier_insert()thoughQuoted 8 lines
Quoted 8 lines
(quoted diff) +8
-0+static void mn_itree_inv_end(struct mmu_notifier_mm *mmn_mm) +{ + struct mmu_range_notifier *mrn; + struct hlist_node *next; + bool need_wake = false; + + spin_lock(&mmn_mm->lock); + if (--mmn_mm->active_invalidate_ranges ||Quoted 8 lines
Review struct mmu_notifier { · line 250
Quoted 6 lines
(quoted diff) +6
-0+ !mn_itree_is_invalidating(mmn_mm)) { + spin_unlock(&mmn_mm->lock); + return; + } + + mmn_mm->invalidate_seq++;Is this the right place for an assertion that this is now an even value?
Quoted 2 lines
Yes, but I'm reluctant to add such a runtime check on this fastish path.. How about a comment?
Sure.
Quoted 11 lines
Quoted 9 lines
Review struct mmu_notifier {
Quoted 6 lines
(quoted diff) +5
-0+ need_wake = true; + + /* + * The inv_end incorporates a deferred mechanism like + * rtnl_lock(). Adds and removes are queued until the final inv_endLet me point out that
rtnl_lock()itself is a one-liner that callsmutex_lock(). But I suppose if one studies that file closely there is more. :)Lets change that to
rtnl_unlock()then
Thanks :)
...
Quoted 10 lines
Quoted 5 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ * mrn->invalidate_seq is always set to an odd value. This ensuresThis claim just looks wrong the first N times one reads the code, given that there is
mmu_range_set_seq()to set it to an arbitrary value! Maybe you mean
mmu_range_set_seq()is NOT to be used to set to an arbitary value, it must only be used to set to the value provided in theinvalidate()callback and that value is always odd. Lets make this super clear:
Quoted 8 lines
/*
*mrn->invalidate_seqmust always be set to an odd value via
*mmu_range_set_seq()using the providedcur_seqfrom
*mn_itree_inv_start_range(). This ensures that if seq does wrap we
* will always clear the below sleep in some reasonable time as
*mmn_mm->invalidate_seqis even in the idle state.
*/
OK, that helps a lot.
...
Quoted 22 lines
Quoted 6 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ mrn->invalidate_seq = mmn_mm->invalidate_seq - 1;Ohhh, checkmate. I lose. Why is *subtracting* the right thing to do for seq numbers here? I'm acutely unhappy trying to figure this out. I suspect it's another unfortunate side effect of trying to use the lower bit of the seq number (even/odd) for something else.
No, this is actually done for the seq number itself. We need to generate a seq number that is != the current
invalidate_seqas this new mrn is not invalidating.The best seq to use is one that the
invalidate_seqwill not reach for a long time, ie 'invalidate_seq+ MAX' which is expressed as -1The even/odd thing just takes care of itself naturally here as
invalidate_seqis guarenteed even and -1 creates both an odd mrn value and a good seq number.The algorithm would actually work correctly if this was '
mrn->invalidate_seq= 1', but occasionally things would block when they don't need to block.
Quoted 8 lines
Lets add a comment:
/*
* The starting seq for a mrn not under invalidation should be
* odd, not equal to the currentinvalidate_seqand
*invalidate_seqshould not 'wrap' to the new seq any time
* soon.
*/
Very helpful. How about this additional tweak:
/*
* The starting seq for a mrn not under invalidation should be
* odd, not equal to the current invalidate_seq and
* invalidate_seq should not 'wrap' to the new seq any time
* soon. Subtracting 1 from the current (even) value achieves that. */
Quoted 11 lines
Quoted 10 lines
Review
Quoted 7 lines
(quoted diff) +6
-0+int mmu_range_notifier_insert(struct mmu_range_notifier *mrn, + unsigned long start, unsigned long length, + struct mm_struct *mm) +{ + struct mmu_notifier_mm *mmn_mm; + int ret;Hmmm, I think a later patch improperly changes the above to "int ret = 0;". I'll check on that. It's correct here, though.
Quoted 1 line
Looks OK in my tree?
Nope, that's how I found it. The top of your mmu_notifier branch has this:
int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
{
struct mmu_notifier_mm *mmn_mm = range->mm->mmu_notifier_mm;
int ret = 0;
if (mmn_mm->has_interval) {
ret = mn_itree_invalidate(mmn_mm, range);
if (ret)
return ret;
}
if (!hlist_empty(&mmn_mm->list))
return mn_hlist_invalidate_range_start(mmn_mm, range);
return 0;
}Quoted 114 lines
Quoted 6 lines
Review
Quoted 4 lines
(quoted diff) +3
-0+ might_lock(&mm->mmap_sem); + + mmn_mm = smp_load_acquire(&mm->mmu_notifier_mm);What does the above pair with? Should have a comment that specifies that.
smp_load_acquire()always pairs withsmp_store_release()to the same memory, there is only one store, is a comment really needed?Below are the comment updates I made, thanks!
Jason
include/linux/mmu_notifier.h +7
-6@@ -302,15 +302,15 @@ void mmu_range_notifier_remove(struct mmu_range_notifier *mrn);/** * mmu_range_set_seq - Save the invalidation sequence * @mrn - The mrn passed to invalidate -* @cur_seq - The cur_seq passed to invalidate +* @cur_seq - The cur_seq passed to the invalidate() callback * * This must be called unconditionally from the invalidate callback of a * struct mmu_range_notifier_ops under the same lock that is used to call * mmu_range_read_retry(). It updates the sequence number for later use by -* mmu_range_read_retry(). +* mmu_range_read_retry(). The provided cur_seq will always be odd. * -* If the user does not call mmu_range_read_begin() or mmu_range_read_retry() -* then this call is not required. +* If the caller does not call mmu_range_read_begin() or +* mmu_range_read_retry() then this call is not required. */ static inline void mmu_range_set_seq (struct mmu_range_notifier *mrn,unsigned long cur_seq)@@ -348,8 +348,9 @@ static inline bool mmu_range_read_retry(struct mmu_range_notifier *mrn,* collided with this lock and a future mmu_range_read_retry() will return * true. * -* False is not reliable and only suggests a collision has not happened. It -* can be called many times and does not have to hold the user provided lock. +* False is not reliable and only suggests a collision may not have +* occured. It can be called many times and does not have to hold the user +* provided lock. * * This call can be used as part of loops and other expensive operations to * expedite a retry. mm/mmu_notifier.c +11
-7@@ -51,7 +51,8 @@ struct mmu_notifier_mm {* This is a collision-retry read-side/write-side 'lock', a lot like a * seqcount, however this allows multiple write-sides to hold it at * once. Conceptually the write side is protecting the values of the PTEs in -* this mm, such that PTES cannot be read into SPTEs while any writer exists. +* this mm, such that PTES cannot be read into SPTEs (shadow PTEs) while any +* writer exists. * * Note that the core mm creates nested invalidate_range_start()/end() regions * within the same thread, and runs invalidate_range_start()/end() in parallel @@ -64,12 +65,13 @@ struct mmu_notifier_mm {* * The write side has two states, fully excluded: * - mm->active_invalidate_ranges != 0 -* - mnn->invalidate_seq & 1 == True +* - mnn->invalidate_seq & 1 == True (odd) * - some range on the mm_struct is being invalidated * - the itree is not allowed to change * * And partially excluded: * - mm->active_invalidate_ranges != 0 +* - mnn->invalidate_seq & 1 == False (even) * - some range on the mm_struct is being invalidated * - the itree is allowed to change * @@ -131,12 +133,13 @@ static void mn_itree_inv_end(struct mmu_notifier_mm *mmn_mm)return ; } +/* Make invalidate_seq even */ mmn_mm->invalidate_seq++; need_wake = true;/* * The inv_end incorporates a deferred mechanism like -* rtnl_lock(). Adds and removes are queued until the final inv_end +* rtnl_unlock(). Adds and removes are queued until the final inv_end * happens then they are progressed. This arrangement for tree updates * is used to avoid using a blocking lock during * invalidate_range_start. @@ -230,10 +233,11 @@ unsigned long mmu_range_read_begin(struct mmu_range_notifier *mrn)spin_unlock (&mmn_mm->lock);/* -* mrn->invalidate_seq is always set to an odd value. This ensures -* that if seq does wrap we will always clear the below sleep in some -* reasonable time as mmn_mm->invalidate_seq is even in the idle -* state. +* mrn->invalidate_seq must always be set to an odd value via +* mmu_range_set_seq() using the provided cur_seq from +* mn_itree_inv_start_range(). This ensures that if seq does wrap we +* will always clear the below sleep in some reasonable time as +* mmn_mm->invalidate_seq is even in the idle state. */ lock_map_acquire (&__mmu_notifier_invalidate_range_start_map);lock_map_release (&__mmu_notifier_invalidate_range_start_map);@@ -892,6 +896,12 @@ static int __mmu_range_notifier_insert(struct mmu_range_notifier *mrn, mrn->invalidate_seq = mmn_mm->invalidate_seq; }else {
mm/mmu_notifier.c static int __mmu_range_notifier_insert(struct mmu_range_notifier *mrn, · line 907
Quoted 10 lines
mm/mmu_notifier.c +6
-0WARN_ON (mn_itree_is_invalidating (mmn_mm)); +/* +* The starting seq for a mrn not under invalidation should be +* odd, not equal to the current invalidate_seq and +* invalidate_seq should not 'wrap' to the new seq any time +* soon. +*/ mrn->invalidate_seq = mmn_mm->invalidate_seq -1 ;interval_tree_insert (&mrn->interval_tree, &mmn_mm->itree); }
Looks good. We're just polishing up minor points now, so you can add:
thanks,
Signature · 6 lines
John Hubbard NVIDIA _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 21 lines · John Hubbard
Quoted 21 lines
Quoted 20 lines
Quoted 6 lines
+/**
+ * structmmu_range_notifier_ops
+ * @invalidate: Upon return the caller must stop using any SPTEs within this
+ * range, this function can sleep. Return false if blocking was
+ * required but range is non-blocking
+ */How about this (I'm not sure I fully understand the return value, though):
/**
* structmmu_range_notifier_ops
* @invalidate: Upon return the caller must stop using any SPTEs within this
* range.
*
* This function is permitted to sleep.
*
* @Return: false if blocking was required, but @range is
* non-blocking.
*
*/
Quoted 8 lines
Quoted 1 line
Is this kdoc format for function pointers?
heh, I'm sort of winging it, I'm not sure how function pointers are supposed to be documented in kdoc. Actually the only key take-away here is to write
"This function can sleep"
as a separate sentence..
Sure
Quoted 7 lines
Quoted 5 lines
This odd duality has already cause some confusion, but names here are hard.
mmu_interval_notifieris the best alternative I've heard.Changing this name is a lot of work - are we happy '
mmu_interval_notifier' is the right choice?Yes, it's my favorite too. I'd vote for going with that.
Okay, lets give it a go
Quoted 2 lines
Very nice, would you be open to putting that into (any) one of the comment headers? That's an unusually clear and concise description:
Yep, done
Quoted 15 lines
Quoted 11 lines
Quoted 9 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+int mmu_range_notifier_insert(struct mmu_range_notifier *mrn, + unsigned long start, unsigned long length, + struct mm_struct *mm) +{ + struct mmu_notifier_mm *mmn_mm; + int ret;Hmmm, I think a later patch improperly changes the above to "int ret = 0;". I'll check on that. It's correct here, though.
Looks OK in my tree?
Nope, that's how I found it. The top of your
mmu_notifierbranch has this:int
__mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
Quoted 13 lines
{
struct mmu_notifier_mm *mmn_mm = range->mm->mmu_notifier_mm;
int ret = 0;
if (mmn_mm->has_interval) {
ret = mn_itree_invalidate(mmn_mm, range);
if (ret)
return ret;
}
if (!hlist_empty(&mmn_mm->list))
return mn_hlist_invalidate_range_start(mmn_mm, range);
return 0;
}Ah, that is a different function :) Fixed
Quoted 3 lines
Looks good. We're just polishing up minor points now, so you can add:
Reviewed-by John Hubbard <[email protected]>
Great, thanks, I'll post a v3 with the rename
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
In reply to Jason GunthorpeQuoted 11 lines · Jason Gunthorpe
Quoted 10 lines
enum
mmu_range_notifier_event{
MMU_NOTIFY_RELEASE,
};...assuming that we stay with "
mmu_range_notifier" as a core name for this whole thing.Also, it is best moved down to be next to the new MNR structs, so that all the MNR stuff is in one group.
Quoted 7 lines
I agree with Jerome, this enum is part of the 'struct
mmu_notifier_range' (ie the description of the invalidation) and it doesn't really matter that only these new notifiers can be called with this type, it is still part of themmu_notifier_range.The comment already says it only applies to the
mmu_range_notifierscheme..
In fact the enum is entirely unused. We might as well just kill it off entirely.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Jason Gunthorpe
Quoted 10 lines
enum
mmu_range_notifier_event{
MMU_NOTIFY_RELEASE,
};...assuming that we stay with "
mmu_range_notifier" as a core name for this whole thing.Also, it is best moved down to be next to the new MNR structs, so that all the MNR stuff is in one group.
I agree with Jerome, this enum is part of the 'struct
mmu_notifier_range' (ie the description of the invalidation) and it doesn't really matter that only these new notifiers can be called with this type, it is still part of themmu_notifier_range.
Quoted 5 lines
Quoted 2 lines
The comment already says it only applies to the
mmu_range_notifierscheme..In fact the enum is entirely unused. We might as well just kill it off entirely.
I had patches to use it, i need to re-post them. I posted them long ago and i droped the ball. I will re-spin after this.
Cheers,
Jérôme
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH v2 3/15 mm/hmm: allow hmm_range to be used with a mmu_range_notifier or hmm_mirror
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
hmm_mirror's handling of ranges does not use a sequence count which results in this bug:
CPU0 CPU1
hmm_range_wait_until_valid(range)
valid == true
hmm_range_fault(range)hmm_invalidate_range_start()
range->valid = falsehmm_invalidate_range_end()
range->valid = true
hmm_range_valid(range)
valid == trueWhere the hmm_range_valid should not have succeeded.
Adding the required sequence count would make it nearly identical to the new mmu_range_notifier. Instead replace the hmm_mirror stuff with mmu_range_notifier.
Co-existence of the two APIs is the first step.
--- include/linux/hmm.h | 5 +++++ mm/hmm.c | 25 +++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-)
include/linux/hmm.h +5 -0
@@ -145,6 +145,9 @@ enum hmm_pfn_value_e {/* * struct hmm_range - track invalidation lock on virtual address range * +* @notifier: an optional mmu_range_notifier +* @notifier_seq: when notifier is used this is the result of +* mmu_range_read_begin() * @hmm: the core HMM structure this range is active against * @vma: the vm area struct for the range * @list: all range lock are on a list
@@ -159,6 +162,8 @@ enum hmm_pfn_value_e {* @valid: pfns array did not change since it has been fill by an HMM function */ struct hmm_range { +struct mmu_range_notifier *notifier; +unsigned long notifier_seq;struct hmm *hmm;struct list_head list;unsigned long start;
mm/hmm.c +19 -6
@@ -852,6 +852,14 @@ void hmm_range_unregister(struct hmm_range *range) }EXPORT_SYMBOL (hmm_range_unregister); +static bool needs_retry (struct hmm_range *range) +{ +if (range->notifier) +return mmu_range_check_retry (range->notifier, + range->notifier_seq); +return !range->valid; +} +static const struct mm_walk_ops hmm_walk_ops = { .pud_entry = hmm_vma_walk_pud, .pmd_entry = hmm_vma_walk_pmd,
@@ -892,18 +900,23 @@ long hmm_range_fault(struct hmm_range *range, unsigned int flags)const unsigned long device_vma =VM_IO |VM_PFNMAP |VM_MIXEDMAP ;unsigned long start = range->start, end;struct hmm_vma_walk hmm_vma_walk; -struct hmm *hmm = range->hmm; +struct mm_struct *mm;struct vm_area_struct *vma;int ret; -lockdep_assert_held (&hmm->mmu_notifier.mm->mmap_sem); +if (range->notifier) + mm = range->notifier->mm; +else + mm = range->hmm->mmu_notifier.mm; + +lockdep_assert_held (&mm->mmap_sem); do { /* If range is no longer valid force retry. */ - if (!range->valid) + if (needs_retry(range)) return -EBUSY; - vma = find_vma(hmm->mmu_notifier.mm, start); + vma = find_vma(mm, start); if (vma == NULL || (vma->vm_flags & device_vma)) return -EFAULT;
@@ -933,7 +946,7 @@ long hmm_range_fault(struct hmm_range *range, unsigned int flags) start = hmm_vma_walk.last;/* Keep trying while the range is valid. */ - }while (ret == -EBUSY && range->valid); + }while (ret == -EBUSY && !needs_retry (range));if (ret) {unsigned long i;
@@ -991,7 +1004,7 @@ long hmm_range_dma_map(struct hmm_range *range, struct device *device,continue ;/* Check if range is being invalidated */ -if (!range->valid) { +if (needs_retry (range)) { ret = -EBUSY ; goto unmap; }
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 4/15 mm/hmm: define the pre-processor related parts of hmm.h even if disabled
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
Only the function calls are stubbed out with static inlines that always fail. This is the standard way to write a header for an optional component and makes it easier for drivers that only optionally need HMM_MIRROR.
--- include/linux/hmm.h | 59 ++++++++++++++++++++++++++++++++++++--------- kernel/fork.c | 1 - 2 files changed, 47 insertions(+), 13 deletions(-)
include/linux/hmm.h +47 -12
@@ -62,8 +62,6 @@#include <linux/kconfig.h> #include <asm/pgtable.h> -#ifdef CONFIG_HMM_MIRROR -#include <linux/device.h> #include <linux/migrate.h> #include <linux/memremap.h>
@@ -374,6 +372,15 @@ struct hmm_mirror {struct list_head list; }; +/* +* Retry fault if non-blocking, drop mmap_sem and return -EAGAIN in that case. +*/ +#define HMM_FAULT_ALLOW_RETRY (1 << 0) + +/* Don't fault in missing PTEs, just snapshot the current state. */ +#define HMM_FAULT_SNAPSHOT (1 << 1) + +#ifdef CONFIG_HMM_MIRROR int hmm_mirror_register (struct hmm_mirror *mirror,struct mm_struct *mm);void hmm_mirror_unregister (struct hmm_mirror *mirror);
@@ -383,14 +390,6 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror);int hmm_range_register (struct hmm_range *range,struct hmm_mirror *mirror);void hmm_range_unregister (struct hmm_range *range); -/* -* Retry fault if non-blocking, drop mmap_sem and return -EAGAIN in that case. -*/ -#define HMM_FAULT_ALLOW_RETRY (1 << 0) - -/* Don't fault in missing PTEs, just snapshot the current state. */ -#define HMM_FAULT_SNAPSHOT (1 << 1) -long hmm_range_fault (struct hmm_range *range,unsigned int flags); long hmm_range_dma_map(struct hmm_range *range,
@@ -401,6 +400,44 @@ long hmm_range_dma_unmap(struct hmm_range *range,struct device *device, dma_addr_t *daddrs, bool dirty); +#else +int hmm_mirror_register (struct hmm_mirror *mirror,struct mm_struct *mm) +{ +return -EOPNOTSUPP ; +} + +void hmm_mirror_unregister (struct hmm_mirror *mirror) +{ +} + +int hmm_range_register (struct hmm_range *range,struct hmm_mirror *mirror) +{ +return -EOPNOTSUPP ; +} + +void hmm_range_unregister (struct hmm_range *range) +{ +} + +static inline long hmm_range_fault (struct hmm_range *range,unsigned int flags) +{ +return -EOPNOTSUPP ; +} + +static inline long hmm_range_dma_map (struct hmm_range *range, +struct device *device,dma_addr_t *daddrs, +unsigned int flags) +{ +return -EOPNOTSUPP ; +} + +static inline long hmm_range_dma_unmap (struct hmm_range *range, +struct device *device, +dma_addr_t *daddrs,bool dirty) +{ +return -EOPNOTSUPP ; +} +#endif /* * HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range
@@ -411,6 +448,4 @@ long hmm_range_dma_unmap(struct hmm_range *range,*/ #define HMM_RANGE_DEFAULT_TIMEOUT 1000 -#endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */ -#endif /* LINUX_HMM_H */
kernel/fork.c +0 -1
@@ -40,7 +40,6 @@#include <linux/binfmts.h> #include <linux/mman.h> #include <linux/mmu_notifier.h> -#include <linux/hmm.h> #include <linux/fs.h> #include <linux/mm.h> #include <linux/vmacache.h>
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 5/15 RDMA/odp: Use mmu_range_notifier_insert()
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
Replace the internal interval tree based mmu notifier with the new common mmu_range_notifier_insert() API. This removes a lot of code and fixes a deadlock that can be triggered in ODP:
zap_page_range()
mmu_notifier_invalidate_range_start()
[..]
ib_umem_notifier_invalidate_range_start()
down_read(&per_mm->umem_rwsem)
unmap_single_vma()
[..]
__split_huge_page_pmd()
mmu_notifier_invalidate_range_start()
[..]
ib_umem_notifier_invalidate_range_start()
down_read(&per_mm->umem_rwsem) // DEADLOCK
mmu_notifier_invalidate_range_end()
up_read(&per_mm->umem_rwsem)
mmu_notifier_invalidate_range_end()
up_read(&per_mm->umem_rwsem)The umem_rwsem is held across the range_start/end as the ODP algorithm for invalidate_range_end cannot tolerate changes to the interval tree. However, due to the nested invalidation regions the second down_read() can deadlock if there are competing writers. The new core code provides an alternative scheme to solve this problem.
--- drivers/infiniband/core/device.c | 1 - drivers/infiniband/core/umem_odp.c | 288 +++------------------------ drivers/infiniband/hw/mlx5/mlx5_ib.h | 7 +- drivers/infiniband/hw/mlx5/mr.c | 3 +- drivers/infiniband/hw/mlx5/odp.c | 50 +++-- include/rdma/ib_umem_odp.h | 65 ++---- include/rdma/ib_verbs.h | 2 - 7 files changed, 69 insertions(+), 347 deletions(-)
drivers/infiniband/core/device.c +0 -1
@@ -2617,7 +2617,6 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, get_vf_config);SET_DEVICE_OP (dev_ops, get_vf_stats);SET_DEVICE_OP (dev_ops, init_port); -SET_DEVICE_OP (dev_ops, invalidate_range);SET_DEVICE_OP (dev_ops, iw_accept);SET_DEVICE_OP (dev_ops, iw_add_ref);SET_DEVICE_OP (dev_ops, iw_connect);
drivers/infiniband/core/umem_odp.c +30 -258
398 lines not on this page · raw message
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -5
@@ -1263,8 +1263,6 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *ibdev);void mlx5_ib_odp_cleanup_one (struct mlx5_ib_dev *ibdev);int __init mlx5_ib_odp_init (void );void mlx5_ib_odp_cleanup (void ); -void mlx5_ib_invalidate_range (struct ib_umem_odp *umem_odp,unsigned long start, -unsigned long end);void mlx5_odp_init_mr_cache_entry (struct mlx5_cache_ent *ent);void mlx5_odp_populate_klm (struct mlx5_klm *pklm,size_t offset,size_t nentries,struct mlx5_ib_mr *mr,int flags);
@@ -1294,11 +1292,10 @@ mlx5_ib_advise_mr_prefetch(struct ib_pd *pd, {return -EOPNOTSUPP ; } -static inline void mlx5_ib_invalidate_range (struct ib_umem_odp *umem_odp, -unsigned long start, -unsigned long end){};#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ +extern const struct mmu_range_notifier_ops mlx5_mn_ops; +/* Needed for rep profile */ void __mlx5_ib_remove(struct mlx5_ib_dev *dev, const struct mlx5_ib_profile *profile,
drivers/infiniband/hw/mlx5/mr.c +2 -1
@@ -743,7 +743,8 @@ static int mr_umem_get(struct mlx5_ib_dev *dev, struct ib_udata *udata,if (access_flags &IB_ACCESS_ON_DEMAND ) {struct ib_umem_odp *odp; - odp =ib_umem_odp_get (udata, start, length, access_flags); + odp =ib_umem_odp_get (udata, start, length, access_flags, + &mlx5_mn_ops);if (IS_ERR (odp)) {mlx5_ib_dbg (dev,"umem get failed (%ld)\n" ,PTR_ERR (odp));
drivers/infiniband/hw/mlx5/odp.c +23 -27
116 lines not on this page · raw message
include/rdma/ib_umem_odp.h +12 -53
116 lines not on this page · raw message
include/rdma/ib_verbs.h +0 -2
@@ -2422,8 +2422,6 @@ struct ib_device_ops { u64 iova);int (*unmap_fmr)(struct list_head *fmr_list);int (*dealloc_fmr)(struct ib_fmr *fmr); -void (*invalidate_range)(struct ib_umem_odp *umem_odp, -unsigned long start,unsigned long end);int (*attach_mcast)(struct ib_qp *qp,union ib_gid *gid,u16 lid);int (*detach_mcast)(struct ib_qp *qp,union ib_gid *gid,u16 lid); struct ib_xrcd *(*alloc_xrcd)(struct ib_device *device,
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 6/15 RDMA/hfi1: Use mmu_range_notifier_inset for user_exp_rcv
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
This converts one of the two users of mmu_notifiers to use the new API. The conversion is fairly straightforward, however the existing use of notifiers here seems to be racey.
--- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/hfi1/hfi.h | 2 +- drivers/infiniband/hw/hfi1/user_exp_rcv.c | 146 +++++++++------------- drivers/infiniband/hw/hfi1/user_exp_rcv.h | 3 +- 4 files changed, 60 insertions(+), 93 deletions(-)
drivers/infiniband/hw/hfi1/file_ops.c +1 -1
@@ -1138,7 +1138,7 @@ static int get_ctxt_info(struct hfi1_filedata *fd, unsigned long arg, u32 len)HFI1_CAP_UGET_MASK (uctxt->flags,MASK ) |HFI1_CAP_KGET_MASK (uctxt->flags,K2U );/* adjust flag if this fd is not able to cache */ -if (!fd->handler) +if (!fd->use_mn) cinfo.runtime_flags |=HFI1_CAP_TID_UNMAP ;/* no caching */ cinfo.num_active =hfi1_count_active_units ();
drivers/infiniband/hw/hfi1/hfi.h +1 -1
@@ -1444,7 +1444,7 @@ struct hfi1_filedata {/* for cpu affinity; -1 if none */ int rec_cpu_num;u32 tid_n_pinned; -struct mmu_rb_handler *handler; +bool use_mn;struct tid_rb_node **entry_to_rb;spinlock_t tid_lock;/* protect tid_[limit,used] counters */ u32 tid_limit;
drivers/infiniband/hw/hfi1/user_exp_rcv.c +56 -90
244 lines not on this page · raw message
drivers/infiniband/hw/hfi1/user_exp_rcv.h +2 -1
@@ -65,7 +65,8 @@ struct tid_user_buf { };struct tid_rb_node { -struct mmu_rb_node mmu; +struct mmu_range_notifier notifier; +struct hfi1_filedata *fdata;unsigned long phys;struct tid_group *grp;u32 rcventry;
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 06/15] RDMA/hfi1: Use mmu_range_notifier_inset for user_exp_rcv
Quoted 9 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
This converts one of the two users of
mmu_notifiersto use the new API. The conversion is fairly straightforward, however the existing use of notifiers here seems to be racey.Cc Mike Marciniszyn <[email protected]>Cc Dennis Dalessandro <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
I tested v1, and replied to it [1]. I can re-test with this version if you like as well.
[1] https://marc.info/?l=linux-rdma&m=157235130606412&w=2
-Denny
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 06/15] RDMA/hfi1: Use mmu_range_notifier_inset for user_exp_rcv
Quoted 7 lines · Dennis Dalessandro
Quoted 6 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
This converts one of the two users of
mmu_notifiersto use the new API. The conversion is fairly straightforward, however the existing use of notifiers here seems to be racey.
Quoted 8 lines
Quoted 3 lines
Cc Mike Marciniszyn <[email protected]>Cc Dennis Dalessandro <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>I tested v1, and replied to it [1]. I can re-test with this version if you like as well.
I think it is fine, nothing really changed in v2, thanks
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 7/15 drm/radeon: use mmu_range_notifier_insert
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
The new API is an exact match for the needs of radeon.
For some reason radeon tries to remove overlapping ranges from the interval tree, but interval trees (and mmu_range_notifier_insert) support overlapping ranges directly. Simply delete all this code.
Since this driver is missing a invalidate_range_end callback, but still calls get_user_pages(), it cannot be correct against all races.
--- drivers/gpu/drm/radeon/radeon.h | 9 +- drivers/gpu/drm/radeon/radeon_mn.c | 219 ++++++----------------------- 2 files changed, 52 insertions(+), 176 deletions(-)
drivers/gpu/drm/radeon/radeon.h +7 -2
@@ -68,6 +68,10 @@#include <linux/hashtable.h> #include <linux/dma-fence.h> +#ifdef CONFIG_MMU_NOTIFIER +#include <linux/mmu_notifier.h> +#endif +#include <drm/ttm/ttm_bo_api.h> #include <drm/ttm/ttm_bo_driver.h> #include <drm/ttm/ttm_placement.h>
@@ -509,8 +513,9 @@ struct radeon_bo {struct ttm_bo_kmap_obj dma_buf_vmap;pid_t pid; -struct radeon_mn *mn; -struct list_head mn_list; +#ifdef CONFIG_MMU_NOTIFIER +struct mmu_range_notifier notifier; +#endif };#define gem_to_radeon_bo (gobj) container_of((gobj), struct radeon_bo, tbo.base)
drivers/gpu/drm/radeon/radeon_mn.c +45 -174
254 lines not on this page · raw message
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 07/15] drm/radeon: use mmu_range_notifier_insert
Am 28.10.19 um 21:10 schrieb Jason Gunthorpe:
Quoted 11 lines
From: Jason Gunthorpe <[email protected]>
The new API is an exact match for the needs of radeon.
For some reason radeon tries to remove overlapping ranges from the interval tree, but interval trees (and
mmu_range_notifier_insert) support overlapping ranges directly. Simply delete all this code.Since this driver is missing a
invalidate_range_endcallback, but still callsget_user_pages(), it cannot be correct against all races.
Quoted 6 lines
Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Cc Petr Cvek <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
Quoted 283 lines
--- drivers/gpu/drm/radeon/radeon.h | 9 +- drivers/gpu/drm/radeon/radeon_mn.c | 219 ++++++----------------------- 2 files changed, 52 insertions(+), 176 deletions(-)drivers/gpu/drm/radeon/radeon.h +7
-2@@ -68,6 +68,10 @@#include <linux/hashtable.h> #include <linux/dma-fence.h> +#ifdef CONFIG_MMU_NOTIFIER +#include <linux/mmu_notifier.h> +#endif +#include <drm/ttm/ttm_bo_api.h> #include <drm/ttm/ttm_bo_driver.h> #include <drm/ttm/ttm_placement.h> @@ -509,8 +513,9 @@ struct radeon_bo {struct ttm_bo_kmap_obj dma_buf_vmap;pid_t pid; -struct radeon_mn *mn; -struct list_head mn_list; +#ifdef CONFIG_MMU_NOTIFIER +struct mmu_range_notifier notifier; +#endif };#define gem_to_radeon_bo (gobj) container_of((gobj), struct radeon_bo, tbo.base)drivers/gpu/drm/radeon/radeon_mn.c +43
-167@@ -36,131 +36,51 @@#include "radeon.h" -struct radeon_mn { -struct mmu_notifier mn; - -/* objects protected by lock */ -struct mutex lock; -struct rb_root_cached objects; -}; - -struct radeon_mn_node { -struct interval_tree_node it; -struct list_head bos; -}; -/** -* radeon_mn_invalidate_range_start - callback to notify about mm change +* radeon_mn_invalidate - callback to notify about mm change * * @mn: our notifier -* @mn: the mm this callback is about -* @start: start of updated range -* @end: end of updated range +* @range: the VMA under invalidation * * We block for all BOs between start and end to be idle and * unmap them by move them into system domain again. */ -static int radeon_mn_invalidate_range_start (struct mmu_notifier *mn, -const struct mmu_notifier_range *range) +static bool radeon_mn_invalidate (struct mmu_range_notifier *mn, +const struct mmu_notifier_range *range, +unsigned long cur_seq) { - struct radeon_mn *rmn = container_of(mn, struct radeon_mn, mn); + struct radeon_bo *bo = container_of(mn, struct radeon_bo, notifier);struct ttm_operation_ctx ctx = { false, false }; -struct interval_tree_node *it; -unsigned long end; -int ret =0 ; - -/* notification is exclusive, but interval is inclusive */ - end = range->end -1 ; - -/* TODO we should be able to split locking for interval tree and -* the tear down. -*/ -if (mmu_notifier_range_blockable (range)) -mutex_lock (&rmn->lock); -else if (!mutex_trylock (&rmn->lock)) -return -EAGAIN ; - - it =interval_tree_iter_first (&rmn->objects, range->start, end); -while (it) { -struct radeon_mn_node *node; -struct radeon_bo *bo; -long r; - -if (!mmu_notifier_range_blockable (range)) { - ret = -EAGAIN ; - goto out_unlock; - } - - node = container_of(it, struct radeon_mn_node, it); - it =interval_tree_iter_next (it, range->start, end); +long r; -list_for_each_entry (bo, &node->bos, mn_list) { +if (!bo->tbo.ttm || bo->tbo.ttm->state != tt_bound) +return true; -if (!bo->tbo.ttm || bo->tbo.ttm->state != tt_bound) -continue ; +if (!mmu_notifier_range_blockable (range)) +return false; - r =radeon_bo_reserve (bo, true); -if (r) { -DRM_ERROR ("(%ld) failed to reserve user bo\n" , r); -continue ; - } - - r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, - true, false,MAX_SCHEDULE_TIMEOUT ); -if (r <=0 ) -DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); - -radeon_ttm_placement_from_domain (bo,RADEON_GEM_DOMAIN_CPU ); - r =ttm_bo_validate (&bo->tbo, &bo->placement, &ctx); -if (r) -DRM_ERROR ("(%ld) failed to validate user bo\n" , r); - -radeon_bo_unreserve (bo); - } + r =radeon_bo_reserve (bo, true); +if (r) { +DRM_ERROR ("(%ld) failed to reserve user bo\n" , r); +return true; } - -out_unlock: -mutex_unlock (&rmn->lock); - -return ret; -} - -static void radeon_mn_release (struct mmu_notifier *mn,struct mm_struct *mm) -{ -struct mmu_notifier_range range = { - .mm = mm, - .start =0 , - .end =ULONG_MAX , - .flags =0 , - .event =MMU_NOTIFY_UNMAP , - }; - -radeon_mn_invalidate_range_start (mn, &range); -} - -static struct mmu_notifier *radeon_mn_alloc_notifier (struct mm_struct *mm) -{ -struct radeon_mn *rmn; - rmn =kzalloc (sizeof (*rmn),GFP_KERNEL ); -if (!rmn) -return ERR_PTR (-ENOMEM ); + r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, true, false, +MAX_SCHEDULE_TIMEOUT ); +if (r <=0 ) +DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); -mutex_init (&rmn->lock); - rmn->objects =RB_ROOT_CACHED ; -return &rmn->mn; -} +radeon_ttm_placement_from_domain (bo,RADEON_GEM_DOMAIN_CPU ); + r =ttm_bo_validate (&bo->tbo, &bo->placement, &ctx); +if (r) +DRM_ERROR ("(%ld) failed to validate user bo\n" , r); -static void radeon_mn_free_notifier (struct mmu_notifier *mn) -{ -kfree (container_of(mn ,struct radeon_mn ,mn )); +radeon_bo_unreserve (bo); +return true; } -static const struct mmu_notifier_ops radeon_mn_ops = { - .release = radeon_mn_release, - .invalidate_range_start = radeon_mn_invalidate_range_start, - .alloc_notifier = radeon_mn_alloc_notifier, - .free_notifier = radeon_mn_free_notifier, +static const struct mmu_range_notifier_ops radeon_mn_ops = { + .invalidate = radeon_mn_invalidate, };/** @@ -174,51 +94,21 @@ static const struct mmu_notifier_ops radeon_mn_ops = {*/ int radeon_mn_register (struct radeon_bo *bo,unsigned long addr) { -unsigned long end = addr +radeon_bo_size (bo) -1 ; -struct mmu_notifier *mn; -struct radeon_mn *rmn; -struct radeon_mn_node *node =NULL ; -struct list_head bos; -struct interval_tree_node *it; - - mn =mmu_notifier_get (&radeon_mn_ops, current->mm); -if (IS_ERR (mn)) -return PTR_ERR (mn); - rmn = container_of(mn, struct radeon_mn, mn); - -INIT_LIST_HEAD (&bos); - -mutex_lock (&rmn->lock); - -while ((it =interval_tree_iter_first (&rmn->objects, addr, end))) { -kfree (node); - node = container_of(it, struct radeon_mn_node, it); -interval_tree_remove (&node->it, &rmn->objects); - addr =min (it->start, addr); - end =max (it->last, end); -list_splice (&node->bos, &bos); - } - -if (!node) { - node =kmalloc (sizeof (struct radeon_mn_node ),GFP_KERNEL ); -if (!node) { -mutex_unlock (&rmn->lock); -return -ENOMEM ; - } - } - - bo->mn = rmn; - - node->it.start = addr; - node->it.last = end; -INIT_LIST_HEAD (&node->bos); -list_splice (&bos, &node->bos); -list_add (&bo->mn_list, &node->bos); - -interval_tree_insert (&node->it, &rmn->objects); - -mutex_unlock (&rmn->lock); - +int ret; + + bo->notifier.ops = &radeon_mn_ops; + ret =mmu_range_notifier_insert (&bo->notifier, addr,radeon_bo_size (bo), + current->mm); +if (ret) +return ret; + +/* +* FIXME: radeon appears to allow get_user_pages to run during +* invalidate_range_start/end, which is not a safe way to read the +* PTEs. It should use the mmu_range_read_begin() scheme around the +* get_user_pages to ensure that the PTEs are read properly +*/ +mmu_range_read_begin (&bo->notifier);return 0 ; }@@ -231,27 +121,8 @@ int radeon_mn_register(struct radeon_bo *bo, unsigned long addr)*/ void radeon_mn_unregister (struct radeon_bo *bo) { -struct radeon_mn *rmn = bo->mn; -struct list_head *head; - -if (!rmn) +if (!bo->notifier.mm)return ; - -mutex_lock (&rmn->lock); -/* save the next list entry for later */ - head = bo->mn_list.next; - -list_del (&bo->mn_list); - -if (list_empty (head)) { -struct radeon_mn_node *node; - node = container_of(head, struct radeon_mn_node, bos); -interval_tree_remove (&node->it, &rmn->objects);
drivers/gpu/drm/radeon/radeon_mn.c int radeon_mn_register(struct radeon_bo *bo, unsigned long addr) · line 128
Quoted 10 lines
drivers/gpu/drm/radeon/radeon_mn.c +2
-7-kfree (node); - } - -mutex_unlock (&rmn->lock); - -mmu_notifier_put (&rmn->mn); - bo->mn =NULL ; +mmu_range_notifier_remove (&bo->notifier); + bo->notifier.mm =NULL ; }
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH v2 8/15 xen/gntdev: Use select for DMA_SHARED_BUFFER
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
DMA_SHARED_BUFFER can not be enabled by the user (it represents a library set in the kernel). The kconfig convention is to use select for such symbols so they are turned on implicitly when the user enables a kconfig that needs them.
Otherwise the XEN_GNTDEV_DMABUF kconfig is overly difficult to enable.
--- drivers/xen/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/xen/Kconfig +2 -1
@@ -141,7 +141,8 @@ config XEN_GNTDEV config XEN_GNTDEV_DMABUF bool "Add support for dma-buf grant access device driver extension" - depends on XEN_GNTDEV && XEN_GRANT_DMA_ALLOC && DMA_SHARED_BUFFER + depends on XEN_GNTDEV && XEN_GRANT_DMA_ALLOC + select DMA_SHARED_BUFFER help Allows userspace processes and kernel modules to use Xen backed dma-buf implementation. With this extension grant references to
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 08/15] xen/gntdev: Use select for DMA_SHARED_BUFFER
Quoted 18 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
DMA_SHARED_BUFFERcan not be enabled by the user (it represents a library set in the kernel). The kconfig convention is to use select for such symbols so they are turned on implicitly when the user enables a kconfig that needs them.Otherwise the
XEN_GNTDEV_DMABUFkconfig is overly difficult to enable.Fixes 932d6562179e ("xen/gntdev: Add initial support for dma-buf UAPI")Cc Oleksandr Andrushchenko <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Juergen Gross <[email protected]>Cc Stefano Stabellini <[email protected]>Reviewed-by Juergen Gross <[email protected]>Reviewed-by Oleksandr Andrushchenko <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
Quoted 3 lines
--- drivers/xen/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Juergen/Oleksandr/Xen Maintainers:
Would you take this patch through a xen related tree? The only reason I had in this series is to make it easier to compile-test the gntdev changes.
Since it is looking like the gntdev rework might not make it this cycle it is probably best for you to take it.
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Jürgen Groß
maintainerRe: [PATCH v2 08/15] xen/gntdev: Use select for DMA_SHARED_BUFFER
Quoted 23 lines · Jason Gunthorpe
Quoted 18 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
DMA_SHARED_BUFFERcan not be enabled by the user (it represents a library set in the kernel). The kconfig convention is to use select for such symbols so they are turned on implicitly when the user enables a kconfig that needs them.Otherwise the
XEN_GNTDEV_DMABUFkconfig is overly difficult to enable.Fixes 932d6562179e ("xen/gntdev: Add initial support for dma-buf UAPI")Cc Oleksandr Andrushchenko <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Juergen Gross <[email protected]>Cc Stefano Stabellini <[email protected]>Reviewed-by Juergen Gross <[email protected]>Reviewed-by Oleksandr Andrushchenko <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Quoted 3 lines
--- drivers/xen/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 5 lines
Juergen/Oleksandr/Xen Maintainers:
Would you take this patch through a xen related tree? The only reason I had in this series is to make it easier to compile-test the gntdev changes.
Yes, I can take it for 5.5.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 08/15] xen/gntdev: Use select for DMA_SHARED_BUFFER
In reply to Jason GunthorpeQuoted 9 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
DMA_SHARED_BUFFERcan not be enabled by the user (it represents a library set in the kernel). The kconfig convention is to use select for such symbols so they are turned on implicitly when the user enables a kconfig that needs them.Otherwise the
XEN_GNTDEV_DMABUFkconfig is overly difficult to enable.
Quoted 9 lines
Fixes 932d6562179e ("xen/gntdev: Add initial support for dma-buf UAPI")Cc Oleksandr Andrushchenko <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Juergen Gross <[email protected]>Cc Stefano Stabellini <[email protected]>Reviewed-by Juergen Gross <[email protected]>Reviewed-by Oleksandr Andrushchenko <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
Applied to xen/tip.git for-linus-5.5a
Juergen
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH v2 9/15 xen/gntdev: use mmu_range_notifier_insert
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
gntdev simply wants to monitor a specific VMA for any notifier events, this can be done straightforwardly using mmu_range_notifier_insert() over the VMA's VA range.
The notifier should be attached until the original VMA is destroyed.
It is unclear if any of this is even sane, but at least a lot of duplicate code is removed.
--- drivers/xen/gntdev-common.h | 8 +- drivers/xen/gntdev.c | 180 ++++++++++-------------------------- 2 files changed, 49 insertions(+), 139 deletions(-)
drivers/xen/gntdev-common.h +1 -7
@@ -21,15 +21,8 @@ struct gntdev_dmabuf_priv;struct gntdev_priv {/* Maps with visible offsets in the file descriptor. */ struct list_head maps; -/* -* Maps that are not visible; will be freed on munmap. -* Only populated if populate_freeable_maps == 1 -*/ -struct list_head freeable_maps;/* lock protects maps and freeable_maps. */ struct mutex lock; -struct mm_struct *mm; -struct mmu_notifier mn;#ifdef CONFIG_XEN_GRANT_DMA_ALLOC /* Device for which DMA memory is allocated. */
@@ -49,6 +42,7 @@ struct gntdev_unmap_notify { };struct gntdev_grant_map { +struct mmu_range_notifier notifier;struct list_head next;struct vm_area_struct *vma;int index;
drivers/xen/gntdev.c +48 -132
278 lines not on this page · raw message
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 10 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
gntdev simply wants to monitor a specific VMA for any notifier events, this can be done straightforwardly using
mmu_range_notifier_insert()over the VMA's VA range.The notifier should be attached until the original VMA is destroyed.
It is unclear if any of this is even sane, but at least a lot of duplicate code is removed.
I didn't have a chance to look at the patch itself yet but as a heads-up --- it crashes dom0.
-boris
Quoted 324 lines
Cc Oleksandr Andrushchenko <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Juergen Gross <[email protected]>Cc Stefano Stabellini <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>--- drivers/xen/gntdev-common.h | 8 +- drivers/xen/gntdev.c | 180 ++++++++++-------------------------- 2 files changed, 49 insertions(+), 139 deletions(-)drivers/xen/gntdev-common.h +1
-7@@ -21,15 +21,8 @@ struct gntdev_dmabuf_priv;struct gntdev_priv {/* Maps with visible offsets in the file descriptor. */ struct list_head maps; -/* -* Maps that are not visible; will be freed on munmap. -* Only populated if populate_freeable_maps == 1 -*/ -struct list_head freeable_maps;/* lock protects maps and freeable_maps. */ struct mutex lock; -struct mm_struct *mm; -struct mmu_notifier mn;#ifdef CONFIG_XEN_GRANT_DMA_ALLOC /* Device for which DMA memory is allocated. */ @@ -49,6 +42,7 @@ struct gntdev_unmap_notify { };struct gntdev_grant_map { +struct mmu_range_notifier notifier;struct list_head next;struct vm_area_struct *vma;int index;drivers/xen/gntdev.c +44
-131@@ -65,7 +65,6 @@ MODULE_PARM_DESC(limit, "Maximum number of grants that may be mapped by "static atomic_t pages_mapped =ATOMIC_INIT (0 );static int use_ptemod; -#define populate_freeable_maps use_ptemodstatic int unmap_grant_pages (struct gntdev_grant_map *map,int offset,int pages);@@ -251,12 +250,6 @@ void gntdev_put_map(struct gntdev_priv *priv, struct gntdev_grant_map *map)evtchn_put (map->notify.event); } -if (populate_freeable_maps && priv) { -mutex_lock (&priv->lock); -list_del (&map->next); -mutex_unlock (&priv->lock); - } -if (map->pages && !use_ptemod)unmap_grant_pages (map,0 , map->count);gntdev_free_map (map);@@ -445,17 +438,9 @@ static void gntdev_vma_close(struct vm_area_struct *vma)struct gntdev_priv *priv = file->private_data;pr_debug ("gntdev_vma_close %p\n" , vma); -if (use_ptemod) { -/* It is possible that an mmu notifier could be running -* concurrently, so take priv->lock to ensure that the vma won't -* vanishing during the unmap_grant_pages call, since we will -* spin here until that completes. Such a concurrent call will -* not do any unmapping, since that has been done prior to -* closing the vma, but it may still iterate the unmap_ops list. -*/ -mutex_lock (&priv->lock); +if (use_ptemod && map->vma == vma) { +mmu_range_notifier_remove (&map->notifier); map->vma =NULL ; -mutex_unlock (&priv->lock); } vma->vm_private_data =NULL ;gntdev_put_map (priv, map);@@ -477,109 +462,44 @@ static const struct vm_operations_struct gntdev_vmops = {/* ------------------------------------------------------------------ */ -static bool in_range (struct gntdev_grant_map *map, -unsigned long start,unsigned long end) -{ -if (!map->vma) -return false; -if (map->vma->vm_start >= end) -return false; -if (map->vma->vm_end <= start) -return false; - -return true; -} - -static int unmap_if_in_range (struct gntdev_grant_map *map, -unsigned long start,unsigned long end, -bool blockable) +static bool gntdev_invalidate (struct mmu_range_notifier *mn, +const struct mmu_notifier_range *range, +unsigned long cur_seq) { +struct gntdev_grant_map *map = + container_of(mn, struct gntdev_grant_map, notifier);unsigned long mstart, mend;int err; -if (!in_range (map, start, end)) -return 0 ; +if (!mmu_notifier_range_blockable (range)) +return false; -if (!blockable) -return -EAGAIN ; +/* +* If the VMA is split or otherwise changed the notifier is not +* updated, but we don't want to process VA's outside the modified +* VMA. FIXME: It would be much more understandable to just prevent +* modifying the VMA in the first place. +*/ +if (map->vma->vm_start >= range->end || + map->vma->vm_end <= range->start) +return true; - mstart =max (start, map->vma->vm_start); - mend =min (end, map->vma->vm_end); + mstart =max (range->start, map->vma->vm_start); + mend =min (range->end, map->vma->vm_end);pr_debug ("map %d+%d (%lx %lx), range %lx %lx, mrange %lx %lx\n" , map->index, map->count, map->vma->vm_start, map->vma->vm_end, - start, end, mstart, mend); + range->start, range->end, mstart, mend); err =unmap_grant_pages (map, (mstart - map->vma->vm_start) >>PAGE_SHIFT , (mend - mstart) >>PAGE_SHIFT );WARN_ON (err); -return 0 ; -} - -static int mn_invl_range_start (struct mmu_notifier *mn, -const struct mmu_notifier_range *range) -{ - struct gntdev_priv *priv = container_of(mn, struct gntdev_priv, mn); -struct gntdev_grant_map *map; -int ret =0 ; - -if (mmu_notifier_range_blockable (range)) -mutex_lock (&priv->lock); -else if (!mutex_trylock (&priv->lock)) -return -EAGAIN ; - -list_for_each_entry (map, &priv->maps, next) { - ret =unmap_if_in_range (map, range->start, range->end, -mmu_notifier_range_blockable (range)); -if (ret) - goto out_unlock; - } -list_for_each_entry (map, &priv->freeable_maps, next) { - ret =unmap_if_in_range (map, range->start, range->end, -mmu_notifier_range_blockable (range)); -if (ret) - goto out_unlock; - } - -out_unlock: -mutex_unlock (&priv->lock); - -return ret; -} - -static void mn_release (struct mmu_notifier *mn, -struct mm_struct *mm) -{ - struct gntdev_priv *priv = container_of(mn, struct gntdev_priv, mn); -struct gntdev_grant_map *map; -int err; - -mutex_lock (&priv->lock); -list_for_each_entry (map, &priv->maps, next) { -if (!map->vma) -continue ; -pr_debug ("map %d+%d (%lx %lx)\n" , - map->index, map->count, - map->vma->vm_start, map->vma->vm_end); - err =unmap_grant_pages (map,/* offset */ 0 , map->count); -WARN_ON (err); - } -list_for_each_entry (map, &priv->freeable_maps, next) { -if (!map->vma) -continue ; -pr_debug ("map %d+%d (%lx %lx)\n" , - map->index, map->count, - map->vma->vm_start, map->vma->vm_end); - err =unmap_grant_pages (map,/* offset */ 0 , map->count); -WARN_ON (err); - } -mutex_unlock (&priv->lock); +return true; } -static const struct mmu_notifier_ops gntdev_mmu_ops = { - .release = mn_release, - .invalidate_range_start = mn_invl_range_start, +static const struct mmu_range_notifier_ops gntdev_mmu_ops = { + .invalidate = gntdev_invalidate, };/* ------------------------------------------------------------------ */ @@ -594,7 +514,6 @@ static int gntdev_open(struct inode *inode, struct file *flip)return -ENOMEM ;INIT_LIST_HEAD (&priv->maps); -INIT_LIST_HEAD (&priv->freeable_maps);mutex_init (&priv->lock);#ifdef CONFIG_XEN_GNTDEV_DMABUF @@ -606,17 +525,6 @@ static int gntdev_open(struct inode *inode, struct file *flip) }#endif -if (use_ptemod) { - priv->mm =get_task_mm (current); -if (!priv->mm) { -kfree (priv); -return -ENOMEM ; - } - priv->mn.ops = &gntdev_mmu_ops; - ret =mmu_notifier_register (&priv->mn, priv->mm); -mmput (priv->mm); - } -if (ret) {kfree (priv);return ret;@@ -653,16 +561,12 @@ static int gntdev_release(struct inode *inode, struct file *flip)list_del (&map->next);gntdev_put_map (NULL /* already removed */ , map); } -WARN_ON (!list_empty (&priv->freeable_maps));mutex_unlock (&priv->lock);#ifdef CONFIG_XEN_GNTDEV_DMABUF gntdev_dmabuf_fini (priv->dmabuf_priv);#endif -if (use_ptemod) -mmu_notifier_unregister (&priv->mn, priv->mm); -kfree (priv);return 0 ; }@@ -723,8 +627,6 @@ static long gntdev_ioctl_unmap_grant_ref(struct gntdev_priv *priv, map =gntdev_find_map_index (priv, op.index >>PAGE_SHIFT , op.count);if (map) {list_del (&map->next); -if (populate_freeable_maps) -list_add_tail (&map->next, &priv->freeable_maps); err =0 ; }mutex_unlock (&priv->lock);@@ -1096,11 +998,6 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma) goto unlock_out;if (use_ptemod && map->vma) goto unlock_out; -if (use_ptemod && priv->mm != vma->vm_mm) { -pr_warn ("Huh? Other mm?\n" ); - goto unlock_out; - } -refcount_inc (&map->users); vma->vm_ops = &gntdev_vmops;@@ -1111,10 +1008,6 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma) vma->vm_flags |=VM_DONTCOPY ; vma->vm_private_data = map; - -if (use_ptemod) - map->vma = vma; -if (map->flags) {if ((vma->vm_flags &VM_WRITE ) && (map->flags & GNTMAP_readonly))@@ -1125,8 +1018,28 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma) map->flags |= GNTMAP_readonly; } +if (use_ptemod) { + map->vma = vma; + err =mmu_range_notifier_insert_locked ( + &map->notifier, vma->vm_start, + vma->vm_end - vma->vm_start, vma->vm_mm); +if (err) + goto out_unlock_put; + }mutex_unlock (&priv->lock); +/* +* gntdev takes the address of the PTE in find_grant_ptes() and passes +* it to the hypervisor in gntdev_map_grant_pages(). The purpose of +* the notifier is to prevent the hypervisor pointer to the PTE from +* going stale. +* +* Since this vma's mappings can't be touched without the mmap_sem, +* and we are holding it now, there is no need for the notifier_range +* locking pattern. +*/ +mmu_range_read_begin (&map->notifier); +if (use_ptemod) { map->pages_vm_start = vma->vm_start; err = apply_to_page_range(vma->vm_mm, vma->vm_start,@@ -1175,8 +1088,11 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma)mutex_unlock (&priv->lock); out_put_map:
drivers/xen/gntdev.c static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma) · line 1098
Quoted 10 lines
drivers/xen/gntdev.c +4
-1if (use_ptemod) { - map->vma =NULL ;unmap_grant_pages (map,0 , map->count); +if (map->vma) { +mmu_range_notifier_remove (&map->notifier); + map->vma =NULL ; + } }gntdev_put_map (priv, map);return err;
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 7 lines · Boris Ostrovsky
Quoted 6 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
gntdev simply wants to monitor a specific VMA for any notifier events, this can be done straightforwardly using
mmu_range_notifier_insert()over the VMA's VA range.
Quoted 7 lines
Quoted 4 lines
The notifier should be attached until the original VMA is destroyed.
It is unclear if any of this is even sane, but at least a lot of duplicate code is removed.
I didn't have a chance to look at the patch itself yet but as a heads-up --- it crashes dom0.
Thanks Boris. I spent a bit of time and got a VM running with a xen 4.9 hypervisor and a kernel with this patch series. It a ubuntu bionic VM with the distro's xen stuff.
Can you give some guidance how you made it crash? I see the VM autoloaded gntdev:
Module Size Used by xen_gntdev 24576 2 xen_evtchn 16384 1 xenfs 16384 1 xen_privcmd 24576 16 xenfs
And lsof says several xen processes have the chardev open:
xenstored 819 root 13u CHR 10,53 0t0 19595 /dev/xen/gntdev xenconsol 857 root 8u CHR 10,53 0t0 19595 /dev/xen/gntdev xenconsol 857 860 root 8u CHR 10,53 0t0 19595 /dev/xen/gntdev
But no crashing..
However, I wasn't able to get my usual debug kernel .config to boot with the xen hypervisor, it crashes on early boot with:
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM on 1 nodes using 8 CPUs
(XEN) .done.
(XEN) Initial low memory virq threshold set at 0x1000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 468kB init memory
(XEN) d0v0 Unhandled page fault fault/trap [#14, ec=0002]
(XEN) Pagetable walk from fffffbfff0480fbe:
(XEN) L4[0x1f7] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S: fault at ffff82d080348a06 entry.o#create_bounce_frame+0x135/0x15f (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.9.2 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff82b9f731>]
(XEN) RFLAGS: 0000000000000296 EM: 1 CONTEXT: pv guest (d0v0)
(XEN) rax: fffffbfff0480fbe rbx: 0000000000000000 rcx: 00000000c0000101 (XEN) rdx: 00000000ffffffff rsi: ffffffff84026000 rdi: ffffffff82cb4a20 (XEN) rbp: ffffffff82407ff8 rsp: ffffffff82407da0 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 1ffffffff0480fbe r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000003506e0 (XEN) cr3: 0000000034027000 cr2: fffffbfff0480fbe (XEN) fsb: 0000000000000000 gsb: ffffffff82b61000 gss: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
Which is surely some .config issue, but I didn't figure out what.
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 12 lines · Jason Gunthorpe
Quoted 11 lines · Boris Ostrovsky
Quoted 10 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
gntdev simply wants to monitor a specific VMA for any notifier events, this can be done straightforwardly using
mmu_range_notifier_insert()over the VMA's VA range.The notifier should be attached until the original VMA is destroyed.
It is unclear if any of this is even sane, but at least a lot of duplicate code is removed.
Quoted 7 lines
Quoted 2 lines
I didn't have a chance to look at the patch itself yet but as a heads-up --- it crashes dom0.
Thanks Boris. I spent a bit of time and got a VM running with a xen 4.9 hypervisor and a kernel with this patch series. It a ubuntu bionic VM with the distro's xen stuff.
Can you give some guidance how you made it crash?
It crashes trying to dereference mrn->ops->invalidate in mn_itree_invalidate() when a guest exits.
I don't think you've initialized notifier ops. I don't see you using gntdev_mmu_ops anywhere.
-boris
Quoted 45 lines
I see the VM
autoloaded gntdev:Module Size Used by xen_gntdev 24576 2 xen_evtchn 16384 1 xenfs 16384 1 xen_privcmd 24576 16 xenfsAnd lsof says several xen processes have the chardev open:
xenstored 819 root 13u CHR 10,53 0t0 19595 /dev/xen/gntdev xenconsol 857 root 8u CHR 10,53 0t0 19595 /dev/xen/gntdev xenconsol 857 860 root 8u CHR 10,53 0t0 19595 /dev/xen/gntdevBut no crashing..
However, I wasn't able to get my usual debug kernel .config to boot with the xen hypervisor, it crashes on early boot with:
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM on 1 nodes using 8 CPUs
(XEN) .done.
(XEN) Initial low memory virq threshold set at0x1000pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 468kB init memory
(XEN) d0v0 Unhandled page fault fault/trap [#14, ec=0002]
(XEN) Pagetable walk fromfffffbfff0480fbe:
(XEN) L4[0x1f7] = 0000000000000000 ffffffffffffffff
(XEN)domain_crash_synccalled fromentry.S: fault atffff82d080348a06entry.o#create_bounce_frame+0x135/0x15f (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.9.2x86_64debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff82b9f731>]
(XEN) RFLAGS: 0000000000000296 EM: 1 CONTEXT: pv guest (d0v0)(XEN) rax: fffffbfff0480fbe rbx: 0000000000000000 rcx: 00000000c0000101 (XEN) rdx: 00000000ffffffff rsi: ffffffff84026000 rdi: ffffffff82cb4a20 (XEN) rbp: ffffffff82407ff8 rsp: ffffffff82407da0 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 1ffffffff0480fbe r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000003506e0 (XEN) cr3: 0000000034027000 cr2: fffffbfff0480fbe (XEN) fsb: 0000000000000000 gsb: ffffffff82b61000 gss: 0000000000000000
Quoted 5 lines
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
Which is surely some .config issue, but I didn't figure out what.
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 18 lines · Boris Ostrovsky
Quoted 17 lines · Jason Gunthorpe
Quoted 7 lines · Boris Ostrovsky
Quoted 6 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
gntdev simply wants to monitor a specific VMA for any notifier events, this can be done straightforwardly using
mmu_range_notifier_insert()over the VMA's VA range.Quoted 5 lines
Quoted 4 lines
The notifier should be attached until the original VMA is destroyed.
It is unclear if any of this is even sane, but at least a lot of duplicate code is removed.
I didn't have a chance to look at the patch itself yet but as a heads-up
Thanks Boris. I spent a bit of time and got a VM running with a xen 4.9 hypervisor and a kernel with this patch series. It a ubuntu bionic VM with the distro's xen stuff.
Quoted 7 lines
Quoted 1 line
Can you give some guidance how you made it crash?
It crashes trying to dereference
mrn->ops->invalidateinmn_itree_invalidate()when a guest exits.I don't think you've initialized notifier ops. I don't see you using
gntdev_mmu_opsanywhere.
So weird the compiler didn't complain about an unused static...
But yes, this is a mistake, it should be:
drivers/xen/gntdev.c +1 -0
@@ -1011,6 +1011,7 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma)if (use_ptemod) { map->vma = vma; + map->notifier.ops = &gntdev_mmu_ops; err =mmu_range_notifier_insert_locked ( &map->notifier, vma->vm_start, vma->vm_end - vma->vm_start, vma->vm_mm);
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
In reply to Jason GunthorpeQuoted 4 lines · Jason Gunthorpe
drivers/xen/gntdev.c +0
-0@@ -445,17 +438,9 @@ static void gntdev_vma_close(struct vm_area_struct *vma)struct gntdev_priv *priv = file->private_data;pr_debug ("gntdev_vma_close %p\n" , vma);
drivers/xen/gntdev.c static void gntdev_vma_close(struct vm_area_struct *vma) · line 441
Quoted 10 lines
drivers/xen/gntdev.c +1
-9-if (use_ptemod) { -/* It is possible that an mmu notifier could be running -* concurrently, so take priv->lock to ensure that the vma won't -* vanishing during the unmap_grant_pages call, since we will -* spin here until that completes. Such a concurrent call will -* not do any unmapping, since that has been done prior to -* closing the vma, but it may still iterate the unmap_ops list. -*/ -mutex_lock (&priv->lock); +if (use_ptemod && map->vma == vma) {
Is it possible for map->vma not to be equal to vma?
-boris
drivers/xen/gntdev.c static void gntdev_vma_close(struct vm_area_struct *vma) · line 446
Quoted 7 lines
drivers/xen/gntdev.c +1
-1+mmu_range_notifier_remove (&map->notifier); map->vma =NULL ; -mutex_unlock (&priv->lock); } vma->vm_private_data =NULL ;gntdev_put_map (priv, map);
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 4 lines · Boris Ostrovsky
Quoted 3 lines · Jason Gunthorpe
drivers/xen/gntdev.c +0
-0@@ -445,17 +438,9 @@ static void gntdev_vma_close(struct vm_area_struct *vma)struct gntdev_priv *priv = file->private_data;
Quoted 14 lines
drivers/xen/gntdev.c static void gntdev_vma_close(struct vm_area_struct *vma) · line 441
Quoted 11 lines
drivers/xen/gntdev.c +1
-9pr_debug ("gntdev_vma_close %p\n" , vma); -if (use_ptemod) { -/* It is possible that an mmu notifier could be running -* concurrently, so take priv->lock to ensure that the vma won't -* vanishing during the unmap_grant_pages call, since we will -* spin here until that completes. Such a concurrent call will -* not do any unmapping, since that has been done prior to -* closing the vma, but it may still iterate the unmap_ops list. -*/ -mutex_lock (&priv->lock); +if (use_ptemod && map->vma == vma) {Is it possible for
map->vmanot to be equal to vma?
It could be NULL at least if use_ptemod is not set.
Otherwise, I'm not sure, the confusing bit is that the map comes from here:
map = gntdev_find_map_index(priv, index, count);
It looks like the intent is that the map->vma is always set to the only vma that has the map as private_data.
So, I suppose it can be relaxed to a null test and a WARN_ON that it hasn't changed?
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 20 lines · Jason Gunthorpe
Quoted 4 lines · Boris Ostrovsky
Quoted 3 lines · Jason Gunthorpe
drivers/xen/gntdev.c +0
-0@@ -445,17 +438,9 @@ static void gntdev_vma_close(struct vm_area_struct *vma)struct gntdev_priv *priv = file->private_data;Quoted 13 lines
drivers/xen/gntdev.c static void gntdev_vma_close(struct vm_area_struct *vma) · line 441
Quoted 11 lines
drivers/xen/gntdev.c +1
-9pr_debug ("gntdev_vma_close %p\n" , vma); -if (use_ptemod) { -/* It is possible that an mmu notifier could be running -* concurrently, so take priv->lock to ensure that the vma won't -* vanishing during the unmap_grant_pages call, since we will -* spin here until that completes. Such a concurrent call will -* not do any unmapping, since that has been done prior to -* closing the vma, but it may still iterate the unmap_ops list. -*/ -mutex_lock (&priv->lock); +if (use_ptemod && map->vma == vma) {Is it possible for
map->vmanot to be equal to vma?It could be NULL at least if
use_ptemodis not set.
Quoted 7 lines
Otherwise, I'm not sure, the confusing bit is that the map comes from here:
map = gntdev_find_map_index(priv, index, count);It looks like the intent is that the
map->vmais always set to the only vma that has the map asprivate_data.
I am not sure how this can work otherwise. We stash map pointer in vm's vm_private_data and vice versa (for use_ptemod) gntdev_mmap() so if they have to match.
That's why I was asking you to see if you had something particular in mind when you added this test.
Quoted 2 lines
So, I suppose it can be relaxed to a null test and a
WARN_ONthat it hasn't changed?
You mean
if (use_ptemod) {
WARN_ON(map->vma != vma);
...
Yes, that sounds good.
-boris
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 11 lines · Boris Ostrovsky
Quoted 2 lines
So, I suppose it can be relaxed to a null test and a
WARN_ONthat it hasn't changed?You mean
if (
use_ptemod) {
WARN_ON(map->vma != vma);
...Yes, that sounds good.
I amended my copy of the patch with the above, has this rework shown signs of working?
@@ -436,7 +436,8 @@ static void gntdev_vma_close(struct vm_area_struct *vma)
struct gntdev_priv *priv = file->private_data;
pr_debug("gntdev_vma_close %p\n", vma);
- if (use_ptemod && map->vma == vma) {
+ if (use_ptemod) {
+ WARN_ON(map->vma != vma);
mmu_range_notifier_remove(&map->notifier);
map->vma = NULL;
}Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Boris Ostrovsky
Reviewed-byRe: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 6 lines · Jason Gunthorpe
Quoted 4 lines · Boris Ostrovsky
Quoted 2 lines
So, I suppose it can be relaxed to a null test and a
WARN_ONthat it hasn't changed?You mean
Quoted 8 lines
Quoted 6 lines
if (
use_ptemod) {
WARN_ON(map->vma != vma);
...Yes, that sounds good.
I amended my copy of the patch with the above, has this rework shown signs of working?
Yes, it works fine.
But please don't forget notifier ops initialization.
With those two changes,
Quoted 12 lines
drivers/xen/gntdev.c +2
-1@@ -436,7 +436,8 @@ static void gntdev_vma_close(struct vm_area_struct *vma)struct gntdev_priv *priv = file->private_data;pr_debug ("gntdev_vma_close %p\n" , vma); -if (use_ptemod && map->vma == vma) { +if (use_ptemod) { +WARN_ON (map->vma != vma);mmu_range_notifier_remove (&map->notifier); map->vma =NULL ; }
Quoted 1 line
Jason
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 09/15] xen/gntdev: use mmu_range_notifier_insert
Quoted 16 lines · Boris Ostrovsky
Quoted 6 lines · Jason Gunthorpe
Quoted 4 lines · Boris Ostrovsky
Quoted 2 lines
So, I suppose it can be relaxed to a null test and a
WARN_ONthat it hasn't changed?You mean
Quoted 8 lines
Quoted 6 lines
if (
use_ptemod) {
WARN_ON(map->vma != vma);
...Yes, that sounds good.
I amended my copy of the patch with the above, has this rework shown signs of working?
Quoted 7 lines
Yes, it works fine.
But please don't forget notifier ops initialization.
With those two changes,
Reviewed-by Boris Ostrovsky <[email protected]>
Thanks, I got both things. I'll forward this toward linux-next and repost a v3
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 10/15 nouveau: use mmu_notifier directly for invalidate_range_start
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
There is no reason to get the invalidate_range_start() callback via an indirection through hmm_mirror, just register a normal notifier directly.
--- drivers/gpu/drm/nouveau/nouveau_svm.c | 95 ++++++++++++++++++--------- 1 file changed, 63 insertions(+), 32 deletions(-)
drivers/gpu/drm/nouveau/nouveau_svm.c +63 -32
@@ -88,6 +88,7 @@ nouveau_ivmm_find(struct nouveau_svm *svm, u64 inst) }struct nouveau_svmm { +struct mmu_notifier notifier;struct nouveau_vmm *vmm;struct {unsigned long start;
@@ -96,7 +97,6 @@ struct nouveau_svmm {struct mutex mutex; -struct mm_struct *mm;struct hmm_mirror mirror; };
@@ -251,10 +251,11 @@ nouveau_svmm_invalidate(struct nouveau_svmm *svmm, u64 start, u64 limit) }static int -nouveau_svmm_sync_cpu_device_pagetables (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) +nouveau_svmm_invalidate_range_start (struct mmu_notifier *mn, +const struct mmu_notifier_range *update) { -struct nouveau_svmm *svmm =container_of (mirror,typeof (*svmm), mirror); +struct nouveau_svmm *svmm = + container_of(mn, struct nouveau_svmm, notifier);unsigned long start = update->start;unsigned long limit = update->end;
@@ -264,6 +265,9 @@ nouveau_svmm_sync_cpu_device_pagetables(struct hmm_mirror *mirror,SVMM_DBG (svmm,"invalidate %016lx-%016lx" , start, limit);mutex_lock (&svmm->mutex); +if (unlikely (!svmm->vmm)) + goto out; +if (limit > svmm->unmanaged.start && start < svmm->unmanaged.limit) {if (start < svmm->unmanaged.start) { nouveau_svmm_invalidate(svmm, start,
@@ -273,19 +277,31 @@ nouveau_svmm_sync_cpu_device_pagetables(struct hmm_mirror *mirror, }nouveau_svmm_invalidate (svmm, start, limit); + +out:mutex_unlock (&svmm->mutex);return 0 ; } -static void -nouveau_svmm_release (struct hmm_mirror *mirror) +static void nouveau_svmm_free_notifier (struct mmu_notifier *mn) +{ +kfree (container_of(mn ,struct nouveau_svmm ,notifier )); +} + +static const struct mmu_notifier_ops nouveau_mn_ops = { + .invalidate_range_start = nouveau_svmm_invalidate_range_start, + .free_notifier = nouveau_svmm_free_notifier, +}; + +static int +nouveau_svmm_sync_cpu_device_pagetables (struct hmm_mirror *mirror, +const struct mmu_notifier_range *update) { +return 0 ; } -static const struct hmm_mirror_ops -nouveau_svmm = { +static const struct hmm_mirror_ops nouveau_svmm = { .sync_cpu_device_pagetables = nouveau_svmm_sync_cpu_device_pagetables, - .release = nouveau_svmm_release, };void
@@ -294,7 +310,10 @@ nouveau_svmm_fini(struct nouveau_svmm **psvmm)struct nouveau_svmm *svmm = *psvmm;if (svmm) {hmm_mirror_unregister (&svmm->mirror); -kfree (*psvmm); +mutex_lock (&svmm->mutex); + svmm->vmm =NULL ; +mutex_unlock (&svmm->mutex); +mmu_notifier_put (&svmm->notifier); *psvmm =NULL ; } }
@@ -320,7 +339,7 @@ nouveau_svmm_init(struct drm_device *dev, void *data,mutex_lock (&cli->mutex);if (cli->svm.cli) { ret = -EBUSY ; - goto done; + goto out_free; }/* Allocate a new GPU VMM that can support SVM (managed by the
@@ -335,24 +354,33 @@ nouveau_svmm_init(struct drm_device *dev, void *data, .fault_replay = true, }, sizeof(struct gp100_vmm_v0), &cli->svm.vmm);if (ret) - goto done; + goto out_free; -/* Enable HMM mirroring of CPU address-space to VMM. */ - svmm->mm =get_task_mm (current); -down_write (&svmm->mm->mmap_sem); +down_write (¤t->mm->mmap_sem); svmm->mirror.ops = &nouveau_svmm; - ret =hmm_mirror_register (&svmm->mirror, svmm->mm); -if (ret ==0 ) { - cli->svm.svmm = svmm; - cli->svm.cli = cli; - } -up_write (&svmm->mm->mmap_sem); -mmput (svmm->mm); + ret =hmm_mirror_register (&svmm->mirror, current->mm); +if (ret) + goto out_mm_unlock; -done: + svmm->notifier.ops = &nouveau_mn_ops; + ret =__mmu_notifier_register (&svmm->notifier, current->mm);if (ret) -nouveau_svmm_fini (&svmm); + goto out_hmm_unregister; +/* Note, ownership of svmm transfers to mmu_notifier */ + + cli->svm.svmm = svmm; + cli->svm.cli = cli; +up_write (¤t->mm->mmap_sem);mutex_unlock (&cli->mutex); +return 0 ; + +out_hmm_unregister: +hmm_mirror_unregister (&svmm->mirror); +out_mm_unlock: +up_write (¤t->mm->mmap_sem); +out_free: +mutex_unlock (&cli->mutex); +kfree (svmm);return ret; }
@@ -494,12 +522,12 @@ nouveau_range_fault(struct nouveau_svmm *svmm, struct hmm_range *range) ret =hmm_range_register (range, &svmm->mirror);if (ret) { -up_read (&svmm->mm->mmap_sem); +up_read (&svmm->notifier.mm->mmap_sem);return (int )ret; }if (!hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT )) { -up_read (&svmm->mm->mmap_sem); +up_read (&svmm->notifier.mm->mmap_sem);return -EBUSY ; }
@@ -507,7 +535,7 @@ nouveau_range_fault(struct nouveau_svmm *svmm, struct hmm_range *range)if (ret <=0 ) {if (ret ==0 ) ret = -EBUSY ; -up_read (&svmm->mm->mmap_sem); +up_read (&svmm->notifier.mm->mmap_sem);hmm_range_unregister (range);return ret; }
@@ -587,12 +615,15 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.version =0 ;for (fi =0 ; fn = fi +1 , fi < buffer->fault_nr; fi = fn) { +struct mm_struct *mm; +/* Cancel any faults from non-SVM channels. */ if (!(svmm = buffer->fault[fi]->svmm)) {nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]);continue ; }SVMM_DBG (svmm,"addr %016llx" , buffer->fault[fi]->addr); + mm = svmm->notifier.mm;/* We try and group handling of faults within a small * window into a single update.
@@ -609,11 +640,11 @@ nouveau_svm_fault(struct nvif_notify *notify)/* Intersect fault window with the CPU VMA, cancelling * the fault if the address is invalid. */ -down_read (&svmm->mm->mmap_sem); - vma =find_vma_intersection (svmm->mm, start, limit); +down_read (&mm->mmap_sem); + vma =find_vma_intersection (mm, start, limit);if (!vma) {SVMM_ERR (svmm,"wndw %016llx-%016llx" , start, limit); -up_read (&svmm->mm->mmap_sem); +up_read (&mm->mmap_sem);nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]);continue ; }
@@ -623,7 +654,7 @@ nouveau_svm_fault(struct nvif_notify *notify)if (buffer->fault[fi]->addr != start) {SVMM_ERR (svmm,"addr %016llx" , buffer->fault[fi]->addr); -up_read (&svmm->mm->mmap_sem); +up_read (&mm->mmap_sem);nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]);continue ; }
@@ -704,7 +735,7 @@ nouveau_svm_fault(struct nvif_notify *notify) NULL); svmm->vmm->vmm.object.client->super = false;mutex_unlock (&svmm->mutex); -up_read (&svmm->mm->mmap_sem); +up_read (&mm->mmap_sem); }/* Cancel any faults in the window whose pages didn't manage
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 11/15 nouveau: use mmu_range_notifier instead of hmm_mirror
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
Remove the hmm_mirror object and use the mmu_range_notifier API instead for the range, and use the normal mmu_notifier API for the general invalidation callback.
While here re-organize the pagefault path so the locking pattern is clear.
nouveau is the only driver that uses a temporary range object and instead forwards nearly every invalidation range directly to the HW. While this is not how the mmu_range_notifier was intended to be used, the overheads on the pagefaulting path are similar to the existing hmm_mirror version. Particularly since the interval tree will be small.
--- drivers/gpu/drm/nouveau/nouveau_svm.c | 180 ++++++++++++++------------ 1 file changed, 100 insertions(+), 80 deletions(-)
drivers/gpu/drm/nouveau/nouveau_svm.c +100 -80
@@ -96,8 +96,6 @@ struct nouveau_svmm { } unmanaged;struct mutex mutex; - -struct hmm_mirror mirror; }; #define SVMM_DBG(s,f,a...) \
@@ -293,23 +291,11 @@ static const struct mmu_notifier_ops nouveau_mn_ops = { .free_notifier = nouveau_svmm_free_notifier, }; -static int -nouveau_svmm_sync_cpu_device_pagetables (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) -{ -return 0 ; -} - -static const struct hmm_mirror_ops nouveau_svmm = { - .sync_cpu_device_pagetables = nouveau_svmm_sync_cpu_device_pagetables, -}; -void nouveau_svmm_fini (struct nouveau_svmm **psvmm) {struct nouveau_svmm *svmm = *psvmm;if (svmm) { -hmm_mirror_unregister (&svmm->mirror);mutex_lock (&svmm->mutex); svmm->vmm =NULL ;mutex_unlock (&svmm->mutex);
@@ -357,15 +343,10 @@ nouveau_svmm_init(struct drm_device *dev, void *data, goto out_free;down_write (¤t->mm->mmap_sem); - svmm->mirror.ops = &nouveau_svmm; - ret =hmm_mirror_register (&svmm->mirror, current->mm); -if (ret) - goto out_mm_unlock; - svmm->notifier.ops = &nouveau_mn_ops; ret =__mmu_notifier_register (&svmm->notifier, current->mm);if (ret) - goto out_hmm_unregister; + goto out_mm_unlock;/* Note, ownership of svmm transfers to mmu_notifier */ cli->svm.svmm = svmm;
@@ -374,8 +355,6 @@ nouveau_svmm_init(struct drm_device *dev, void *data,mutex_unlock (&cli->mutex);return 0 ; -out_hmm_unregister: -hmm_mirror_unregister (&svmm->mirror); out_mm_unlock:up_write (¤t->mm->mmap_sem); out_free:
@@ -503,43 +482,91 @@ nouveau_svm_fault_cache(struct nouveau_svm *svm, fault->inst, fault->addr, fault->access); } -static inline bool -nouveau_range_done (struct hmm_range *range) +struct svm_notifier { +struct mmu_range_notifier notifier; +struct nouveau_svmm *svmm; +}; + +static bool nouveau_svm_range_invalidate (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range, +unsigned long cur_seq) { -bool ret =hmm_range_valid (range); +struct svm_notifier *sn = + container_of(mrn, struct svm_notifier, notifier); -hmm_range_unregister (range); -return ret; +/* +* serializes the update to mrn->invalidate_seq done by caller and +* prevents invalidation of the PTE from progressing while HW is being +* programmed. This is very hacky and only works because the normal +* notifier that does invalidation is always called after the range +* notifier. +*/ +if (mmu_notifier_range_blockable (range)) +mutex_lock (&sn->svmm->mutex); +else if (!mutex_trylock (&sn->svmm->mutex)) +return false; +mmu_range_set_seq (mrn, cur_seq); +mutex_unlock (&sn->svmm->mutex); +return true; } -static int -nouveau_range_fault (struct nouveau_svmm *svmm,struct hmm_range *range) +static const struct mmu_range_notifier_ops nouveau_svm_mrn_ops = { + .invalidate = nouveau_svm_range_invalidate, +}; + +static int nouveau_range_fault (struct nouveau_svmm *svmm, +struct nouveau_drm *drm,void *data,u32 size, +u64 *pfns, +struct svm_notifier *notifier) { +unsigned long timeout = + jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT ); +/* Have HMM fault pages within the fault window to the GPU. */ +struct hmm_range range = { + .notifier = ¬ifier->notifier, + .start = notifier->notifier.interval_tree.start, + .end = notifier->notifier.interval_tree.last +1 , + .pfns = pfns, + .flags = nouveau_svm_pfn_flags, + .values = nouveau_svm_pfn_values, + .pfn_shift =NVIF_VMM_PFNMAP_V0_ADDR_SHIFT , + }; +struct mm_struct *mm = notifier->notifier.mm;long ret; - range->default_flags =0 ; - range->pfn_flags_mask =-1UL ; +while (true) { +if (time_after (jiffies, timeout)) +return -EBUSY ; - ret =hmm_range_register (range, &svmm->mirror); -if (ret) { -up_read (&svmm->notifier.mm->mmap_sem); -return (int )ret; - } + range.notifier_seq =mmu_range_read_begin (range.notifier); + range.default_flags =0 ; + range.pfn_flags_mask =-1UL ; +down_read (&mm->mmap_sem); + ret =hmm_range_fault (&range,0 ); +up_read (&mm->mmap_sem); +if (ret <=0 ) { +if (ret ==0 || ret == -EBUSY ) +continue ; +return ret; + } -if (!hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT )) { -up_read (&svmm->notifier.mm->mmap_sem); -return -EBUSY ; +mutex_lock (&svmm->mutex); +if (mmu_range_read_retry (range.notifier, + range.notifier_seq)) { +mutex_unlock (&svmm->mutex); +continue ; + } +break ; } - ret =hmm_range_fault (range,0 ); -if (ret <=0 ) { -if (ret ==0 ) - ret = -EBUSY ; -up_read (&svmm->notifier.mm->mmap_sem); -hmm_range_unregister (range); -return ret; - } -return 0 ; +nouveau_dmem_convert_pfn (drm, &range); + + svmm->vmm->vmm.object.client->super = true; + ret =nvif_object_ioctl (&svmm->vmm->vmm.object, data, size,NULL ); + svmm->vmm->vmm.object.client->super = false; +mutex_unlock (&svmm->mutex); + +return ret; } static int
@@ -559,7 +586,6 @@ nouveau_svm_fault(struct nvif_notify *notify) } i;u64 phys[16 ]; } args; -struct hmm_range range;struct vm_area_struct *vma;u64 inst, start, limit;int fi, fn, pi, fill;
@@ -615,6 +641,7 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.version =0 ;for (fi =0 ; fn = fi +1 , fi < buffer->fault_nr; fi = fn) { +struct svm_notifier notifier;struct mm_struct *mm;/* Cancel any faults from non-SVM channels. */
@@ -623,7 +650,6 @@ nouveau_svm_fault(struct nvif_notify *notify)continue ; }SVMM_DBG (svmm,"addr %016llx" , buffer->fault[fi]->addr); - mm = svmm->notifier.mm;/* We try and group handling of faults within a small * window into a single update.
@@ -637,6 +663,12 @@ nouveau_svm_fault(struct nvif_notify *notify) start =max_t (u64, start, svmm->unmanaged.limit);SVMM_DBG (svmm,"wndw %016llx-%016llx" , start, limit); + mm = svmm->notifier.mm; +if (!mmget_not_zero (mm)) { +nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]); +continue ; + } +/* Intersect fault window with the CPU VMA, cancelling * the fault if the address is invalid. */
@@ -645,16 +677,18 @@ nouveau_svm_fault(struct nvif_notify *notify)if (!vma) {SVMM_ERR (svmm,"wndw %016llx-%016llx" , start, limit);up_read (&mm->mmap_sem); +mmput (mm);nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]);continue ; } start =max_t (u64, start, vma->vm_start); limit =min_t (u64, limit, vma->vm_end); +up_read (&mm->mmap_sem);SVMM_DBG (svmm,"wndw %016llx-%016llx" , start, limit);if (buffer->fault[fi]->addr != start) {SVMM_ERR (svmm,"addr %016llx" , buffer->fault[fi]->addr); -up_read (&mm->mmap_sem); +mmput (mm);nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]);continue ; }
@@ -710,33 +744,19 @@ nouveau_svm_fault(struct nvif_notify *notify) args.i.p.addr, args.i.p.addr + args.i.p.size, fn - fi); -/* Have HMM fault pages within the fault window to the GPU. */ - range.start = args.i.p.addr; - range.end = args.i.p.addr + args.i.p.size; - range.pfns = args.phys; - range.flags = nouveau_svm_pfn_flags; - range.values = nouveau_svm_pfn_values; - range.pfn_shift =NVIF_VMM_PFNMAP_V0_ADDR_SHIFT ; -again: - ret =nouveau_range_fault (svmm, &range); -if (ret ==0 ) { -mutex_lock (&svmm->mutex); -if (!nouveau_range_done (&range)) { -mutex_unlock (&svmm->mutex); - goto again; - } - -nouveau_dmem_convert_pfn (svm->drm, &range); - - svmm->vmm->vmm.object.client->super = true; - ret =nvif_object_ioctl (&svmm->vmm->vmm.object, - &args,sizeof (args.i) + - pi *sizeof (args.phys[0 ]), -NULL ); - svmm->vmm->vmm.object.client->super = false; -mutex_unlock (&svmm->mutex); -up_read (&mm->mmap_sem); + notifier.svmm = svmm; + notifier.notifier.ops = &nouveau_svm_mrn_ops; + ret =mmu_range_notifier_insert (¬ifier.notifier, + args.i.p.addr, args.i.p.size, + svmm->notifier.mm); +if (!ret) { + ret =nouveau_range_fault ( + svmm, svm->drm, &args, +sizeof (args.i) + pi *sizeof (args.phys[0 ]), + args.phys, ¬ifier); +mmu_range_notifier_remove (¬ifier.notifier); } +mmput (mm);/* Cancel any faults in the window whose pages didn't manage * to keep their valid bit, or stay writeable when required.
@@ -745,10 +765,10 @@ nouveau_svm_fault(struct nvif_notify *notify)*/ while (fi < fn) {struct nouveau_svm_fault *fault = buffer->fault[fi++]; - pi = (fault->addr - range.start) >>PAGE_SHIFT ; + pi = (fault->addr - args.i.p.addr) >>PAGE_SHIFT ;if (ret || - !(range.pfns[pi] &NVIF_VMM_PFNMAP_V0_V ) || - (!(range.pfns[pi] &NVIF_VMM_PFNMAP_V0_W ) && + !(args.phys[pi] &NVIF_VMM_PFNMAP_V0_V ) || + (!(args.phys[pi] &NVIF_VMM_PFNMAP_V0_W ) && fault->access !=0 && fault->access !=3 )) {nouveau_svm_fault_cancel_fault (svm, fault);continue ;
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 12/15 drm/amdgpu: Call find_vma under mmap_sem
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
find_vma() must be called under the mmap_sem, reorganize this code to do the vma check after entering the lock.
Further, fix the unlocked use of struct task_struct's mm, instead use the mm from hmm_mirror which has an active mm_grab. Also the mm_grab must be converted to a mm_get before acquiring mmap_sem or calling find_vma().
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +21 -16
@@ -35,6 +35,7 @@#include <linux/hmm.h> #include <linux/pagemap.h> #include <linux/sched/task.h> +#include <linux/sched/mm.h> #include <linux/seq_file.h> #include <linux/slab.h> #include <linux/swap.h>
@@ -788,7 +789,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)struct hmm_mirror *mirror = bo->mn ? &bo->mn->mirror :NULL ;struct ttm_tt *ttm = bo->tbo.ttm;struct amdgpu_ttm_tt *gtt = (void *)ttm; -struct mm_struct *mm = gtt->usertask->mm; +struct mm_struct *mm;unsigned long start = gtt->userptr;struct vm_area_struct *vma;struct hmm_range *range;
@@ -796,25 +797,14 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)uint64_t *pfns;int r =0 ; -if (!mm)/* Happens during process shutdown */ -return -ESRCH ; -if (unlikely (!mirror)) {DRM_DEBUG_DRIVER ("Failed to get hmm_mirror\n" ); - r = -EFAULT ; - goto out; +return -EFAULT ; } - vma =find_vma (mm, start); -if (unlikely (!vma || start < vma->vm_start)) { - r = -EFAULT ; - goto out; - } -if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - vma->vm_file)) { - r = -EPERM ; - goto out; - } + mm = mirror->hmm->mmu_notifier.mm; +if (!mmget_not_zero (mm))/* Happens during process shutdown */ +return -ESRCH ; range =kzalloc (sizeof (*range),GFP_KERNEL );if (unlikely (!range)) {
@@ -847,6 +837,17 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT );down_read (&mm->mmap_sem); + vma =find_vma (mm, start); +if (unlikely (!vma || start < vma->vm_start)) { + r = -EFAULT ; + goto out_unlock; + } +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vma->vm_file)) { + r = -EPERM ; + goto out_unlock; + } + r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem);
@@ -865,15 +866,19 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) } gtt->range = range; +mmput (mm);return 0 ; +out_unlock: +up_read (&mm->mmap_sem); out_free_pfns:hmm_range_unregister (range);kvfree (pfns); out_free_ranges:kfree (range); out: +mmput (mm);return r; }
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 12/15] drm/amdgpu: Call find_vma under mmap_sem
Am 28.10.19 um 21:10 schrieb Jason Gunthorpe:
Quoted 10 lines
From: Jason Gunthorpe <[email protected]>
find_vma()must be called under themmap_sem, reorganize this code to do the vma check after entering the lock.Further, fix the unlocked use of struct task_struct's mm, instead use the mm from
hmm_mirrorwhich has an activemm_grab. Also themm_grabmust be converted to amm_getbefore acquiringmmap_semor callingfind_vma().
Quoted 7 lines
Fixes 66c45500bfdc ("drm/amdgpu: use new HMM APIs and helpers")Fixes 0919195f2b0d ("drm/amdgpu: Enable amdgpu_ttm_tt_get_user_pages in worker threads")Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
Quoted 83 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +20
-16@@ -35,6 +35,7 @@#include <linux/hmm.h> #include <linux/pagemap.h> #include <linux/sched/task.h> +#include <linux/sched/mm.h> #include <linux/seq_file.h> #include <linux/slab.h> #include <linux/swap.h> @@ -788,7 +789,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)struct hmm_mirror *mirror = bo->mn ? &bo->mn->mirror :NULL ;struct ttm_tt *ttm = bo->tbo.ttm;struct amdgpu_ttm_tt *gtt = (void *)ttm; -struct mm_struct *mm = gtt->usertask->mm; +struct mm_struct *mm;unsigned long start = gtt->userptr;struct vm_area_struct *vma;struct hmm_range *range;@@ -796,25 +797,14 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)uint64_t *pfns;int r =0 ; -if (!mm)/* Happens during process shutdown */ -return -ESRCH ; -if (unlikely (!mirror)) {DRM_DEBUG_DRIVER ("Failed to get hmm_mirror\n" ); - r = -EFAULT ; - goto out; +return -EFAULT ; } - vma =find_vma (mm, start); -if (unlikely (!vma || start < vma->vm_start)) { - r = -EFAULT ; - goto out; - } -if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - vma->vm_file)) { - r = -EPERM ; - goto out; - } + mm = mirror->hmm->mmu_notifier.mm; +if (!mmget_not_zero (mm))/* Happens during process shutdown */ +return -ESRCH ; range =kzalloc (sizeof (*range),GFP_KERNEL );if (unlikely (!range)) {@@ -847,6 +837,17 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT );down_read (&mm->mmap_sem); + vma =find_vma (mm, start); +if (unlikely (!vma || start < vma->vm_start)) { + r = -EFAULT ; + goto out_unlock; + } +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vma->vm_file)) { + r = -EPERM ; + goto out_unlock; + } + r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem);@@ -865,15 +866,19 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) } gtt->range = range; +mmput (mm);return 0 ; +out_unlock: +up_read (&mm->mmap_sem);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 884
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0out_free_pfns:hmm_range_unregister (range);kvfree (pfns); out_free_ranges:kfree (range); out: +mmput (mm);return r; }
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 12/15] drm/amdgpu: Call find_vma under mmap_sem
In reply to Jason GunthorpeQuoted 10 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
find_vma()must be called under themmap_sem, reorganize this code to do the vma check after entering the lock.Further, fix the unlocked use of struct task_struct's mm, instead use the mm from
hmm_mirrorwhich has an activemm_grab. Also themm_grabmust be converted to amm_getbefore acquiringmmap_semor callingfind_vma().
Quoted 7 lines
Fixes 66c45500bfdc ("drm/amdgpu: use new HMM APIs and helpers")Fixes 0919195f2b0d ("drm/amdgpu: Enable amdgpu_ttm_tt_get_user_pages in worker threads")Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
One question inline to confirm my understanding. Otherwise this patch is
Quoted 41 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3
-8@@ -35,6 +35,7 @@#include <linux/hmm.h> #include <linux/pagemap.h> #include <linux/sched/task.h> +#include <linux/sched/mm.h> #include <linux/seq_file.h> #include <linux/slab.h> #include <linux/swap.h> @@ -788,7 +789,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)struct hmm_mirror *mirror = bo->mn ? &bo->mn->mirror :NULL ;struct ttm_tt *ttm = bo->tbo.ttm;struct amdgpu_ttm_tt *gtt = (void *)ttm; -struct mm_struct *mm = gtt->usertask->mm; +struct mm_struct *mm;unsigned long start = gtt->userptr;struct vm_area_struct *vma;struct hmm_range *range;@@ -796,25 +797,14 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)uint64_t *pfns;int r =0 ; -if (!mm)/* Happens during process shutdown */ -return -ESRCH ; -if (unlikely (!mirror)) {DRM_DEBUG_DRIVER ("Failed to get hmm_mirror\n" ); - r = -EFAULT ; - goto out; +return -EFAULT ; } - vma =find_vma (mm, start); -if (unlikely (!vma || start < vma->vm_start)) {
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 806
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-8- r = -EFAULT ; - goto out; - } -if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - vma->vm_file)) { - r = -EPERM ; - goto out; - } + mm = mirror->hmm->mmu_notifier.mm; +if (!mmget_not_zero (mm))/* Happens during process shutdown */
This works because mirror->hmm->mmu_notifier holds an mmgrab reference to the mm? So the MM will not just go away, but if the mmget refcount is 0, it means the mm is marked for destruction and shouldn't be used any more.
Quoted 32 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15
-0+return -ESRCH ; range =kzalloc (sizeof (*range),GFP_KERNEL );if (unlikely (!range)) {@@ -847,6 +837,17 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT );down_read (&mm->mmap_sem); + vma =find_vma (mm, start); +if (unlikely (!vma || start < vma->vm_start)) { + r = -EFAULT ; + goto out_unlock; + } +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vma->vm_file)) { + r = -EPERM ; + goto out_unlock; + } + r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem);@@ -865,15 +866,19 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) } gtt->range = range; +mmput (mm);return 0 ; +out_unlock: +up_read (&mm->mmap_sem);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 884
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0out_free_pfns:hmm_range_unregister (range);kvfree (pfns); out_free_ranges:kfree (range); out: +mmput (mm);return r; }
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 12/15] drm/amdgpu: Call find_vma under mmap_sem
Am 29.10.19 um 17:28 schrieb Kuehling, Felix:
Quoted 62 lines · Jason Gunthorpe
Quoted 10 lines
From: Jason Gunthorpe <[email protected]>
find_vma()must be called under themmap_sem, reorganize this code to do the vma check after entering the lock.Further, fix the unlocked use of struct task_struct's mm, instead use the mm from
hmm_mirrorwhich has an activemm_grab. Also themm_grabmust be converted to amm_getbefore acquiringmmap_semor callingfind_vma().Quoted 7 lines
Fixes 66c45500bfdc ("drm/amdgpu: use new HMM APIs and helpers")Fixes 0919195f2b0d ("drm/amdgpu: Enable amdgpu_ttm_tt_get_user_pages in worker threads")Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>One question inline to confirm my understanding. Otherwise this patch is
Reviewed-by Felix Kuehling <[email protected]>Quoted 39 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3
-6@@ -35,6 +35,7 @@#include <linux/hmm.h> #include <linux/pagemap.h> #include <linux/sched/task.h> +#include <linux/sched/mm.h> #include <linux/seq_file.h> #include <linux/slab.h> #include <linux/swap.h> @@ -788,7 +789,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)struct hmm_mirror *mirror = bo->mn ? &bo->mn->mirror :NULL ;struct ttm_tt *ttm = bo->tbo.ttm;struct amdgpu_ttm_tt *gtt = (void *)ttm; -struct mm_struct *mm = gtt->usertask->mm; +struct mm_struct *mm;unsigned long start = gtt->userptr;struct vm_area_struct *vma;struct hmm_range *range;@@ -796,25 +797,14 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)uint64_t *pfns;int r =0 ; -if (!mm)/* Happens during process shutdown */ -return -ESRCH ; -if (unlikely (!mirror)) {DRM_DEBUG_DRIVER ("Failed to get hmm_mirror\n" ); - r = -EFAULT ; - goto out; +return -EFAULT ; }
Quoted 15 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 806
Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-10- vma =find_vma (mm, start); -if (unlikely (!vma || start < vma->vm_start)) { - r = -EFAULT ; - goto out; - } -if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - vma->vm_file)) { - r = -EPERM ; - goto out; - } + mm = mirror->hmm->mmu_notifier.mm; +if (!mmget_not_zero (mm))/* Happens during process shutdown */ This works because
mirror->hmm->mmu_notifierholds an mmgrab reference to the mm? So the MM will not just go away, but if the mmget refcount is 0, it means the mm is marked for destruction and shouldn't be used any more.
Yes, exactly. That is a rather common pattern, one reference count for the functionality and one for the structure.
When the functionality is gone the structure might still be alive for some reason. TTM and a couple of other structures use the same approach.
Christian.
Quoted 32 lines
Quoted 32 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +13
-0+return -ESRCH ; range =kzalloc (sizeof (*range),GFP_KERNEL );if (unlikely (!range)) {@@ -847,6 +837,17 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT );down_read (&mm->mmap_sem); + vma =find_vma (mm, start); +if (unlikely (!vma || start < vma->vm_start)) { + r = -EFAULT ; + goto out_unlock; + } +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vma->vm_file)) { + r = -EPERM ; + goto out_unlock; + } + r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem);@@ -865,15 +866,19 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) } gtt->range = range; +mmput (mm);return 0 ;
Quoted 16 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 884
Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3
-0+out_unlock: +up_read (&mm->mmap_sem); out_free_pfns:hmm_range_unregister (range);kvfree (pfns); out_free_ranges:kfree (range); out: +mmput (mm);return r; }_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 12/15] drm/amdgpu: Call find_vma under mmap_sem
In reply to Kuehling, FelixQuoted 19 lines · Felix
Quoted 10 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
find_vma()must be called under themmap_sem, reorganize this code to do the vma check after entering the lock.Further, fix the unlocked use of struct task_struct's mm, instead use the mm from
hmm_mirrorwhich has an activemm_grab. Also themm_grabmust be converted to amm_getbefore acquiringmmap_semor callingfind_vma().Quoted 7 lines
Fixes 66c45500bfdc ("drm/amdgpu: use new HMM APIs and helpers")Fixes 0919195f2b0d ("drm/amdgpu: Enable amdgpu_ttm_tt_get_user_pages in worker threads")Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>
Quoted 3 lines
One question inline to confirm my understanding. Otherwise this patch is
Reviewed-by Felix Kuehling <[email protected]>
Thanks
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 806
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-5-if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - vma->vm_file)) { - r = -EPERM ; - goto out; - } + mm = mirror->hmm->mmu_notifier.mm; +if (!mmget_not_zero (mm))/* Happens during process shutdown */ This works because
mirror->hmm->mmu_notifierholds an mmgrab reference to the mm?
Yes, this makes sure the mm pointer remains valid
Quoted 3 lines
So the MM will not just go away, but if the mmget refcount is 0, it means the mm is marked for destruction and shouldn't be used any more.
Not just marked for destruction, but that another thread is progressing or finished release().
The other detail here is that in general you can't get the mmap_sem without also having a mmget as exit_mmap() does not lock the mmap_sem in some places where it alters the datastructures. ie racing find_vma() with exit_mmap() is not allowed.
This means we have to hold the mmget across the hmm_range_fault(), but we can drop the mmget and then test mmu_range_read_retry() under the driver lock. It will return true if the mmget refcount has gone to zero in the mean time.
But I think this is probably a poor driver design, a driver should just hold the mmget() until it has completed establishing the shadow PTEs, as it is hard to see a reason not to..
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 13/15 drm/amdgpu: Use mmu_range_insert instead of hmm_mirror
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
Remove the interval tree in the driver and rely on the tree maintained by the mmu_notifier for delivering mmu_notifier invalidation callbacks.
For some reason amdgpu has a very complicated arrangement where it tries to prevent duplicate entries in the interval_tree, this is not necessary, each amdgpu_bo can be its own stand alone entry. interval_tree already allows duplicates and overlaps in the tree.
Also, there is no need to remove entries upon a release callback, the mmu_range API safely allows objects to remain registered beyond the lifetime of the mm. The driver only has to stop touching the pages during release.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 5 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 341 ++++-------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 4 - drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- 6 files changed, 84 insertions(+), 282 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
@@ -1006,6 +1006,8 @@ struct amdgpu_device {struct mutex lock_reset;struct amdgpu_doorbell_index doorbell_index; +struct mutex notifier_lock; +int asic_reset_res;struct work_struct xgmi_reset_work;
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +2 -3
@@ -481,8 +481,7 @@ static void remove_kgd_mem_from_kfd_bo_list(struct kgd_mem *mem,* * Returns 0 for success, negative errno for errors. */ -static int init_user_pages (struct kgd_mem *mem,struct mm_struct *mm, -uint64_t user_addr) +static int init_user_pages (struct kgd_mem *mem,uint64_t user_addr) {struct amdkfd_process_info *process_info = mem->process_info;struct amdgpu_bo *bo = mem->bo;
@@ -1195,7 +1194,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(add_kgd_mem_to_kfd_bo_list (*mem, avm->process_info, user_addr);if (user_addr) { - ret =init_user_pages (*mem, current->mm, user_addr); + ret =init_user_pages (*mem, user_addr);if (ret) goto allocate_init_user_pages_failed; }
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -2633,6 +2633,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,mutex_init (&adev->virt.vf_errors.lock);hash_init (adev->mn_hash);mutex_init (&adev->lock_reset); +mutex_init (&adev->notifier_lock);mutex_init (&adev->virt.dpm_mutex);mutex_init (&adev->psp.mutex);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +70 -271
422 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0 -4
@@ -39,12 +39,10 @@ enum amdgpu_mn_type {* struct amdgpu_mn * * @adev: amdgpu device pointer -* @mm: process address space * @type: type of MMU notifier * @work: destruction work item * @node: hash table node to find structure by adev and mn * @lock: rw semaphore protecting the notifier nodes -* @objects: interval tree containing amdgpu_mn_nodes * @mirror: HMM mirror function support * * Data for each amdgpu device and process address space.
@@ -52,7 +50,6 @@ enum amdgpu_mn_type {struct amdgpu_mn {/* constant after initialisation */ struct amdgpu_device *adev; -struct mm_struct *mm;enum amdgpu_mn_type type;/* only used on destruction */
@@ -63,7 +60,6 @@ struct amdgpu_mn {/* objects protected by lock */ struct rw_semaphore lock; -struct rb_root_cached objects;#ifdef CONFIG_HMM_MIRROR /* HMM mirror */
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +9 -4
@@ -30,6 +30,9 @@#include <drm/amdgpu_drm.h> #include "amdgpu.h" +#ifdef CONFIG_MMU_NOTIFIER +#include <linux/mmu_notifier.h> +#endif #define AMDGPU_BO_INVALID_OFFSET LONG_MAX#define AMDGPU_BO_MAX_PLACEMENTS 3
@@ -100,10 +103,12 @@ struct amdgpu_bo {struct ttm_bo_kmap_obj dma_buf_vmap;struct amdgpu_mn *mn; -union { -struct list_head mn_list; -struct list_head shadow_list; - }; + +#ifdef CONFIG_MMU_NOTIFIER +struct mmu_range_notifier notifier; +#endif + +struct list_head shadow_list;struct kgd_mem *kfd_bo; };
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Koenig, Christian
maintainerRe: [PATCH v2 13/15] drm/amdgpu: Use mmu_range_insert instead of hmm_mirror
Am 28.10.19 um 21:10 schrieb Jason Gunthorpe:
Quoted 201 lines
From: Jason Gunthorpe <[email protected]>
Remove the interval tree in the driver and rely on the tree maintained by the
mmu_notifierfor deliveringmmu_notifierinvalidation callbacks.For some reason amdgpu has a very complicated arrangement where it tries to prevent duplicate entries in the
interval_tree, this is not necessary, eachamdgpu_bocan be its own stand alone entry.interval_treealready allows duplicates and overlaps in the tree.Also, there is no need to remove entries upon a release callback, the
mmu_rangeAPI safely allows objects to remain registered beyond the lifetime of the mm. The driver only has to stop touching the pages during release.Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 5 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 341 ++++-------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 4 - drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- 6 files changed, 84 insertions(+), 282 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +2
-0@@ -1006,6 +1006,8 @@ struct amdgpu_device {struct mutex lock_reset;struct amdgpu_doorbell_index doorbell_index; +struct mutex notifier_lock; +int asic_reset_res;struct work_struct xgmi_reset_work;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +2
-3@@ -481,8 +481,7 @@ static void remove_kgd_mem_from_kfd_bo_list(struct kgd_mem *mem,* * Returns 0 for success, negative errno for errors. */ -static int init_user_pages (struct kgd_mem *mem,struct mm_struct *mm, -uint64_t user_addr) +static int init_user_pages (struct kgd_mem *mem,uint64_t user_addr) {struct amdkfd_process_info *process_info = mem->process_info;struct amdgpu_bo *bo = mem->bo;@@ -1195,7 +1194,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(add_kgd_mem_to_kfd_bo_list (*mem, avm->process_info, user_addr);if (user_addr) { - ret =init_user_pages (*mem, current->mm, user_addr); + ret =init_user_pages (*mem, user_addr);if (ret) goto allocate_init_user_pages_failed; }drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -2633,6 +2633,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,mutex_init (&adev->virt.vf_errors.lock);hash_init (adev->mn_hash);mutex_init (&adev->lock_reset); +mutex_init (&adev->notifier_lock);mutex_init (&adev->virt.dpm_mutex);mutex_init (&adev->psp.mutex);drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +7
-94@@ -50,66 +50,6 @@#include "amdgpu.h" #include "amdgpu_amdkfd.h" -/** -* struct amdgpu_mn_node -* -* @it: interval node defining start-last of the affected address range -* @bos: list of all BOs in the affected address range -* -* Manages all BOs which are affected of a certain range of address space. -*/ -struct amdgpu_mn_node { -struct interval_tree_node it; -struct list_head bos; -}; - -/** -* amdgpu_mn_destroy - destroy the HMM mirror -* -* @work: previously sheduled work item -* -* Lazy destroys the notifier from a work item -*/ -static void amdgpu_mn_destroy (struct work_struct *work) -{ - struct amdgpu_mn *amn = container_of(work, struct amdgpu_mn, work); -struct amdgpu_device *adev = amn->adev; -struct amdgpu_mn_node *node, *next_node; -struct amdgpu_bo *bo, *next_bo; - -mutex_lock (&adev->mn_lock); -down_write (&amn->lock); -hash_del (&amn->node); -rbtree_postorder_for_each_entry_safe (node, next_node, - &amn->objects.rb_root, it.rb) { -list_for_each_entry_safe (bo, next_bo, &node->bos, mn_list) { - bo->mn =NULL ; -list_del_init (&bo->mn_list); - } -kfree (node); - } -up_write (&amn->lock); -mutex_unlock (&adev->mn_lock); - -hmm_mirror_unregister (&amn->mirror); -kfree (amn); -} - -/** -* amdgpu_hmm_mirror_release - callback to notify about mm destruction -* -* @mirror: the HMM mirror (mm) this callback is about -* -* Shedule a work item to lazy destroy HMM mirror. -*/ -static void amdgpu_hmm_mirror_release (struct hmm_mirror *mirror) -{ - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); - -INIT_WORK (&amn->work, amdgpu_mn_destroy); -schedule_work (&amn->work); -} -/** * amdgpu_mn_lock - take the write side lock for this notifier * @@ -133,157 +73,86 @@ void amdgpu_mn_unlock(struct amdgpu_mn *mn) }/** -* amdgpu_mn_read_lock - take the read side lock for this notifier -* -* @amn: our notifier -*/ -static int amdgpu_mn_read_lock (struct amdgpu_mn *amn,bool blockable) -{ -if (blockable) -down_read (&amn->lock); -else if (!down_read_trylock (&amn->lock)) -return -EAGAIN ; - -return 0 ; -} - -/** -* amdgpu_mn_read_unlock - drop the read side lock for this notifier -* -* @amn: our notifier -*/ -static void amdgpu_mn_read_unlock (struct amdgpu_mn *amn) -{ -up_read (&amn->lock); -} - -/** -* amdgpu_mn_invalidate_node - unmap all BOs of a node +* amdgpu_mn_invalidate_gfx - callback to notify about mm change * -* @node: the node with the BOs to unmap -* @start: start of address range affected -* @end: end of address range affected +* @mrn: the range (mm) is about to update +* @range: details on the invalidation * * Block for operations on BOs to finish and mark pages as accessed and * potentially dirty. */ -static void amdgpu_mn_invalidate_node (struct amdgpu_mn_node *node, -unsigned long start, -unsigned long end) +static bool amdgpu_mn_invalidate_gfx (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) { -struct amdgpu_bo *bo; + struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier); +struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; -list_for_each_entry (bo, &node->bos, mn_list) {
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c void amdgpu_mn_unlock(struct amdgpu_mn *mn) · line 91
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +1
-9- -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, start, end)) -continue ; - - r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, - true, false,MAX_SCHEDULE_TIMEOUT ); -if (r <=0 ) -DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); - } +/* FIXME: Is this necessary? */
Most likely not.
Christian.
Quoted 352 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +62
-168+if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, + range->end)) +return true; + +if (!mmu_notifier_range_blockable (range)) +return false; + +mutex_lock (&adev->notifier_lock); + r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, true, false, +MAX_SCHEDULE_TIMEOUT ); +mutex_unlock (&adev->notifier_lock); +if (r <=0 ) +DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); +return true; } +static const struct mmu_range_notifier_ops amdgpu_mn_gfx_ops = { + .invalidate = amdgpu_mn_invalidate_gfx, +}; +/** -* amdgpu_mn_sync_pagetables_gfx - callback to notify about mm change +* amdgpu_mn_invalidate_hsa - callback to notify about mm change * -* @mirror: the hmm_mirror (mm) is about to update -* @update: the update start, end address +* @mrn: the range (mm) is about to update +* @range: details on the invalidation * -* Block for operations on BOs to finish and mark pages as accessed and -* potentially dirty. +* We temporarily evict the BO attached to this range. This necessitates +* evicting all user-mode queues of the process. */ -static int -amdgpu_mn_sync_pagetables_gfx (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) +static bool amdgpu_mn_invalidate_hsa (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) { - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); -unsigned long start = update->start; -unsigned long end = update->end; -bool blockable =mmu_notifier_range_blockable (update); -struct interval_tree_node *it; - -/* notification is exclusive, but interval is inclusive */ - end -=1 ; - -/* TODO we should be able to split locking for interval tree and -* amdgpu_mn_invalidate_node -*/ -if (amdgpu_mn_read_lock (amn, blockable)) -return -EAGAIN ; - - it =interval_tree_iter_first (&amn->objects, start, end); -while (it) { -struct amdgpu_mn_node *node; - -if (!blockable) { -amdgpu_mn_read_unlock (amn); -return -EAGAIN ; - } + struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier); +struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); - node = container_of(it, struct amdgpu_mn_node, it); - it =interval_tree_iter_next (it, start, end); +/* FIXME: Is this necessary? */ +if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, + range->end)) +return true; -amdgpu_mn_invalidate_node (node, start, end); - } +if (!mmu_notifier_range_blockable (range)) +return false; -amdgpu_mn_read_unlock (amn); +mutex_lock (&adev->notifier_lock); +amdgpu_amdkfd_evict_userptr (bo->kfd_bo, bo->notifier.mm); +mutex_unlock (&adev->notifier_lock); -return 0 ; +return true; } -/** -* amdgpu_mn_sync_pagetables_hsa - callback to notify about mm change -* -* @mirror: the hmm_mirror (mm) is about to update -* @update: the update start, end address -* -* We temporarily evict all BOs between start and end. This -* necessitates evicting all user-mode queues of the process. The BOs -* are restorted in amdgpu_mn_invalidate_range_end_hsa. -*/ -static int -amdgpu_mn_sync_pagetables_hsa (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) +static const struct mmu_range_notifier_ops amdgpu_mn_hsa_ops = { + .invalidate = amdgpu_mn_invalidate_hsa, +}; + +static int amdgpu_mn_sync_pagetables (struct hmm_mirror *mirror, +const struct mmu_notifier_range *update) { struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); -unsigned long start = update->start; -unsigned long end = update->end; -bool blockable =mmu_notifier_range_blockable (update); -struct interval_tree_node *it; -/* notification is exclusive, but interval is inclusive */ - end -=1 ; - -if (amdgpu_mn_read_lock (amn, blockable)) -return -EAGAIN ; - - it =interval_tree_iter_first (&amn->objects, start, end); -while (it) { -struct amdgpu_mn_node *node; -struct amdgpu_bo *bo; - -if (!blockable) { -amdgpu_mn_read_unlock (amn); -return -EAGAIN ; - } - - node = container_of(it, struct amdgpu_mn_node, it); - it =interval_tree_iter_next (it, start, end); - -list_for_each_entry (bo, &node->bos, mn_list) { -struct kgd_mem *mem = bo->kfd_bo; - -if (amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, - start, end)) -amdgpu_amdkfd_evict_userptr (mem, amn->mm); - } - } - -amdgpu_mn_read_unlock (amn); +if (!mmu_notifier_range_blockable (update)) +return false; +down_read (&amn->lock); +up_read (&amn->lock);return 0 ; }@@ -295,12 +164,10 @@ amdgpu_mn_sync_pagetables_hsa(struct hmm_mirror *mirror,static struct hmm_mirror_ops amdgpu_hmm_mirror_ops[] = { [AMDGPU_MN_TYPE_GFX ] = { - .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables_gfx, - .release = amdgpu_hmm_mirror_release + .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables, }, [AMDGPU_MN_TYPE_HSA ] = { - .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables_hsa, - .release = amdgpu_hmm_mirror_release + .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables, }, };@@ -327,7 +194,8 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, }hash_for_each_possible (adev->mn_hash, amn, node, key) -if (AMDGPU_MN_KEY (amn->mm, amn->type) == key) +if (AMDGPU_MN_KEY (amn->mirror.hmm->mmu_notifier.mm, + amn->type) == key) goto release_locks; amn =kzalloc (sizeof (*amn),GFP_KERNEL );@@ -337,10 +205,8 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, } amn->adev = adev; - amn->mm = mm;init_rwsem (&amn->lock); amn->type = type; - amn->objects =RB_ROOT_CACHED ; amn->mirror.ops = &amdgpu_hmm_mirror_ops[type]; r =hmm_mirror_register (&amn->mirror, mm);@@ -369,100 +235,33 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev,* @bo: amdgpu buffer object * @addr: userptr addr we should monitor * -* Registers an HMM mirror for the given BO at the specified address. +* Registers a mmu_notifier for the given BO at the specified address. * Returns 0 on success, -ERRNO if anything goes wrong. */ int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { -unsigned long end = addr +amdgpu_bo_size (bo) -1 ; -struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); -enum amdgpu_mn_type type = - bo->kfd_bo ?AMDGPU_MN_TYPE_HSA :AMDGPU_MN_TYPE_GFX ; -struct amdgpu_mn *amn; -struct amdgpu_mn_node *node =NULL , *new_node; -struct list_head bos; -struct interval_tree_node *it; - - amn =amdgpu_mn_get (adev, type); -if (IS_ERR (amn)) -return PTR_ERR (amn); - - new_node =kmalloc (sizeof (*new_node),GFP_KERNEL ); -if (!new_node) -return -ENOMEM ; - -INIT_LIST_HEAD (&bos); - -down_write (&amn->lock); - -while ((it =interval_tree_iter_first (&amn->objects, addr, end))) { -kfree (node); - node = container_of(it, struct amdgpu_mn_node, it); -interval_tree_remove (&node->it, &amn->objects); - addr =min (it->start, addr); - end =max (it->last, end); -list_splice (&node->bos, &bos); - } - -if (!node) - node = new_node; +if (bo->kfd_bo) + bo->notifier.ops = &amdgpu_mn_hsa_ops;else -kfree (new_node); - - bo->mn = amn; - - node->it.start = addr; - node->it.last = end; -INIT_LIST_HEAD (&node->bos); -list_splice (&bos, &node->bos); -list_add (&bo->mn_list, &node->bos); + bo->notifier.ops = &amdgpu_mn_gfx_ops; -interval_tree_insert (&node->it, &amn->objects); - -up_write (&amn->lock); - -return 0 ; +return mmu_range_notifier_insert (&bo->notifier, addr, +amdgpu_bo_size (bo), current->mm); }/** -* amdgpu_mn_unregister - unregister a BO for HMM mirror updates +* amdgpu_mn_unregister - unregister a BO for notifier updates * * @bo: amdgpu buffer object * -* Remove any registration of HMM mirror updates from the buffer object. +* Remove any registration of mmu notifier updates from the buffer object. */ void amdgpu_mn_unregister (struct amdgpu_bo *bo) { -struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); -struct amdgpu_mn *amn; -struct list_head *head; - -mutex_lock (&adev->mn_lock); - - amn = bo->mn; -if (amn ==NULL ) { -mutex_unlock (&adev->mn_lock); +if (!bo->notifier.mm)return ; - } - -down_write (&amn->lock); - -/* save the next list entry for later */ - head = bo->mn_list.next; - - bo->mn =NULL ; -list_del_init (&bo->mn_list); - -if (list_empty (head)) { -struct amdgpu_mn_node *node; - - node = container_of(head, struct amdgpu_mn_node, bos); -interval_tree_remove (&node->it, &amn->objects); -kfree (node); - } - -up_write (&amn->lock); -mutex_unlock (&adev->mn_lock); +mmu_range_notifier_remove (&bo->notifier); + bo->notifier.mm =NULL ; }/* flags used by HMM internal, not related to CPU/GPU PTE flags */ drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0
-4@@ -39,12 +39,10 @@ enum amdgpu_mn_type {* struct amdgpu_mn * * @adev: amdgpu device pointer -* @mm: process address space * @type: type of MMU notifier * @work: destruction work item * @node: hash table node to find structure by adev and mn * @lock: rw semaphore protecting the notifier nodes -* @objects: interval tree containing amdgpu_mn_nodes * @mirror: HMM mirror function support * * Data for each amdgpu device and process address space. @@ -52,7 +50,6 @@ enum amdgpu_mn_type {struct amdgpu_mn {/* constant after initialisation */ struct amdgpu_device *adev; -struct mm_struct *mm;enum amdgpu_mn_type type;/* only used on destruction */ @@ -63,7 +60,6 @@ struct amdgpu_mn {/* objects protected by lock */ struct rw_semaphore lock; -struct rb_root_cached objects;#ifdef CONFIG_HMM_MIRROR /* HMM mirror */ drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +3
-3@@ -30,6 +30,9 @@#include <drm/amdgpu_drm.h> #include "amdgpu.h" +#ifdef CONFIG_MMU_NOTIFIER +#include <linux/mmu_notifier.h> +#endif #define AMDGPU_BO_INVALID_OFFSET LONG_MAX#define AMDGPU_BO_MAX_PLACEMENTS 3@@ -100,10 +103,12 @@ struct amdgpu_bo {struct ttm_bo_kmap_obj dma_buf_vmap;struct amdgpu_mn *mn; -union { -struct list_head mn_list; -struct list_head shadow_list;
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h struct amdgpu_bo { · line 114
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +6
-1- }; + +#ifdef CONFIG_MMU_NOTIFIER +struct mmu_range_notifier notifier; +#endif + +struct list_head shadow_list;struct kgd_mem *kfd_bo; };
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 13/15] drm/amdgpu: Use mmu_range_insert instead of hmm_mirror
Quoted 22 lines · Christian
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +4
-2+static bool amdgpu_mn_invalidate_gfx (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) { -struct amdgpu_bo *bo; + struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier); +struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; -list_for_each_entry (bo, &node->bos, mn_list) {drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c void amdgpu_mn_unlock(struct amdgpu_mn *mn) · line 91
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +1
-9- -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, start, end)) -continue ; - - r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, - true, false,MAX_SCHEDULE_TIMEOUT ); -if (r <=0 ) -DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); - } +/* FIXME: Is this necessary? */ Most likely not.
Quoted 5 lines
Christian.
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c void amdgpu_mn_unlock(struct amdgpu_mn *mn) · line 94
Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +3
-0+if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, + range->end)) +return true;
So is the bo->tbo.mem.num_pages == bo->tbo.ttm.num_pages always?
And userptr can't be zero here, or at least it doesn't matter if it is?
Quoted 36 lines
Quoted 36 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +8
-25+static bool amdgpu_mn_invalidate_hsa (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) { - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); -unsigned long start = update->start; -unsigned long end = update->end; -bool blockable =mmu_notifier_range_blockable (update); -struct interval_tree_node *it; - -/* notification is exclusive, but interval is inclusive */ - end -=1 ; - -/* TODO we should be able to split locking for interval tree and -* amdgpu_mn_invalidate_node -*/ -if (amdgpu_mn_read_lock (amn, blockable)) -return -EAGAIN ; - - it =interval_tree_iter_first (&amn->objects, start, end); -while (it) { -struct amdgpu_mn_node *node; - -if (!blockable) { -amdgpu_mn_read_unlock (amn); -return -EAGAIN ; - } + struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier); +struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); - node = container_of(it, struct amdgpu_mn_node, it); - it =interval_tree_iter_next (it, start, end); +/* FIXME: Is this necessary? */ +if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, + range->end)) +return true;
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +0
-2- amdgpu_mn_invalidate_node(node, start, end); - }
This one too right?
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Kuehling, Felix
maintainerRe: [PATCH v2 13/15] drm/amdgpu: Use mmu_range_insert instead of hmm_mirror
In reply to Jason GunthorpeQuoted 28 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
Remove the interval tree in the driver and rely on the tree maintained by the
mmu_notifierfor deliveringmmu_notifierinvalidation callbacks.For some reason amdgpu has a very complicated arrangement where it tries to prevent duplicate entries in the
interval_tree, this is not necessary, eachamdgpu_bocan be its own stand alone entry.interval_treealready allows duplicates and overlaps in the tree.Also, there is no need to remove entries upon a release callback, the
mmu_rangeAPI safely allows objects to remain registered beyond the lifetime of the mm. The driver only has to stop touching the pages during release.Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 5 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 341 ++++-------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 4 - drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- 6 files changed, 84 insertions(+), 282 deletions(-)
[snip]
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +0
-0
[snip]
Quoted 264 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +51
-178@@ -50,66 +50,6 @@#include "amdgpu.h" #include "amdgpu_amdkfd.h" -/** -* struct amdgpu_mn_node -* -* @it: interval node defining start-last of the affected address range -* @bos: list of all BOs in the affected address range -* -* Manages all BOs which are affected of a certain range of address space. -*/ -struct amdgpu_mn_node { -struct interval_tree_node it; -struct list_head bos; -}; - -/** -* amdgpu_mn_destroy - destroy the HMM mirror -* -* @work: previously sheduled work item -* -* Lazy destroys the notifier from a work item -*/ -static void amdgpu_mn_destroy (struct work_struct *work) -{ - struct amdgpu_mn *amn = container_of(work, struct amdgpu_mn, work); -struct amdgpu_device *adev = amn->adev; -struct amdgpu_mn_node *node, *next_node; -struct amdgpu_bo *bo, *next_bo; - -mutex_lock (&adev->mn_lock); -down_write (&amn->lock); -hash_del (&amn->node); -rbtree_postorder_for_each_entry_safe (node, next_node, - &amn->objects.rb_root, it.rb) { -list_for_each_entry_safe (bo, next_bo, &node->bos, mn_list) { - bo->mn =NULL ; -list_del_init (&bo->mn_list); - } -kfree (node); - } -up_write (&amn->lock); -mutex_unlock (&adev->mn_lock); - -hmm_mirror_unregister (&amn->mirror); -kfree (amn); -} - -/** -* amdgpu_hmm_mirror_release - callback to notify about mm destruction -* -* @mirror: the HMM mirror (mm) this callback is about -* -* Shedule a work item to lazy destroy HMM mirror. -*/ -static void amdgpu_hmm_mirror_release (struct hmm_mirror *mirror) -{ - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); - -INIT_WORK (&amn->work, amdgpu_mn_destroy); -schedule_work (&amn->work); -} -/** * amdgpu_mn_lock - take the write side lock for this notifier * @@ -133,157 +73,86 @@ void amdgpu_mn_unlock(struct amdgpu_mn *mn) }/** -* amdgpu_mn_read_lock - take the read side lock for this notifier -* -* @amn: our notifier -*/ -static int amdgpu_mn_read_lock (struct amdgpu_mn *amn,bool blockable) -{ -if (blockable) -down_read (&amn->lock); -else if (!down_read_trylock (&amn->lock)) -return -EAGAIN ; - -return 0 ; -} - -/** -* amdgpu_mn_read_unlock - drop the read side lock for this notifier -* -* @amn: our notifier -*/ -static void amdgpu_mn_read_unlock (struct amdgpu_mn *amn) -{ -up_read (&amn->lock); -} - -/** -* amdgpu_mn_invalidate_node - unmap all BOs of a node +* amdgpu_mn_invalidate_gfx - callback to notify about mm change * -* @node: the node with the BOs to unmap -* @start: start of address range affected -* @end: end of address range affected +* @mrn: the range (mm) is about to update +* @range: details on the invalidation * * Block for operations on BOs to finish and mark pages as accessed and * potentially dirty. */ -static void amdgpu_mn_invalidate_node (struct amdgpu_mn_node *node, -unsigned long start, -unsigned long end) +static bool amdgpu_mn_invalidate_gfx (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) { -struct amdgpu_bo *bo; + struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier); +struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; -list_for_each_entry (bo, &node->bos, mn_list) { - -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, start, end)) -continue ; - - r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, - true, false,MAX_SCHEDULE_TIMEOUT ); -if (r <=0 ) -DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); - } +/* FIXME: Is this necessary? */ +if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, + range->end)) +return true; + +if (!mmu_notifier_range_blockable (range)) +return false; + +mutex_lock (&adev->notifier_lock); + r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, true, false, +MAX_SCHEDULE_TIMEOUT ); +mutex_unlock (&adev->notifier_lock); +if (r <=0 ) +DRM_ERROR ("(%ld) failed to wait for user bo\n" , r); +return true; } +static const struct mmu_range_notifier_ops amdgpu_mn_gfx_ops = { + .invalidate = amdgpu_mn_invalidate_gfx, +}; +/** -* amdgpu_mn_sync_pagetables_gfx - callback to notify about mm change +* amdgpu_mn_invalidate_hsa - callback to notify about mm change * -* @mirror: the hmm_mirror (mm) is about to update -* @update: the update start, end address +* @mrn: the range (mm) is about to update +* @range: details on the invalidation * -* Block for operations on BOs to finish and mark pages as accessed and -* potentially dirty. +* We temporarily evict the BO attached to this range. This necessitates +* evicting all user-mode queues of the process. */ -static int -amdgpu_mn_sync_pagetables_gfx (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) +static bool amdgpu_mn_invalidate_hsa (struct mmu_range_notifier *mrn, +const struct mmu_notifier_range *range) { - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); -unsigned long start = update->start; -unsigned long end = update->end; -bool blockable =mmu_notifier_range_blockable (update); -struct interval_tree_node *it; - -/* notification is exclusive, but interval is inclusive */ - end -=1 ; - -/* TODO we should be able to split locking for interval tree and -* amdgpu_mn_invalidate_node -*/ -if (amdgpu_mn_read_lock (amn, blockable)) -return -EAGAIN ; - - it =interval_tree_iter_first (&amn->objects, start, end); -while (it) { -struct amdgpu_mn_node *node; - -if (!blockable) { -amdgpu_mn_read_unlock (amn); -return -EAGAIN ; - } + struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier); +struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); - node = container_of(it, struct amdgpu_mn_node, it); - it =interval_tree_iter_next (it, start, end); +/* FIXME: Is this necessary? */ +if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, + range->end)) +return true; -amdgpu_mn_invalidate_node (node, start, end); - } +if (!mmu_notifier_range_blockable (range)) +return false; -amdgpu_mn_read_unlock (amn); +mutex_lock (&adev->notifier_lock); +amdgpu_amdkfd_evict_userptr (bo->kfd_bo, bo->notifier.mm); +mutex_unlock (&adev->notifier_lock); -return 0 ; +return true; } -/** -* amdgpu_mn_sync_pagetables_hsa - callback to notify about mm change -* -* @mirror: the hmm_mirror (mm) is about to update -* @update: the update start, end address -* -* We temporarily evict all BOs between start and end. This -* necessitates evicting all user-mode queues of the process. The BOs -* are restorted in amdgpu_mn_invalidate_range_end_hsa. -*/ -static int -amdgpu_mn_sync_pagetables_hsa (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) +static const struct mmu_range_notifier_ops amdgpu_mn_hsa_ops = { + .invalidate = amdgpu_mn_invalidate_hsa, +}; + +static int amdgpu_mn_sync_pagetables (struct hmm_mirror *mirror, +const struct mmu_notifier_range *update) { struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); -unsigned long start = update->start; -unsigned long end = update->end; -bool blockable =mmu_notifier_range_blockable (update); -struct interval_tree_node *it; -/* notification is exclusive, but interval is inclusive */ - end -=1 ; - -if (amdgpu_mn_read_lock (amn, blockable)) -return -EAGAIN ; - - it =interval_tree_iter_first (&amn->objects, start, end); -while (it) { -struct amdgpu_mn_node *node; -struct amdgpu_bo *bo; - -if (!blockable) { -amdgpu_mn_read_unlock (amn); -return -EAGAIN ; - } - - node = container_of(it, struct amdgpu_mn_node, it); - it =interval_tree_iter_next (it, start, end); - -list_for_each_entry (bo, &node->bos, mn_list) { -struct kgd_mem *mem = bo->kfd_bo;
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c void amdgpu_mn_unlock(struct amdgpu_mn *mn) · line 152
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-8- -if (amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, - start, end)) -amdgpu_amdkfd_evict_userptr (mem, amn->mm); - } - } - -amdgpu_mn_read_unlock (amn); +if (!mmu_notifier_range_blockable (update)) +return false;
This should return -EAGAIN. Not sure it matters much, because this whole function disappears in the next commit in the series. It seems to be only vestigial at this point.
Regards,
Felix
Quoted 144 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +15
-80+down_read (&amn->lock); +up_read (&amn->lock);return 0 ; }@@ -295,12 +164,10 @@ amdgpu_mn_sync_pagetables_hsa(struct hmm_mirror *mirror,static struct hmm_mirror_ops amdgpu_hmm_mirror_ops[] = { [AMDGPU_MN_TYPE_GFX ] = { - .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables_gfx, - .release = amdgpu_hmm_mirror_release + .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables, }, [AMDGPU_MN_TYPE_HSA ] = { - .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables_hsa, - .release = amdgpu_hmm_mirror_release + .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables, }, };@@ -327,7 +194,8 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, }hash_for_each_possible (adev->mn_hash, amn, node, key) -if (AMDGPU_MN_KEY (amn->mm, amn->type) == key) +if (AMDGPU_MN_KEY (amn->mirror.hmm->mmu_notifier.mm, + amn->type) == key) goto release_locks; amn =kzalloc (sizeof (*amn),GFP_KERNEL );@@ -337,10 +205,8 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, } amn->adev = adev; - amn->mm = mm;init_rwsem (&amn->lock); amn->type = type; - amn->objects =RB_ROOT_CACHED ; amn->mirror.ops = &amdgpu_hmm_mirror_ops[type]; r =hmm_mirror_register (&amn->mirror, mm);@@ -369,100 +235,33 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev,* @bo: amdgpu buffer object * @addr: userptr addr we should monitor * -* Registers an HMM mirror for the given BO at the specified address. +* Registers a mmu_notifier for the given BO at the specified address. * Returns 0 on success, -ERRNO if anything goes wrong. */ int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { -unsigned long end = addr +amdgpu_bo_size (bo) -1 ; -struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); -enum amdgpu_mn_type type = - bo->kfd_bo ?AMDGPU_MN_TYPE_HSA :AMDGPU_MN_TYPE_GFX ; -struct amdgpu_mn *amn; -struct amdgpu_mn_node *node =NULL , *new_node; -struct list_head bos; -struct interval_tree_node *it; - - amn =amdgpu_mn_get (adev, type); -if (IS_ERR (amn)) -return PTR_ERR (amn); - - new_node =kmalloc (sizeof (*new_node),GFP_KERNEL ); -if (!new_node) -return -ENOMEM ; - -INIT_LIST_HEAD (&bos); - -down_write (&amn->lock); - -while ((it =interval_tree_iter_first (&amn->objects, addr, end))) { -kfree (node); - node = container_of(it, struct amdgpu_mn_node, it); -interval_tree_remove (&node->it, &amn->objects); - addr =min (it->start, addr); - end =max (it->last, end); -list_splice (&node->bos, &bos); - } - -if (!node) - node = new_node; +if (bo->kfd_bo) + bo->notifier.ops = &amdgpu_mn_hsa_ops;else -kfree (new_node); - - bo->mn = amn; - - node->it.start = addr; - node->it.last = end; -INIT_LIST_HEAD (&node->bos); -list_splice (&bos, &node->bos); -list_add (&bo->mn_list, &node->bos); + bo->notifier.ops = &amdgpu_mn_gfx_ops; -interval_tree_insert (&node->it, &amn->objects); - -up_write (&amn->lock); - -return 0 ; +return mmu_range_notifier_insert (&bo->notifier, addr, +amdgpu_bo_size (bo), current->mm); }/** -* amdgpu_mn_unregister - unregister a BO for HMM mirror updates +* amdgpu_mn_unregister - unregister a BO for notifier updates * * @bo: amdgpu buffer object * -* Remove any registration of HMM mirror updates from the buffer object. +* Remove any registration of mmu notifier updates from the buffer object. */ void amdgpu_mn_unregister (struct amdgpu_bo *bo) { -struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); -struct amdgpu_mn *amn; -struct list_head *head; - -mutex_lock (&adev->mn_lock); - - amn = bo->mn; -if (amn ==NULL ) { -mutex_unlock (&adev->mn_lock); +if (!bo->notifier.mm)return ; - } - -down_write (&amn->lock); - -/* save the next list entry for later */ - head = bo->mn_list.next; - - bo->mn =NULL ; -list_del_init (&bo->mn_list); - -if (list_empty (head)) { -struct amdgpu_mn_node *node; - - node = container_of(head, struct amdgpu_mn_node, bos); -interval_tree_remove (&node->it, &amn->objects);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, · line 267
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-5-kfree (node); - } - -up_write (&amn->lock); -mutex_unlock (&adev->mn_lock); +mmu_range_notifier_remove (&bo->notifier); + bo->notifier.mm =NULL ; }/* flags used by HMM internal, not related to CPU/GPU PTE flags */
[snip]
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 13/15] drm/amdgpu: Use mmu_range_insert instead of hmm_mirror
Quoted 40 lines · Felix
Quoted 40 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +6
-31+static const struct mmu_range_notifier_ops amdgpu_mn_hsa_ops = { + .invalidate = amdgpu_mn_invalidate_hsa, +}; + +static int amdgpu_mn_sync_pagetables (struct hmm_mirror *mirror, +const struct mmu_notifier_range *update) { struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); -unsigned long start = update->start; -unsigned long end = update->end; -bool blockable =mmu_notifier_range_blockable (update); -struct interval_tree_node *it; -/* notification is exclusive, but interval is inclusive */ - end -=1 ; - -if (amdgpu_mn_read_lock (amn, blockable)) -return -EAGAIN ; - - it =interval_tree_iter_first (&amn->objects, start, end); -while (it) { -struct amdgpu_mn_node *node; -struct amdgpu_bo *bo; - -if (!blockable) { -amdgpu_mn_read_unlock (amn); -return -EAGAIN ; - } - - node = container_of(it, struct amdgpu_mn_node, it); - it =interval_tree_iter_next (it, start, end); - -list_for_each_entry (bo, &node->bos, mn_list) { -struct kgd_mem *mem = bo->kfd_bo; - -if (amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, - start, end)) -amdgpu_amdkfd_evict_userptr (mem, amn->mm); - } - }
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c void amdgpu_mn_unlock(struct amdgpu_mn *mn) · line 152
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-2- -amdgpu_mn_read_unlock (amn); +if (!mmu_notifier_range_blockable (update)) +return false;This should return
-EAGAIN. Not sure it matters much, because this whole function disappears in the next commit in the series. It seems to be only vestigial at this point.
Right, the only reason it is still here is that I couldn't really tell if this:
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c void amdgpu_mn_unlock(struct amdgpu_mn *mn) · line 157
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-0+down_read (&amn->lock); +up_read (&amn->lock);return 0 ; }
Was serving as the 'driver lock' in the hmm scheme... If not then the whole thing should just be deleted at this point.
I fixed the EAGAIN though
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 14/15 drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
Convert the collision-retry lock around hmm_range_fault to use the one now provided by the mmu_range notifier.
Although this driver does not seem to use the collision retry lock that hmm provides correctly, it can still be converted over to use the mmu_range_notifier api instead of hmm_mirror without too much trouble.
This also deletes another place where a driver is associating additional data (struct amdgpu_mn) with a mmu_struct.
--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 148 ++---------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 49 ------ drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 76 ++++----- 5 files changed, 66 insertions(+), 225 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +4 -0
@@ -1738,6 +1738,10 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info,return ret; } +/* +* FIXME: Cannot ignore the return code, must hold +* notifier_lock +*/ amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm);/* Mark the BO as valid unless it was invalidated
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +6 -8
@@ -607,8 +607,6 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, e->tv.num_shared =2 ;amdgpu_bo_list_get_list (p->bo_list, &p->validated); -if (p->bo_list->first_userptr != p->bo_list->num_entries) - p->mn =amdgpu_mn_get (p->adev,AMDGPU_MN_TYPE_GFX );INIT_LIST_HEAD (&duplicates);amdgpu_vm_get_pd_bo (&fpriv->vm, &p->validated, &p->vm_pd);
@@ -1291,11 +1289,11 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,if (r) goto error_unlock; -/* No memory allocation is allowed while holding the mn lock. -* p->mn is hold until amdgpu_cs_submit is finished and fence is added -* to BOs. +/* No memory allocation is allowed while holding the notifier lock. +* The lock is held until amdgpu_cs_submit is finished and fence is +* added to BOs. */ -amdgpu_mn_lock (p->mn); +mutex_lock (&p->adev->notifier_lock);/* If userptr are invalidated after amdgpu_cs_parser_bos(), return * -EAGAIN, drmIoctl in libdrm will restart the amdgpu_cs_ioctl.
@@ -1338,13 +1336,13 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,amdgpu_vm_move_to_lru_tail (p->adev, &fpriv->vm);ttm_eu_fence_buffer_objects (&p->ticket, &p->validated, p->fence); -amdgpu_mn_unlock (p->mn); +mutex_unlock (&p->adev->notifier_lock);return 0 ; error_abort:drm_sched_job_cleanup (&job->base); -amdgpu_mn_unlock (p->mn); +mutex_unlock (&p->adev->notifier_lock); error_unlock:amdgpu_job_free (job);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +16 -132
184 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0 -49
59 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +40 -36
132 lines not on this page · raw message
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Hi Jason,
I did quick test after merging amd-staging-drm-next with the mmu_notifier branch, which includes this set changes. The test result has different failures, app stuck intermittently, GUI no display etc. I am understanding the changes and will try to figure out the cause.
Regards,
Philip
Quoted 475 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
Convert the collision-retry lock around
hmm_range_faultto use the one now provided by themmu_rangenotifier.Although this driver does not seem to use the collision retry lock that hmm provides correctly, it can still be converted over to use the
mmu_range_notifierapi instead ofhmm_mirrorwithout too much trouble.This also deletes another place where a driver is associating additional data (struct
amdgpu_mn) with ammu_struct.Cc Alex Deucher <[email protected]>Cc Christian König <[email protected]>Cc David (ChunMing) Zhou <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 148 ++---------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 49 ------ drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 76 ++++----- 5 files changed, 66 insertions(+), 225 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +4
-0@@ -1738,6 +1738,10 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info,return ret; } +/* +* FIXME: Cannot ignore the return code, must hold +* notifier_lock +*/ amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm);/* Mark the BO as valid unless it was invalidated drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +6
-8@@ -607,8 +607,6 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, e->tv.num_shared =2 ;amdgpu_bo_list_get_list (p->bo_list, &p->validated); -if (p->bo_list->first_userptr != p->bo_list->num_entries) - p->mn =amdgpu_mn_get (p->adev,AMDGPU_MN_TYPE_GFX );INIT_LIST_HEAD (&duplicates);amdgpu_vm_get_pd_bo (&fpriv->vm, &p->validated, &p->vm_pd);@@ -1291,11 +1289,11 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,if (r) goto error_unlock; -/* No memory allocation is allowed while holding the mn lock. -* p->mn is hold until amdgpu_cs_submit is finished and fence is added -* to BOs. +/* No memory allocation is allowed while holding the notifier lock. +* The lock is held until amdgpu_cs_submit is finished and fence is +* added to BOs. */ -amdgpu_mn_lock (p->mn); +mutex_lock (&p->adev->notifier_lock);/* If userptr are invalidated after amdgpu_cs_parser_bos(), return * -EAGAIN, drmIoctl in libdrm will restart the amdgpu_cs_ioctl. @@ -1338,13 +1336,13 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,amdgpu_vm_move_to_lru_tail (p->adev, &fpriv->vm);ttm_eu_fence_buffer_objects (&p->ticket, &p->validated, p->fence); -amdgpu_mn_unlock (p->mn); +mutex_unlock (&p->adev->notifier_lock);return 0 ; error_abort:drm_sched_job_cleanup (&job->base); -amdgpu_mn_unlock (p->mn); +mutex_unlock (&p->adev->notifier_lock); error_unlock:amdgpu_job_free (job);drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +16
-132@@ -50,28 +50,6 @@#include "amdgpu.h" #include "amdgpu_amdkfd.h" -/** -* amdgpu_mn_lock - take the write side lock for this notifier -* -* @mn: our notifier -*/ -void amdgpu_mn_lock (struct amdgpu_mn *mn) -{ -if (mn) -down_write (&mn->lock); -} - -/** -* amdgpu_mn_unlock - drop the write side lock for this notifier -* -* @mn: our notifier -*/ -void amdgpu_mn_unlock (struct amdgpu_mn *mn) -{ -if (mn) -up_write (&mn->lock); -} -/** * amdgpu_mn_invalidate_gfx - callback to notify about mm change * @@ -82,12 +60,19 @@ void amdgpu_mn_unlock(struct amdgpu_mn *mn)* potentially dirty. */ static bool amdgpu_mn_invalidate_gfx (struct mmu_range_notifier *mrn, -const struct mmu_notifier_range *range) +const struct mmu_notifier_range *range, +unsigned long cur_seq) { struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier);struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; +/* +* FIXME: Must hold some lock shared with +* amdgpu_ttm_tt_get_user_pages_done() +*/ +mmu_range_set_seq (mrn, cur_seq); +/* FIXME: Is this necessary? */ if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, range->end))@@ -119,11 +104,18 @@ static const struct mmu_range_notifier_ops amdgpu_mn_gfx_ops = {* evicting all user-mode queues of the process. */ static bool amdgpu_mn_invalidate_hsa (struct mmu_range_notifier *mrn, -const struct mmu_notifier_range *range) +const struct mmu_notifier_range *range, +unsigned long cur_seq) { struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier);struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); +/* +* FIXME: Must hold some lock shared with +* amdgpu_ttm_tt_get_user_pages_done() +*/ +mmu_range_set_seq (mrn, cur_seq); +/* FIXME: Is this necessary? */ if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, range->end))@@ -143,92 +135,6 @@ static const struct mmu_range_notifier_ops amdgpu_mn_hsa_ops = { .invalidate = amdgpu_mn_invalidate_hsa, }; -static int amdgpu_mn_sync_pagetables (struct hmm_mirror *mirror, -const struct mmu_notifier_range *update) -{ - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror); - -if (!mmu_notifier_range_blockable (update)) -return false; - -down_read (&amn->lock); -up_read (&amn->lock); -return 0 ; -} - -/* Low bits of any reasonable mm pointer will be unused due to struct -* alignment. Use these bits to make a unique key from the mm pointer -* and notifier type. -*/ -#define AMDGPU_MN_KEY (mm, type) ((unsigned long)(mm) + (type)) - -static struct hmm_mirror_ops amdgpu_hmm_mirror_ops[] = { - [AMDGPU_MN_TYPE_GFX ] = { - .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables, - }, - [AMDGPU_MN_TYPE_HSA ] = { - .sync_cpu_device_pagetables = amdgpu_mn_sync_pagetables, - }, -}; - -/** -* amdgpu_mn_get - create HMM mirror context -* -* @adev: amdgpu device pointer -* @type: type of MMU notifier context -* -* Creates a HMM mirror context for current->mm. -*/ -struct amdgpu_mn *amdgpu_mn_get (struct amdgpu_device *adev, -enum amdgpu_mn_type type) -{ -struct mm_struct *mm = current->mm; -struct amdgpu_mn *amn; -unsigned long key =AMDGPU_MN_KEY (mm, type); -int r; - -mutex_lock (&adev->mn_lock); -if (down_write_killable (&mm->mmap_sem)) { -mutex_unlock (&adev->mn_lock); -return ERR_PTR (-EINTR ); - } - -hash_for_each_possible (adev->mn_hash, amn, node, key) -if (AMDGPU_MN_KEY (amn->mirror.hmm->mmu_notifier.mm, - amn->type) == key) - goto release_locks; - - amn =kzalloc (sizeof (*amn),GFP_KERNEL ); -if (!amn) { - amn =ERR_PTR (-ENOMEM ); - goto release_locks; - } - - amn->adev = adev; -init_rwsem (&amn->lock); - amn->type = type; - - amn->mirror.ops = &amdgpu_hmm_mirror_ops[type]; - r =hmm_mirror_register (&amn->mirror, mm); -if (r) - goto free_amn; - -hash_add (adev->mn_hash, &amn->node,AMDGPU_MN_KEY (mm, type)); - -release_locks: -up_write (&mm->mmap_sem); -mutex_unlock (&adev->mn_lock); - -return amn; - -free_amn: -up_write (&mm->mmap_sem); -mutex_unlock (&adev->mn_lock); -kfree (amn); - -return ERR_PTR (r); -} -/** * amdgpu_mn_register - register a BO for notifier updates * @@ -263,25 +169,3 @@ void amdgpu_mn_unregister(struct amdgpu_bo *bo)mmu_range_notifier_remove (&bo->notifier); bo->notifier.mm =NULL ; } - -/* flags used by HMM internal, not related to CPU/GPU PTE flags */ -static const uint64_t hmm_range_flags[HMM_PFN_FLAG_MAX ] = { - (1 <<0 ),/* HMM_PFN_VALID */ - (1 <<1 ),/* HMM_PFN_WRITE */ -0 /* HMM_PFN_DEVICE_PRIVATE */ -}; - -static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX ] = { -0xfffffffffffffffeUL ,/* HMM_PFN_ERROR */ -0 ,/* HMM_PFN_NONE */ -0xfffffffffffffffcUL /* HMM_PFN_SPECIAL */ -}; - -void amdgpu_hmm_init_range (struct hmm_range *range) -{ -if (range) { - range->flags = hmm_range_flags; - range->values = hmm_range_values; - range->pfn_shift =PAGE_SHIFT ; - } -}drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0
-49@@ -30,59 +30,10 @@#include <linux/workqueue.h> #include <linux/interval_tree.h> -enum amdgpu_mn_type { -AMDGPU_MN_TYPE_GFX , -AMDGPU_MN_TYPE_HSA , -}; - -/** -* struct amdgpu_mn -* -* @adev: amdgpu device pointer -* @type: type of MMU notifier -* @work: destruction work item -* @node: hash table node to find structure by adev and mn -* @lock: rw semaphore protecting the notifier nodes -* @mirror: HMM mirror function support -* -* Data for each amdgpu device and process address space. -*/ -struct amdgpu_mn { -/* constant after initialisation */ -struct amdgpu_device *adev; -enum amdgpu_mn_type type; - -/* only used on destruction */ -struct work_struct work; - -/* protected by adev->mn_lock */ -struct hlist_node node; - -/* objects protected by lock */ -struct rw_semaphore lock; - -#ifdef CONFIG_HMM_MIRROR -/* HMM mirror */ -struct hmm_mirror mirror; -#endif -}; -#if defined(CONFIG_HMM_MIRROR ) -void amdgpu_mn_lock (struct amdgpu_mn *mn); -void amdgpu_mn_unlock (struct amdgpu_mn *mn); -struct amdgpu_mn *amdgpu_mn_get (struct amdgpu_device *adev, -enum amdgpu_mn_type type);int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr);void amdgpu_mn_unregister (struct amdgpu_bo *bo); -void amdgpu_hmm_init_range (struct hmm_range *range);#else -static inline void amdgpu_mn_lock (struct amdgpu_mn *mn) {} -static inline void amdgpu_mn_unlock (struct amdgpu_mn *mn) {} -static inline struct amdgpu_mn *amdgpu_mn_get (struct amdgpu_device *adev, -enum amdgpu_mn_type type) -{ -return NULL ; -}static inline int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { DRM_WARN_ONCE("HMM_MIRROR kernel config option is not enabled, "drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +38
-33@@ -773,6 +773,20 @@ struct amdgpu_ttm_tt {#endif }; +#ifdef CONFIG_DRM_AMDGPU_USERPTR +/* flags used by HMM internal, not related to CPU/GPU PTE flags */ +static const uint64_t hmm_range_flags[HMM_PFN_FLAG_MAX ] = { + (1 <<0 ),/* HMM_PFN_VALID */ + (1 <<1 ),/* HMM_PFN_WRITE */ +0 /* HMM_PFN_DEVICE_PRIVATE */ +}; + +static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX ] = { +0xfffffffffffffffeUL ,/* HMM_PFN_ERROR */ +0 ,/* HMM_PFN_NONE */ +0xfffffffffffffffcUL /* HMM_PFN_SPECIAL */ +}; +/** * amdgpu_ttm_tt_get_user_pages - get device accessible pages that back user * memory and start HMM tracking CPU page table update @@ -780,29 +794,27 @@ struct amdgpu_ttm_tt {* Calling function must call amdgpu_ttm_tt_userptr_range_done() once and only * once afterwards to stop HMM tracking */ -#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) - -#define MAX_RETRY_HMM_RANGE_FAULT 16 -int amdgpu_ttm_tt_get_user_pages (struct amdgpu_bo *bo,struct page **pages) { -struct hmm_mirror *mirror = bo->mn ? &bo->mn->mirror :NULL ;struct ttm_tt *ttm = bo->tbo.ttm;struct amdgpu_ttm_tt *gtt = (void *)ttm;struct mm_struct *mm; +struct hmm_range *range;unsigned long start = gtt->userptr;struct vm_area_struct *vma; -struct hmm_range *range;unsigned long i; -uint64_t *pfns;int r =0 ; -if (unlikely (!mirror)) { -DRM_DEBUG_DRIVER ("Failed to get hmm_mirror\n" ); + mm = bo->notifier.mm; +if (unlikely (!mm)) { +DRM_DEBUG_DRIVER ("BO is not registered?\n" );return -EFAULT ; } - mm = mirror->hmm->mmu_notifier.mm; +/* Another get_user_pages is running at the same time?? */ +if (WARN_ON (gtt->range)) +return -EFAULT ; +if (!mmget_not_zero (mm))/* Happens during process shutdown */ return -ESRCH ;@@ -811,30 +823,24 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) r = -ENOMEM ; goto out; } + range->notifier = &bo->notifier; + range->flags = hmm_range_flags; + range->values = hmm_range_values; + range->pfn_shift =PAGE_SHIFT ; + range->start = bo->notifier.interval_tree.start; + range->end = bo->notifier.interval_tree.last +1 ; + range->default_flags = hmm_range_flags[HMM_PFN_VALID ]; +if (!amdgpu_ttm_tt_is_readonly (ttm)) + range->default_flags |= range->flags[HMM_PFN_WRITE ]; - pfns =kvmalloc_array (ttm->num_pages,sizeof (*pfns),GFP_KERNEL ); -if (unlikely (!pfns)) { + range->pfns =kvmalloc_array (ttm->num_pages,sizeof (*range->pfns), +GFP_KERNEL ); +if (unlikely (!range->pfns)) { r = -ENOMEM ; goto out_free_ranges; } -amdgpu_hmm_init_range (range); - range->default_flags = range->flags[HMM_PFN_VALID ]; - range->default_flags |=amdgpu_ttm_tt_is_readonly (ttm) ? -0 : range->flags[HMM_PFN_WRITE ]; - range->pfn_flags_mask =0 ; - range->pfns = pfns; - range->start = start; - range->end = start + ttm->num_pages *PAGE_SIZE ; - -hmm_range_register (range, mirror); - -/* -* Just wait for range to be valid, safe to ignore return value as we -* will use the return value of hmm_range_fault() below under the -* mmap_sem to ascertain the validity of the range. -*/ -hmm_range_wait_until_valid (range,HMM_RANGE_DEFAULT_TIMEOUT ); + range->notifier_seq =mmu_range_read_begin (&bo->notifier);down_read (&mm->mmap_sem); vma =find_vma (mm, start);@@ -855,10 +861,10 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) goto out_free_pfns;for (i =0 ; i < ttm->num_pages; i++) { - pages[i] =hmm_device_entry_to_page (range, pfns[i]); + pages[i] =hmm_device_entry_to_page (range, range->pfns[i]);if (unlikely (!pages[i])) {pr_err ("Page fault failed for pfn[%lu] = 0x%llx\n" , - i, pfns[i]); + i, range->pfns[i]); r = -ENOMEM ; goto out_free_pfns;@@ -873,8 +879,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) out_unlock:up_read (&mm->mmap_sem); out_free_pfns: -hmm_range_unregister (range); -kvfree (pfns); +kvfree (range->pfns); out_free_ranges:kfree (range); out:@@ -903,9 +908,8 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) "No user pages to check\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) · line 915
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-3if (gtt->range) { - r =hmm_range_valid (gtt->range); -hmm_range_unregister (gtt->range); - + r =mmu_range_read_retry (gtt->range->notifier, + gtt->range->notifier_seq);kvfree (gtt->range->pfns);kfree (gtt->range); gtt->range =NULL ;
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Quoted 6 lines · Philip
Hi Jason,
I did quick test after merging amd-staging-drm-next with the
mmu_notifierbranch, which includes this set changes. The test result has different failures, app stuck intermittently, GUI no display etc. I am understanding the changes and will try to figure out the cause.
Thanks! I'm not surprised by this given how difficult this patch was to make. Let me know if I can assist in any way
Please ensure to run with lockdep enabled.. Your symptops sounds sort of like deadlocking?
Regards,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Quoted 8 lines · Jason Gunthorpe
Quoted 6 lines · Philip
Hi Jason,
I did quick test after merging amd-staging-drm-next with the
mmu_notifierbranch, which includes this set changes. The test result has different failures, app stuck intermittently, GUI no display etc. I am understanding the changes and will try to figure out the cause.
Quoted 6 lines
Thanks! I'm not surprised by this given how difficult this patch was to make. Let me know if I can assist in any way
Please ensure to run with lockdep enabled.. Your symptops sounds sort of like deadlocking?
Hi Jason,
Attached patch fix several issues in amdgpu driver, maybe you can squash this into patch 14. With this is done, patch 12, 13, 14 is Reviewed-by and Tested-by Philip Yang <[email protected]>
Regards,
Philip
Quoted 3 lines
Regards,
Jason
From 5a0bd4d8cef8472fe2904550142d288feed8cd81 Mon Sep 17 00:00:00 2001
From: Philip Yang <[email protected]>
Date: Thu, 31 Oct 2019 09:10:30 -0400
Subject: [PATCH] drm/amdgpu: issues with new mmu_range_notifier api
put mmu_range_set_seq under the same lock which is used to call mmu_range_read_retry.
fix amdgpu_ttm_tt_get_user_pages_done return value, because mmu_range_read_retry means !hmm_range_valid
retry if hmm_range_fault return -EBUSY
fix false WARN for missing get_user_page_done, we should check all pages not just the first page, don't understand why this issue is triggered by this change.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 32 +++++++-------------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 37 +++++++++++++++++-------- 2 files changed, 36 insertions(+), 33 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +10 -22
@@ -67,21 +67,15 @@ static bool amdgpu_mn_invalidate_gfx(struct mmu_range_notifier *mrn,struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; -/* -* FIXME: Must hold some lock shared with -* amdgpu_ttm_tt_get_user_pages_done() -*/ -mmu_range_set_seq (mrn, cur_seq); +mutex_lock (&adev->notifier_lock); -/* FIXME: Is this necessary? */ -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, - range->end)) -return true; +mmu_range_set_seq (mrn, cur_seq); -if (!mmu_notifier_range_blockable (range)) +if (!mmu_notifier_range_blockable (range)) { +mutex_unlock (&adev->notifier_lock);return false; + } -mutex_lock (&adev->notifier_lock); r =dma_resv_wait_timeout_rcu (bo->tbo.base.resv, true, false,MAX_SCHEDULE_TIMEOUT );mutex_unlock (&adev->notifier_lock);
@@ -110,21 +104,15 @@ static bool amdgpu_mn_invalidate_hsa(struct mmu_range_notifier *mrn, struct amdgpu_bo *bo = container_of(mrn, struct amdgpu_bo, notifier);struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev); -/* -* FIXME: Must hold some lock shared with -* amdgpu_ttm_tt_get_user_pages_done() -*/ -mmu_range_set_seq (mrn, cur_seq); +mutex_lock (&adev->notifier_lock); -/* FIXME: Is this necessary? */ -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, - range->end)) -return true; +mmu_range_set_seq (mrn, cur_seq); -if (!mmu_notifier_range_blockable (range)) +if (!mmu_notifier_range_blockable (range)) { +mutex_unlock (&adev->notifier_lock);return false; + } -mutex_lock (&adev->notifier_lock);amdgpu_amdkfd_evict_userptr (bo->kfd_bo, bo->notifier.mm);mutex_unlock (&adev->notifier_lock);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +26 -11
@@ -799,10 +799,11 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) {struct ttm_tt *ttm = bo->tbo.ttm;struct amdgpu_ttm_tt *gtt = (void *)ttm; -struct mm_struct *mm; -struct hmm_range *range;unsigned long start = gtt->userptr;struct vm_area_struct *vma; +struct hmm_range *range; +unsigned long timeout; +struct mm_struct *mm;unsigned long i;int r =0 ;
@@ -841,8 +842,6 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) goto out_free_ranges; } - range->notifier_seq =mmu_range_read_begin (&bo->notifier); -down_read (&mm->mmap_sem); vma =find_vma (mm, start);if (unlikely (!vma || start < vma->vm_start)) {
@@ -854,12 +853,20 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) r = -EPERM ; goto out_unlock; } +up_read (&mm->mmap_sem); + timeout = jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT ); + +retry: + range->notifier_seq =mmu_range_read_begin (&bo->notifier); +down_read (&mm->mmap_sem); r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem); - -if (unlikely (r <0 )) +if (unlikely (r <=0 )) { +if ((r ==0 || r == -EBUSY ) && !time_after (jiffies, timeout)) + goto retry; goto out_free_pfns; + }for (i =0 ; i < ttm->num_pages; i++) { pages[i] =hmm_device_entry_to_page (range, range->pfns[i]);
@@ -916,7 +923,7 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) gtt->range = NULL; } -return r; +return !r; }#endif
@@ -997,10 +1004,18 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range && - ttm->pages[0 ] ==hmm_device_entry_to_page (gtt->range, - gtt->range->pfns[0 ])) -WARN_ONCE (1 ,"Missing get_user_page_done\n" ); +if (gtt->range) { +unsigned long i; + +for (i =0 ; i < ttm->num_pages; i++) { +if (ttm->pages[i] != +hmm_device_entry_to_page (gtt->range, + gtt->range->pfns[i])) +break ; + } + +WARN ((i == ttm->num_pages),"Missing get_user_page_done\n" ); + }#endif }
Signature · 1 line
2.17.1
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Quoted 14 lines · Philip
Quoted 11 lines · Jason Gunthorpe
Quoted 6 lines · Philip
Hi Jason,
I did quick test after merging amd-staging-drm-next with the
mmu_notifierbranch, which includes this set changes. The test result has different failures, app stuck intermittently, GUI no display etc. I am understanding the changes and will try to figure out the cause.Thanks! I'm not surprised by this given how difficult this patch was to make. Let me know if I can assist in any way
Quoted 8 lines
Quoted 3 lines
Please ensure to run with lockdep enabled.. Your symptops sounds sort of like deadlocking?
Hi Jason,
Attached patch fix several issues in amdgpu driver, maybe you can squash this into patch 14. With this is done, patch 12, 13, 14 is Reviewed-by and Tested-by Philip Yang <[email protected]>
Wow, this is great thanks! Can you clarify what the problems you found were? Was the bug the 'return !r' below?
I'll also add your signed off by
Here are some remarks:
Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-5+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c@@ -67,21 +67,15 @@ static bool amdgpu_mn_invalidate_gfx(struct mmu_range_notifier *mrn,struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; -/* -* FIXME: Must hold some lock shared with -* amdgpu_ttm_tt_get_user_pages_done() -*/ -mmu_range_set_seq (mrn, cur_seq); +mutex_lock (&adev->notifier_lock);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c static bool amdgpu_mn_invalidate_gfx(struct mmu_range_notifier *mrn, · line 76
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +3
-5-/* FIXME: Is this necessary? */ -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, - range->end)) -return true; +mmu_range_set_seq (mrn, cur_seq); -if (!mmu_notifier_range_blockable (range)) +if (!mmu_notifier_range_blockable (range)) { +mutex_unlock (&adev->notifier_lock);return false;
This test for range_blockable should be before mutex_lock, I can move it up
Also, do you know if notifier_lock is held while calling amdgpu_ttm_tt_get_user_pages_done()? Can we add a 'lock assert held' to amdgpu_ttm_tt_get_user_pages_done()?
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5
-0@@ -854,12 +853,20 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) r = -EPERM ; goto out_unlock; } +up_read (&mm->mmap_sem); + timeout = jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT ); + +retry: + range->notifier_seq =mmu_range_read_begin (&bo->notifier);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 869
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5
-2+down_read (&mm->mmap_sem); r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem); - -if (unlikely (r <0 )) +if (unlikely (r <=0 )) { +if ((r ==0 || r == -EBUSY ) && !time_after (jiffies, timeout)) + goto retry; goto out_free_pfns; + }
This isn't really right, a retry loop like this needs to go all the way to mmu_range_read_retry() and done under the notifier_lock. ie mmu_range_read_retry() can fail just as likely as hmm_range_fault() can, and drivers are supposed to retry in both cases, with a single timeout.
AFAICT it is a major bug that many places ignore the return code of amdgpu_ttm_tt_get_user_pages_done() ???
However, this is all pre-existing bugs, so I'm OK go ahead with this patch as modified. I advise AMD to make a followup patch ..
I'll add a FIXME note to this effect.
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) · line 926
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +1
-1for (i =0 ; i < ttm->num_pages; i++) { pages[i] =hmm_device_entry_to_page (range, range->pfns[i]);@@ -916,7 +923,7 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) gtt->range =NULL ; } -return r; +return !r;
Ah is this the major error? hmm_range_valid() is inverted vs mmu_range_read_retry()?
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-4}#endif @@ -997,10 +1004,18 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range && - ttm->pages[0 ] ==hmm_device_entry_to_page (gtt->range, - gtt->range->pfns[0 ])) -WARN_ONCE (1 ,"Missing get_user_page_done\n" ); +if (gtt->range) { +unsigned long i;
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 1018
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +10
-0+ +for (i =0 ; i < ttm->num_pages; i++) { +if (ttm->pages[i] != +hmm_device_entry_to_page (gtt->range, + gtt->range->pfns[i])) +break ; + } + +WARN ((i == ttm->num_pages),"Missing get_user_page_done\n" ); + }
Is this related/necessary? I can put it in another patch if it is just debugging improvement? Please advise
Thanks a lot,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Quoted 18 lines · Jason Gunthorpe
Quoted 17 lines · Philip
Quoted 14 lines · Jason Gunthorpe
Quoted 6 lines · Philip
Hi Jason,
I did quick test after merging amd-staging-drm-next with the
mmu_notifierbranch, which includes this set changes. The test result has different failures, app stuck intermittently, GUI no display etc. I am understanding the changes and will try to figure out the cause.Thanks! I'm not surprised by this given how difficult this patch was to make. Let me know if I can assist in any way
Please ensure to run with lockdep enabled.. Your symptops sounds sort of like deadlocking?
Quoted 9 lines
Quoted 5 lines
Hi Jason,
Attached patch fix several issues in amdgpu driver, maybe you can squash this into patch 14. With this is done, patch 12, 13, 14 is Reviewed-by and Tested-by Philip Yang <[email protected]>
Wow, this is great thanks! Can you clarify what the problems you found were? Was the bug the 'return !r' below?
Yes. return !r is critical one, and retry if hmm_range_fault return -EBUSY is needed too.
Quoted 24 lines
I'll also add your signed off by
Here are some remarks:
Quoted 20 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +3
-9+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c@@ -67,21 +67,15 @@ static bool amdgpu_mn_invalidate_gfx(struct mmu_range_notifier *mrn,struct amdgpu_device *adev =amdgpu_ttm_adev (bo->tbo.bdev);long r; -/* -* FIXME: Must hold some lock shared with -* amdgpu_ttm_tt_get_user_pages_done() -*/ -mmu_range_set_seq (mrn, cur_seq); +mutex_lock (&adev->notifier_lock); -/* FIXME: Is this necessary? */ -if (!amdgpu_ttm_tt_affect_userptr (bo->tbo.ttm, range->start, - range->end)) -return true; +mmu_range_set_seq (mrn, cur_seq);
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c static bool amdgpu_mn_invalidate_gfx(struct mmu_range_notifier *mrn, · line 76
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +2
-1-if (!mmu_notifier_range_blockable (range)) +if (!mmu_notifier_range_blockable (range)) { +mutex_unlock (&adev->notifier_lock);return false;This test for
range_blockableshould be beforemutex_lock, I can move it up
yes, thanks.
Quoted 4 lines
Also, do you know if
notifier_lockis held while callingamdgpu_ttm_tt_get_user_pages_done()? Can we add a 'lock assert held' toamdgpu_ttm_tt_get_user_pages_done()?
gpu side hold notifier_lock but kfd side doesn't. kfd side doesn't check amdgpu_ttm_tt_get_user_pages_done/mmu_range_read_retry return value but check mem->invalid flag which is updated from invalidate callback. It takes more time to change, I will come to another patch to fix it later.
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5
-0@@ -854,12 +853,20 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) r = -EPERM ; goto out_unlock; } +up_read (&mm->mmap_sem); + timeout = jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT ); + +retry: + range->notifier_seq =mmu_range_read_begin (&bo->notifier);
Quoted 17 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 869
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5
-2+down_read (&mm->mmap_sem); r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem); - -if (unlikely (r <0 )) +if (unlikely (r <=0 )) { +if ((r ==0 || r == -EBUSY ) && !time_after (jiffies, timeout)) + goto retry; goto out_free_pfns; + }This isn't really right, a retry loop like this needs to go all the way to
mmu_range_read_retry()and done under thenotifier_lock. iemmu_range_read_retry()can fail just as likely ashmm_range_fault()can, and drivers are supposed to retry in both cases, with a single timeout.
For gpu, check mmu_range_read_retry return value under the notifier_lock to do retry is in seperate location, not in same retry loop.
Quoted 3 lines
AFAICT it is a major bug that many places ignore the return code of
amdgpu_ttm_tt_get_user_pages_done()???
For kfd, explained above.
Quoted 3 lines
However, this is all pre-existing bugs, so I'm OK go ahead with this patch as modified. I advise AMD to make a followup patch ..
yes, I will.
Quoted 8 lines
I'll add a FIXME note to this effect.
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +0
-0for (i =0 ; i < ttm->num_pages; i++) { pages[i] =hmm_device_entry_to_page (range, range->pfns[i]);@@ -916,7 +923,7 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) gtt->range =NULL ; }
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) · line 926
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +1
-1-return r; +return !r;Ah is this the major error?
hmm_range_valid()is inverted vsmmu_range_read_retry()?
yes.
Quoted 18 lines
Quoted 18 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +7
-4}#endif @@ -997,10 +1004,18 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range && - ttm->pages[0 ] ==hmm_device_entry_to_page (gtt->range, - gtt->range->pfns[0 ])) -WARN_ONCE (1 ,"Missing get_user_page_done\n" ); +if (gtt->range) { +unsigned long i; + +for (i =0 ; i < ttm->num_pages; i++) { +if (ttm->pages[i] != +hmm_device_entry_to_page (gtt->range, + gtt->range->pfns[i]))
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 1018
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5
-0+break ; + } + +WARN ((i == ttm->num_pages),"Missing get_user_page_done\n" ); + }Is this related/necessary? I can put it in another patch if it is just debugging improvement? Please advise
I see this WARN backtrace now, but I didn't see it before. This is somehow related.
Quoted 3 lines
Thanks a lot,
Jason
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Quoted 4 lines · Philip
Quoted 3 lines
This test for
range_blockableshould be beforemutex_lock, I can move it upyes, thanks.
Okay, I wrote it like this:
if (mmu_notifier_range_blockable(range)) mutex_lock(&adev->notifier_lock); else if (!mutex_trylock(&adev->notifier_lock)) return false;
Quoted 8 lines
Quoted 3 lines
Also, do you know if
notifier_lockis held while callingamdgpu_ttm_tt_get_user_pages_done()? Can we add a 'lock assert held' toamdgpu_ttm_tt_get_user_pages_done()?gpu side hold
notifier_lockbut kfd side doesn't. kfd side doesn't check amdgpu_ttm_tt_get_user_pages_done/mmu_range_read_retry return value but checkmem->invalidflag which is updated from invalidate callback. It takes more time to change, I will come to another patch to fix it later.
Ah.. confusing, OK, I'll let you sort that
Quoted 4 lines
Quoted 3 lines
However, this is all pre-existing bugs, so I'm OK go ahead with this patch as modified. I advise AMD to make a followup patch ..
yes, I will.
While you are here, this is also wrong:
int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)
{
down_read(&mm->mmap_sem);
r = hmm_range_fault(range, 0);
up_read(&mm->mmap_sem);
if (unlikely(r <= 0)) {
if ((r == 0 || r == -EBUSY) && !time_after(jiffies, timeout))
goto retry;
goto out_free_pfns;
}
for (i = 0; i < ttm->num_pages; i++) {
pages[i] = hmm_device_entry_to_page(range, range->pfns[i]);It is not allowed to read the results of hmm_range_fault() outside locking, and in particular, we can't convert to a struct page.
This must be done inside the notifier_lock, after checking mmu_range_read_retry(), all handling of the struct page must be structured like that.
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +7
-4@@ -997,10 +1004,18 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range && - ttm->pages[0 ] ==hmm_device_entry_to_page (gtt->range, - gtt->range->pfns[0 ])) -WARN_ONCE (1 ,"Missing get_user_page_done\n" ); +if (gtt->range) { +unsigned long i; + +for (i =0 ; i < ttm->num_pages; i++) { +if (ttm->pages[i] != +hmm_device_entry_to_page (gtt->range, + gtt->range->pfns[i]))Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 1018
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-0+break ; + } + +WARN ((i == ttm->num_pages),"Missing get_user_page_done\n" ); + }Is this related/necessary? I can put it in another patch if it is just debugging improvement? Please advise
Quoted 2 lines
I see this WARN backtrace now, but I didn't see it before. This is somehow related.
Hm, might be instructive to learn what is going on..
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Quoted 4 lines · Jason Gunthorpe
Quoted 3 lines · Philip
Quoted 3 lines
This test for
range_blockableshould be beforemutex_lock, I can move it up
Quoted 8 lines
Quoted 1 line
yes, thanks.
Okay, I wrote it like this:
if (mmu_notifier_range_blockable(range)) mutex_lock(&adev->notifier_lock); else if (!mutex_trylock(&adev->notifier_lock)) return false;
Never mind, this routine sleeps for other reasons it should just be as it was:
if (!mmu_notifier_range_blockable(range)) return false; mutex_lock(&adev->notifier_lock);
Jason
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
In reply to Jason GunthorpeQuoted 45 lines · Jason Gunthorpe
Quoted 4 lines · Philip
Quoted 3 lines
This test for
range_blockableshould be beforemutex_lock, I can move it upyes, thanks.
Okay, I wrote it like this:
if (mmu_notifier_range_blockable(range)) mutex_lock(&adev->notifier_lock); else if (!mutex_trylock(&adev->notifier_lock)) return false;Quoted 8 lines
Quoted 3 lines
Also, do you know if
notifier_lockis held while callingamdgpu_ttm_tt_get_user_pages_done()? Can we add a 'lock assert held' toamdgpu_ttm_tt_get_user_pages_done()?gpu side hold
notifier_lockbut kfd side doesn't. kfd side doesn't check amdgpu_ttm_tt_get_user_pages_done/mmu_range_read_retry return value but checkmem->invalidflag which is updated from invalidate callback. It takes more time to change, I will come to another patch to fix it later.Ah.. confusing, OK, I'll let you sort that
Quoted 4 lines
Quoted 3 lines
However, this is all pre-existing bugs, so I'm OK go ahead with this patch as modified. I advise AMD to make a followup patch ..
yes, I will.
While you are here, this is also wrong:
int
amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages){ down_read(&mm->mmap_sem); r = hmm_range_fault(range, 0); up_read(&mm->mmap_sem); if (unlikely(r <= 0)) { if ((r == 0 || r == -EBUSY) && !time_after(jiffies, timeout)) goto retry; goto out_free_pfns; } for (i = 0; i < ttm->num_pages; i++) { pages[i] = hmm_device_entry_to_page(range, range->pfns[i]);
Quoted 7 lines
It is not allowed to read the results of
hmm_range_fault()outside locking, and in particular, we can't convert to a struct page.This must be done inside the
notifier_lock, after checkingmmu_range_read_retry(), all handling of the struct page must be structured like that.
Below change will fix this, then driver will call mmu_range_read_retry second time using same range->notifier_seq to check if range is invalidated inside amdgpu_cs_submit, this looks ok for me.
@@ -868,6 +869,13 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)
goto out_free_pfns;
}+ mutex_lock(&adev->notifier_lock);
+
+ if (mmu_range_read_retry(&bo->notifier, range->notifier_seq)) {
+ mutex_unlock(&adev->notifier_lock);
+ goto retry;
+ }
+
for (i = 0; i < ttm->num_pages; i++) {
pages[i] = hmm_device_entry_to_page(range, range->pfns[i]);
if (unlikely(!pages[i])) {
@@ -875,10 +883,12 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo
*bo, struct page **pages)
i, range->pfns[i]);
r = -ENOMEM;+ mutex_unlock(&adev->notifier_lock);
goto out_free_pfns;
}
}+ mutex_unlock(&adev->notifier_lock);
gtt->range = range;
mmput(mm);Philip
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-4@@ -997,10 +1004,18 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range && - ttm->pages[0 ] ==hmm_device_entry_to_page (gtt->range, - gtt->range->pfns[0 ])) -WARN_ONCE (1 ,"Missing get_user_page_done\n" ); +if (gtt->range) { +unsigned long i;drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 1018
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10
-0+ +for (i =0 ; i < ttm->num_pages; i++) { +if (ttm->pages[i] != +hmm_device_entry_to_page (gtt->range, + gtt->range->pfns[i])) +break ; + } + +WARN ((i == ttm->num_pages),"Missing get_user_page_done\n" ); + }Is this related/necessary? I can put it in another patch if it is just debugging improvement? Please advise
Quoted 8 lines
Quoted 2 lines
I see this WARN backtrace now, but I didn't see it before. This is somehow related.
Hm, might be instructive to learn what is going on..
Thanks,
Jason
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
Sorry, resend patch, the one in previous email missed couple of lines duo to copy/paste.
Quoted 51 lines · Philip
Quoted 48 lines · Jason Gunthorpe
Quoted 4 lines · Philip
Quoted 3 lines
This test for
range_blockableshould be beforemutex_lock, I can move it upyes, thanks.
Okay, I wrote it like this:
if (mmu_notifier_range_blockable(range)) mutex_lock(&adev->notifier_lock); else if (!mutex_trylock(&adev->notifier_lock)) return false;Quoted 8 lines
Quoted 3 lines
Also, do you know if
notifier_lockis held while callingamdgpu_ttm_tt_get_user_pages_done()? Can we add a 'lock assert held' toamdgpu_ttm_tt_get_user_pages_done()?gpu side hold
notifier_lockbut kfd side doesn't. kfd side doesn't check amdgpu_ttm_tt_get_user_pages_done/mmu_range_read_retry return value but checkmem->invalidflag which is updated from invalidate callback. It takes more time to change, I will come to another patch to fix it later.Ah.. confusing, OK, I'll let you sort that
Quoted 4 lines
Quoted 3 lines
However, this is all pre-existing bugs, so I'm OK go ahead with this patch as modified. I advise AMD to make a followup patch ..
yes, I will.
While you are here, this is also wrong:
int
amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages){ down_read(&mm->mmap_sem); r = hmm_range_fault(range, 0); up_read(&mm->mmap_sem); if (unlikely(r <= 0)) { if ((r == 0 || r == -EBUSY) && !time_after(jiffies, timeout)) goto retry; goto out_free_pfns; } for (i = 0; i < ttm->num_pages; i++) { pages[i] = hmm_device_entry_to_page(range, range->pfns[i]);It is not allowed to read the results of
hmm_range_fault()outside locking, and in particular, we can't convert to a struct page.
Quoted 8 lines
Quoted 4 lines
This must be done inside the
notifier_lock, after checkingmmu_range_read_retry(), all handling of the struct page must be structured like that.Below change will fix this, then driver will call
mmu_range_read_retrysecond time using samerange->notifier_seqto check if range is invalidated insideamdgpu_cs_submit, this looks ok for me.
@@ -797,6 +797,7 @@ static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX] = {
*/
int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page
**pages)
{
+ struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
struct ttm_tt *ttm = bo->tbo.ttm;
struct amdgpu_ttm_tt *gtt = (void *)ttm;
unsigned long start = gtt->userptr;
@@ -868,6 +869,13 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo
*bo, struct page **pages)
goto out_free_pfns;
}+ mutex_lock(&adev->notifier_lock);
+
+ if (mmu_range_read_retry(&bo->notifier, range->notifier_seq)) {
+ mutex_unlock(&adev->notifier_lock);
+ goto retry;
+ }
+
for (i = 0; i < ttm->num_pages; i++) {
pages[i] = hmm_device_entry_to_page(range, range->pfns[i]);
if (unlikely(!pages[i])) {
@@ -875,10 +883,12 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo
*bo, struct page **pages)
i, range->pfns[i]);
r = -ENOMEM;+ mutex_unlock(&adev->notifier_lock);
goto out_free_pfns;
}
}+ mutex_unlock(&adev->notifier_lock);
gtt->range = range;
mmput(mm);Quoted 32 lines
Philip
Quoted 29 lines
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +7
-4@@ -997,10 +1004,18 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range && - ttm->pages[0 ] ==hmm_device_entry_to_page (gtt->range, - gtt->range->pfns[0 ])) -WARN_ONCE (1 ,"Missing get_user_page_done\n" ); +if (gtt->range) { +unsigned long i; + +for (i =0 ; i < ttm->num_pages; i++) { +if (ttm->pages[i] != +hmm_device_entry_to_page (gtt->range, + gtt->range->pfns[i]))Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 1018
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-0+break ; + } + +WARN ((i == ttm->num_pages),"Missing get_user_page_done\n" ); + }Is this related/necessary? I can put it in another patch if it is just debugging improvement? Please advise
Quoted 2 lines
I see this WARN backtrace now, but I didn't see it before. This is somehow related.
Hm, might be instructive to learn what is going on..
Quoted 8 lines
Quoted 3 lines
Thanks,
Jason_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
In reply to Yang, PhilipQuoted 7 lines · Philip
Quoted 4 lines
This must be done inside the
notifier_lock, after checkingmmu_range_read_retry(), all handling of the struct page must be structured like that.Below change will fix this, then driver will call
mmu_range_read_retrysecond time using samerange->notifier_seqto check if range is invalidated insideamdgpu_cs_submit, this looks ok for me.
Lets defer this to some patch trying to fix it, I find it hard to follow..
Quoted 16 lines
(quoted diff) +7
-0@@ -868,6 +869,13 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) goto out_free_pfns; } + mutex_lock(&adev->notifier_lock); + + if (mmu_range_read_retry(&bo->notifier, range->notifier_seq)) { + mutex_unlock(&adev->notifier_lock); + goto retry; + } + for (i = 0; i < ttm->num_pages; i++) { pages[i] = hmm_device_entry_to_page(range, range->pfns[i]);if (unlikely(!pages[i])) {
Review int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo · line 890
Quoted 8 lines
(quoted diff) +1
-0@@ -875,10 +883,12 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) i, range->pfns[i]); r = -ENOMEM; + mutex_unlock(&adev->notifier_lock); goto out_free_pfns; } }
Well, maybe?
The question now is what happens to 'pages' ? With this arrangment the driver cannot touch 'pages' without also again going under the lock and checking retry.
If it doesn't touch it, then lets just move this device_entry_to_page to a more appropriate place?
I'd prefer it if the driver could be structured in the normal way, with a clear locked region where the page list is handled..
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v2 14/15] drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
In reply to Yang, PhilipQuoted 10 lines · Philip
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-0@@ -854,12 +853,20 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) r = -EPERM ; goto out_unlock; } +up_read (&mm->mmap_sem); + timeout = jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT ); + +retry: + range->notifier_seq =mmu_range_read_begin (&bo->notifier);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) · line 869
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-2+down_read (&mm->mmap_sem); r =hmm_range_fault (range,0 );up_read (&mm->mmap_sem); - -if (unlikely (r <0 )) +if (unlikely (r <=0 )) { +if ((r ==0 || r == -EBUSY ) && !time_after (jiffies, timeout)) + goto retry; goto out_free_pfns; + }
I was reflecting on why this suddently became necessary, and I think what might be happening is that hmm_range_fault() is trigging invalidations as it runs (ie it is faulting in pages or something) and that in turn causes the mrn to need retry.
The hmm version of this had a bug where a full invalidate_range_start/end pair would not trigger retry, so this this didn't happen.
This is unfortunate as the retry is unnecessary, but at this time I can't think of a good way to separate an ignorable synchronous invalidation caused by hmm_range_fault from an async one that cannot be ignored..
A basic fix would be to not update the mrq seq in the notifier if the invalidate is triggered by hmm_range_fault, but that seems difficult to determine..
Any thoughts Jerome?
Jason
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 14/15 drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror
In reply to Yang, PhilipConvert the collision-retry lock around hmm_range_fault to use the one now provided by the mmu_range notifier.
Although this driver does not seem to use the collision retry lock that hmm provides correctly, it can still be converted over to use the mmu_range_notifier api instead of hmm_mirror without too much trouble.
This also deletes another place where a driver is associating additional data (struct amdgpu_mn) with a mmu_struct.
--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 150 ++---------------- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 49 ------ drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 116 +++++++++----- 5 files changed, 96 insertions(+), 237 deletions(-)
Philip, here is what it loos like after combining the two patches, thanks
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +4 -0
@@ -1738,6 +1738,10 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info,return ret; } +/* +* FIXME: Cannot ignore the return code, must hold +* notifier_lock +*/ amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm);/* Mark the BO as valid unless it was invalidated
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +6 -8
@@ -603,8 +603,6 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, e->tv.num_shared =2 ;amdgpu_bo_list_get_list (p->bo_list, &p->validated); -if (p->bo_list->first_userptr != p->bo_list->num_entries) - p->mn =amdgpu_mn_get (p->adev,AMDGPU_MN_TYPE_GFX );INIT_LIST_HEAD (&duplicates);amdgpu_vm_get_pd_bo (&fpriv->vm, &p->validated, &p->vm_pd);
@@ -1287,11 +1285,11 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,if (r) goto error_unlock; -/* No memory allocation is allowed while holding the mn lock. -* p->mn is hold until amdgpu_cs_submit is finished and fence is added -* to BOs. +/* No memory allocation is allowed while holding the notifier lock. +* The lock is held until amdgpu_cs_submit is finished and fence is +* added to BOs. */ -amdgpu_mn_lock (p->mn); +mutex_lock (&p->adev->notifier_lock);/* If userptr are invalidated after amdgpu_cs_parser_bos(), return * -EAGAIN, drmIoctl in libdrm will restart the amdgpu_cs_ioctl.
@@ -1334,13 +1332,13 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,amdgpu_vm_move_to_lru_tail (p->adev, &fpriv->vm);ttm_eu_fence_buffer_objects (&p->ticket, &p->validated, p->fence); -amdgpu_mn_unlock (p->mn); +mutex_unlock (&p->adev->notifier_lock);return 0 ; error_abort:drm_sched_job_cleanup (&job->base); -amdgpu_mn_unlock (p->mn); +mutex_unlock (&p->adev->notifier_lock); error_unlock:amdgpu_job_free (job);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +14 -136
190 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0 -49
59 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +72 -44
189 lines not on this page · raw message
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 15/15 mm/hmm: remove hmm_mirror and related
In reply to Jason GunthorpeFrom: Jason Gunthorpe <[email protected]>
The only two users of this are now converted to use mmu_range_notifier, delete all the code and update hmm.rst.
--- Documentation/vm/hmm.rst | 105 ++++----------- include/linux/hmm.h | 183 +------------------------ mm/Kconfig | 1 - mm/hmm.c | 284 +-------------------------------------- 4 files changed, 33 insertions(+), 540 deletions(-)
Documentation/vm/hmm.rst +24 -81
@@ -147,49 +147,16 @@ Address space mirroring implementation and API Address space mirroring's main objective is to allow duplication of a range of CPU page table into a device page table; HMM helps keep both synchronized. A device driver that wants to mirror a process address space must start with the -registration of an hmm_mirror struct:: - - int hmm_mirror_register(struct hmm_mirror *mirror, - struct mm_struct *mm); - -The mirror struct has a set of callbacks that are used -to propagate CPU page tables:: - - struct hmm_mirror_ops { - /* release() - release hmm_mirror - * - * @mirror: pointer to struct hmm_mirror - * - * This is called when the mm_struct is being released. The callback - * must ensure that all access to any pages obtained from this mirror - * is halted before the callback returns. All future access should - * fault. - */ - void (*release)(struct hmm_mirror *mirror); - - /* sync_cpu_device_pagetables() - synchronize page tables - * - * @mirror: pointer to struct hmm_mirror - * @update: update information (see struct mmu_notifier_range) - * Return: -EAGAIN if update.blockable false and callback need to - * block, 0 otherwise. - * - * This callback ultimately originates from mmu_notifiers when the CPU - * page table is updated. The device driver must update its page table - * in response to this callback. The update argument tells what action - * to perform. - * - * The device driver must not return from this callback until the device - * page tables are completely updated (TLBs flushed, etc); this is a - * synchronous call. - */ - int (*sync_cpu_device_pagetables)(struct hmm_mirror *mirror, - const struct hmm_update *update); - }; - -The device driver must perform the update action to the range (mark range -read only, or fully unmap, etc.). The device must complete the update before -the driver callback returns. +registration of a mmu_range_notifier:: + + mrn->ops = &driver_ops; + int mmu_range_notifier_insert(struct mmu_range_notifier *mrn, + unsigned long start, unsigned long length, + struct mm_struct *mm); + +During the driver_ops->invalidate() callback the device driver must perform +the update action to the range (mark range read only, or fully unmap, +etc.). The device must complete the update before the driver callback returns. When the device driver wants to populate a range of virtual addresses, it can use::
@@ -216,70 +183,46 @@ The usage pattern is:: struct hmm_range range; ... + range.notifier = &mrn; range.start = ...; range.end = ...; range.pfns = ...; range.flags = ...; range.values = ...; range.pfn_shift = ...; - hmm_range_register(&range, mirror); - /* - * Just wait for range to be valid, safe to ignore return value as we - * will use the return value of hmm_range_fault() below under the - * mmap_sem to ascertain the validity of the range. - */ - hmm_range_wait_until_valid(&range, TIMEOUT_IN_MSEC); + if (!mmget_not_zero(mrn->notifier.mm)) + return -EFAULT; again: + range.notifier_seq = mmu_range_read_begin(&mrn); down_read(&mm->mmap_sem); ret = hmm_range_fault(&range, HMM_RANGE_SNAPSHOT); if (ret) { up_read(&mm->mmap_sem); - if (ret == -EBUSY) { - /* - * No need to check hmm_range_wait_until_valid() return value - * on retry we will get proper error with hmm_range_fault() - */ - hmm_range_wait_until_valid(&range, TIMEOUT_IN_MSEC); - goto again; - } - hmm_range_unregister(&range); + if (ret == -EBUSY) + goto again; return ret; } + up_read(&mm->mmap_sem); + take_lock(driver->update); - if (!hmm_range_valid(&range)) { + if (mmu_range_read_retry(&mrn, range.notifier_seq) { release_lock(driver->update); - up_read(&mm->mmap_sem); goto again; } - // Use pfns array content to update device page table + /* Use pfns array content to update device page table, + * under the update lock */ - hmm_range_unregister(&range); release_lock(driver->update); - up_read(&mm->mmap_sem); return 0; } The driver->update lock is the same lock that the driver takes inside its -sync_cpu_device_pagetables() callback. That lock must be held before calling -hmm_range_valid() to avoid any race with a concurrent CPU page table update. - -HMM implements all this on top of the mmu_notifier API because we wanted a -simpler API and also to be able to perform optimizations latter on like doing -concurrent device updates in multi-devices scenario. - -HMM also serves as an impedance mismatch between how CPU page table updates -are done (by CPU write to the page table and TLB flushes) and how devices -update their own page table. Device updates are a multi-step process. First, -appropriate commands are written to a buffer, then this buffer is scheduled for -execution on the device. It is only once the device has executed commands in -the buffer that the update is done. Creating and scheduling the update command -buffer can happen concurrently for multiple devices. Waiting for each device to -report commands as executed is serialized (there is no point in doing this -concurrently). - +invalidate() callback. That lock must be held before calling +mmu_range_read_retry() to avoid any race with a concurrent CPU page table +update. Leverage default_flags and pfn_flags_mask =========================================
include/linux/hmm.h +2 -181
230 lines not on this page · raw message
mm/Kconfig +0 -1
@@ -675,7 +675,6 @@ config DEV_PAGEMAP_OPS config HMM_MIRROR bool depends on MMU - depends on MMU_NOTIFIER config DEVICE_PRIVATE bool "Unaddressable device memory (GPU memory, ...)"
mm/hmm.c +7 -277
321 lines not on this page · raw message
Signature · 7 lines
2.23.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 8 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
8 of the
mmu_notifierusing drivers (i915_gem,radeon_mn,umem_odp, hfi1,scif_dma, vhost, gntdev, hmm) drivers are using a common pattern where they only use invalidate_range_start/end and immediately check the invalidating range against some driver data structure to tell if the driver is interested. Half of them use aninterval_tree, the others are simple linear search lists.
Now that we have the most of the driver changes tested and reviewed I'm going to move this series into linux-next via the hmm tree, minus the xen gntdev patches as they are not working yet.
I will keep collecting acks and any additional changes.
Thanks,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 88 lines · Jason Gunthorpe
From: Jason Gunthorpe <[email protected]>
8 of the
mmu_notifierusing drivers (i915_gem,radeon_mn,umem_odp, hfi1,scif_dma, vhost, gntdev, hmm) drivers are using a common pattern where they only use invalidate_range_start/end and immediately check the invalidating range against some driver data structure to tell if the driver is interested. Half of them use aninterval_tree, the others are simple linear search lists.Of the ones I checked they largely seem to have various kinds of races, bugs and poor implementation. This is a result of the complexity in how the notifier interacts with
get_user_pages(). It is extremely difficult to use it correctly.Consolidate all of this code together into the core
mmu_notifierand provide a locking scheme similar tohmm_mirrorthat allows the user to safely useget_user_pages()and reliably know if the page list still matches the mm.This new arrangment plays nicely with the !blockable mode for OOM. Scanning the interval tree is done such that the intersection test will always succeed, and since there is no
invalidate_range_endexposed to drivers the scheme safely allows multiple drivers to be subscribed.Four places are converted as an example of how the new API is used. Four are left for future patches:
-i915_gemhas complex locking around destruction of a registration, needs more study
- hfi1 (2nd user) needs access to the rbtree
-scif_dmahas a complicated logic flow
- vhost's mmu notifiers are already being rewrittenThis series, and the other code it depends on is available on my github:
https://github.com/jgunthorpe/linux/commits/mmu_notifier
v2 changes:
- Addmmu_range_set_seq()to set the mrn sequence number under the driver lock and make the locking more understandable
- Add some additional comments around locking/READ_ONCe
- Make theWARN_ONflow inmn_itree_invalidatea bit easier to follow
- Fix wrongWARN_ONJason Gunthorpe (15):
mm/mmu_notifier: define the header pre-processor parts even if
disabled
mm/mmu_notifier: add an interval tree notifier mm/hmm: allowhmm_rangeto be used with ammu_range_notifieror
hmm_mirror
mm/hmm: define the pre-processor related parts ofhmm.heven if
disabled
RDMA/odp: Usemmu_range_notifier_insert()
RDMA/hfi1: Usemmu_range_notifier_insetforuser_exp_rcvdrm/radeon: usemmu_range_notifier_insertxen/gntdev: Use select forDMA_SHARED_BUFFERxen/gntdev: usemmu_range_notifier_insert
nouveau: usemmu_notifierdirectly forinvalidate_range_start
nouveau: usemmu_range_notifierinstead ofhmm_mirrordrm/amdgpu: Callfind_vmaundermmap_sem
drm/amdgpu: Usemmu_range_insertinstead ofhmm_mirrordrm/amdgpu: Usemmu_range_notifierinstead ofhmm_mirrormm/hmm: removehmm_mirrorand relatedDocumentation/vm/hmm.rst | 105 +--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 9 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 457 +++------------ drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 53 -- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 111 ++-- drivers/gpu/drm/nouveau/nouveau_svm.c | 231 +++++--- drivers/gpu/drm/radeon/radeon.h | 9 +- drivers/gpu/drm/radeon/radeon_mn.c | 219 ++----- drivers/infiniband/core/device.c | 1 - drivers/infiniband/core/umem_odp.c | 288 +-------- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/hfi1/hfi.h | 2 +- drivers/infiniband/hw/hfi1/user_exp_rcv.c | 146 ++--- drivers/infiniband/hw/hfi1/user_exp_rcv.h | 3 +- drivers/infiniband/hw/mlx5/mlx5_ib.h | 7 +- drivers/infiniband/hw/mlx5/mr.c | 3 +- drivers/infiniband/hw/mlx5/odp.c | 50 +- drivers/xen/Kconfig | 3 +- drivers/xen/gntdev-common.h | 8 +- drivers/xen/gntdev.c | 180 ++---- include/linux/hmm.h | 195 +------
Quoted 9 lines
include/linux/mmu_notifier.h | 144 ++++- include/rdma/ib_umem_odp.h | 65 +-- include/rdma/ib_verbs.h | 2 - kernel/fork.c | 1 - mm/Kconfig | 2 +- mm/hmm.c | 275 +-------- mm/mmu_notifier.c | 546 +++++++++++++++++- 32 files changed, 1225 insertions(+), 1922 deletions(-)
You can add my Tested-by for the mm and nouveau changes. IOW, patches 1-4, 10-11, and 15.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines · Ralph Campbell
You can add my Tested-by for the mm and nouveau changes. IOW, patches 1-4, 10-11, and 15.
Tested-by Ralph Campbell <[email protected]>
Got it, thanks
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel