mm/mmu_notifiers: Esnure range_end() is paired with range_start()
Subsystems (2)
PATCH v2 1/1 mm/mmu_notifiers: Esnure range_end() is paired with range_start()
If one or more notifiers fails .invalidate_range_start(), invoke .invalidate_range_end() for "all" notifiers. If there are multiple notifiers, those that did not fail are expecting _start() and _end() to be paired, e.g. KVM's mmu_notifier_count would become imbalanced. Disallow notifiers that can fail _start() from implementing _end() so that it's unnecessary to either track which notifiers rejected _start(), or had already succeeded prior to a failed _start().
Note, the existing behavior of calling _start() on all notifiers even after a previous notifier failed _start() was an unintented "feature". Make it canon now that the behavior is depended on for correctness.
As of today, the bug is likely benign:
1. The only caller of the non-blocking notifier is OOM kill.
2. The only notifiers that can fail _start() are the i915 and Nouveau drivers.
3. The only notifiers that utilize _end() are the SGI UV GRU driver and KVM.
4. The GRU driver will never coincide with the i195/Nouveau drivers.
5. An imbalanced kvm->mmu_notifier_count only causes soft lockup in the _guest_, and the guest is already doomed due to being an OOM victim.
Fix the bug now to play nice with future usage, e.g. KVM has a potential use case for blocking memslot updates in KVM while an invalidation is in-progress, and failure to unblock would result in said updates being blocked indefinitely and hanging.
Found by inspection. Verified by adding a second notifier in KVM that periodically returns -EAGAIN on non-blockable ranges, triggering OOM, and observing that KVM exits with an elevated notifier count.
---
v2: Reimplemented as suggested by Jason. Only functional change relative
to Jason's suggestion is to check invalidate_range_end before calling to avoid a NULL pointer dereference. I also added more comments, hopefully they're helpful...
v1: https://lkml.kernel.org/r/[email protected]
include/linux/mmu_notifier.h | 10 +++++----- mm/mmu_notifier.c | 23 +++++++++++++++++++++++ 2 files changed, 28 insertions(+), 5 deletions(-)
include/linux/mmu_notifier.h +5 -5
@@ -169,11 +169,11 @@ struct mmu_notifier_ops {* the last refcount is dropped. * * If blockable argument is set to false then the callback cannot -* sleep and has to return with -EAGAIN. 0 should be returned -* otherwise. Please note that if invalidate_range_start approves -* a non-blocking behavior then the same applies to -* invalidate_range_end. -* +* sleep and has to return with -EAGAIN if sleeping would be required. +* 0 should be returned otherwise. Please note that notifiers that can +* fail invalidate_range_start are not allowed to implement +* invalidate_range_end, as there is no mechanism for informing the +* notifier that its start failed. */ int (*invalidate_range_start)(struct mmu_notifier *subscription,const struct mmu_notifier_range *range);
mm/mmu_notifier.c +23 -0
@@ -501,10 +501,33 @@ static int mn_hlist_invalidate_range_start( "");WARN_ON (mmu_notifier_range_blockable (range) || _ret != -EAGAIN ); +/* +* We call all the notifiers on any EAGAIN, +* there is no way for a notifier to know if +* its start method failed, thus a start that +* does EAGAIN can't also do end. +*/ +WARN_ON (ops->invalidate_range_end); ret = _ret; } } } + +if (ret) { +/* +* Must be non-blocking to get here. If there are multiple +* notifiers and one or more failed start, any that succeeded +* start are expecting their end to be called. Do so now. +*/ +hlist_for_each_entry_rcu (subscription, &subscriptions->list, + hlist,srcu_read_lock_held (&srcu)) { +if (!subscription->ops->invalidate_range_end) +continue ; + + subscription->ops->invalidate_range_end (subscription, + range); + } + }srcu_read_unlock (&srcu, id);return ret;
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Quoted 44 lines · Sean Christopherson
If one or more notifiers fails .invalidate_range_start(), invoke .invalidate_range_end() for "all" notifiers. If there are multiple notifiers, those that did not fail are expecting
_start()and_end()to be paired, e.g. KVM'smmu_notifier_countwould become imbalanced. Disallow notifiers that can fail_start()from implementing_end()so that it's unnecessary to either track which notifiers rejected_start(), or had already succeeded prior to a failed_start().Note, the existing behavior of calling
_start()on all notifiers even after a previous notifier failed_start()was an unintented "feature". Make it canon now that the behavior is depended on for correctness.As of today, the bug is likely benign:
1. The only caller of the non-blocking notifier is OOM kill.
2. The only notifiers that can fail_start()are the i915 and Nouveau drivers.
3. The only notifiers that utilize_end()are the SGI UV GRU driver and KVM.
4. The GRU driver will never coincide with the i195/Nouveau drivers.
5. An imbalancedkvm->mmu_notifier_countonly causes soft lockup in the _guest_, and the guest is already doomed due to being an OOM victim.Fix the bug now to play nice with future usage, e.g. KVM has a potential use case for blocking memslot updates in KVM while an invalidation is in-progress, and failure to unblock would result in said updates being blocked indefinitely and hanging.
Found by inspection. Verified by adding a second notifier in KVM that periodically returns
-EAGAINon non-blockable ranges, triggering OOM, and observing that KVM exits with an elevated notifier count.Fixes 93065ac753e4 ("mm, oom: distinguish blockable mode for mmu notifiers")Suggested-by Jason Gunthorpe <[email protected]>Cc David Rientjes <[email protected]>Cc Ben Gardon <[email protected]>Cc Michal Hocko <[email protected]>Cc "Jérôme Glisse" <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Johannes Weiner <[email protected]>Cc Dimitri Sivanich <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Quoted 6 lines
v2: Reimplemented as suggested by Jason. Only functional change relative
to Jason's suggestion is to checkinvalidate_range_endbefore calling to avoid a NULL pointer dereference. I also added more comments, hopefully they're helpful...
Looks fine, thanks. I think you need some commit message remark to discourage backporting, the added WARN_ON will trigger on older kernels that have many more things implementing invalidate_range_end(). It should not be backported to anything that has more invalidate_range_ends()'s than today's kernel.
Jason