iommu/amd/iommu_v2: Fix refcount related issues
Subsystems (2)
Vasant Hegde
maintainercover iommu/amd/iommu_v2: Fix refcount related issues
This patch series fixes various issues around pasid refcount handling.
Issues addressed in this patchset are as follows:
1. refcount_dec assumes that refcount will never reach 0:
threfore: REFCOUNT_WARN("decrement hit 0; leaking memory") will be invoked on pasid unbind
* Fixed by changing refcount_dec to refcount_dec_and_test to explicitly handle refcount=1 (skip waiting for wait queue)
2. ppr_handler calls the wake_up API on the pasid_state wait_queue. However, once refcount has been decremented, unbind_pasid may have already freed this object - causing unallocated memory access
* Fixed by adding spinlock to ensure atomicity between refcount check and event queue handling. Moved wait_queue APIs to wake_up_locked / wake_event_interruptible_locked accordingly
3. If new ppr_handler arrives after get_pasid_wait once pasid unbind has decremented refcount to 0 - get_pasid_state can still acquire a pointer to pasid_state after being freed by unbind_pasid causing unallocated memory access
* Fixed by changing ref_count_inc to ref_count_inc_not_zero to ensure no new ppr_handle starts after pasid has been unbound and NULL (invalid) pasid_state is returned on zero.
Base commit: v6.2-rc5
Daniel Marcovitch (3):
iommu/amd/iommu_v2: Fix pasid_state refcount dec hit 0 warning on pasid unbind iommu/amd/iommu_v2: Fix pasid_state->wq race
iommu/amd/iommu_v2: Prevent scheduling new ppr notifier during unbind_pasid
drivers/iommu/amd/iommu_v2.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
Signature · 1 line
2.31.1
Vasant Hegde
maintainerPATCH 1/3 iommu/amd/iommu_v2: Fix pasid_state refcount dec hit 0 warning on pasid unbind
From: Daniel Marcovitch <[email protected]>
When unbinding pasid - a race condition exists vs outstanding page faults.
To prevent this, the pasid_state object contains a refcount.
* set to 1 on pasid bind
* incremented on each ppr notification start
* decremented on each ppr notification done
* decremented on pasid unbind
Since refcount_dec assumes that refcount will never reach 0:
the current implementation causes the following to be invoked on pasid unbind:
REFCOUNT_WARN("decrement hit 0; leaking memory")
Fix this issue by changing refcount_dec to refcount_dec_and_test to explicitly handle refcount=1.
--- drivers/iommu/amd/iommu_v2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/iommu/amd/iommu_v2.c +2 -2
@@ -262,8 +262,8 @@ static void put_pasid_state(struct pasid_state *pasid_state)static void put_pasid_state_wait (struct pasid_state *pasid_state) { -refcount_dec (&pasid_state->count); -wait_event (pasid_state->wq, !refcount_read (&pasid_state->count)); +if (!refcount_dec_and_test (&pasid_state->count)) +wait_event (pasid_state->wq, !refcount_read (&pasid_state->count));free_pasid_state (pasid_state); }
Signature · 1 line
2.31.1
Re: [PATCH 1/3] iommu/amd/iommu_v2: Fix pasid_state refcount dec hit 0 warning on pasid unbind
Quoted 9 lines · Vasant Hegde
From: Daniel Marcovitch <[email protected]>
When unbinding pasid - a race condition exists vs outstanding page faults.
To prevent this, the
pasid_stateobject contains a refcount.
* set to 1 on pasid bind
* incremented on each ppr notification start
* decremented on each ppr notification done
* decremented on pasid unbind
I've no objection to this quick fix
But the issue here is using a refcount for something that is not a refcount. This usage model is should just be a normal atomic if it is only counting the number of notifications in progress.
To be a refcount to have to follow one of the standard refcount patterns, ie get rid of the event queue and consistently do:
if (refcount_dec_and_test())
free_pasid_state()
Which implies the ppr notification done can do a defered free of the pasid_state.
Jason
Vasant Hegde
maintainerRe: [PATCH 1/3] iommu/amd/iommu_v2: Fix pasid_state refcount dec hit 0 warning on pasid unbind
Hi Jason,
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · Vasant Hegde
From: Daniel Marcovitch <[email protected]>
When unbinding pasid - a race condition exists vs outstanding page faults.
To prevent this, the
pasid_stateobject contains a refcount.
* set to 1 on pasid bind
* incremented on each ppr notification start
* decremented on each ppr notification done
* decremented on pasid unbind
Quoted 5 lines
I've no objection to this quick fix
But the issue here is using a refcount for something that is not a refcount. This usage model is should just be a normal atomic if it is only counting the number of notifications in progress.
Originally it was using atomic counter.. which got converted to refcount by below commit:
commit 8bc54824da4e8fcf0ed679cf09ac32f23d83254a
Author: Xiyu Yang via iommu <[email protected]> Date: Mon Jul 19 16:32:58 2021 +0800
iommu/amd: Convert from atomic_t to refcount_t on pasid_state->count
.. after this commit it started throwing warnings. We thought of fixing the warning instead of reverting above commit.
-Vasant
Quoted 11 lines
To be a refcount to have to follow one of the standard refcount patterns, ie get rid of the event queue and consistently do:
if (
refcount_dec_and_test())
free_pasid_state()Which implies the ppr notification done can do a defered free of the
pasid_state.Jason
Re: [PATCH 1/3] iommu/amd/iommu_v2: Fix pasid_state refcount dec hit 0 warning on pasid unbind
Quoted 24 lines · Vasant Hegde
Hi Jason,
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · Vasant Hegde
From: Daniel Marcovitch <[email protected]>
When unbinding pasid - a race condition exists vs outstanding page faults.
To prevent this, the
pasid_stateobject contains a refcount.
* set to 1 on pasid bind
* incremented on each ppr notification start
* decremented on each ppr notification done
* decremented on pasid unbindQuoted 5 lines
I've no objection to this quick fix
But the issue here is using a refcount for something that is not a refcount. This usage model is should just be a normal atomic if it is only counting the number of notifications in progress.
Originally it was using atomic counter.. which got converted to refcount by below commit:
Quoted 8 lines
commit
8bc54824da4e8fcf0ed679cf09ac32f23d83254aAuthor: Xiyu Yang via iommu <[email protected]> Date: Mon Jul 19 16:32:58 2021 +0800iommu/amd: Convert from atomic_t to refcount_t on pasid_state->count.. after this commit it started throwing warnings. We thought of fixing the warning instead of reverting above commit.
The mechanical clean up was wrong because it subtly wasn't a refcount :)
But given the other problems here you can fix it with the approach I gave in the other email
Jason
Vasant Hegde
maintainerPATCH 2/3 iommu/amd/iommu_v2: Fix pasid_state->wq race
In reply to Vasant HegdeFrom: Daniel Marcovitch <[email protected]>
pasid_state->wq is used to wait for all outstanding pri handling prior to pasid unbind.
ppr handler calls the wake_up API on the pasid_state->wq once pasid_state refcount has reached 0. Since refcount is initialized to 1, this will only occur during pasid unbind.
However, unbind_pasid may reach refcount check at this point of ppr_handler execution. Since refcount is 0, it will not wait on wait_queue - but rather proceed with pasid_state deallocation, including wait_queue.
This can cause unallocated memory access.
Fixed by adding spinlock to ensure atomicity between refcount check and event queue handling. Moved wait_queue APIs to wake_up_locked / wake_event_interruptible_locked accordingly.
--- drivers/iommu/amd/iommu_v2.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/iommu/amd/iommu_v2.c +6 -2
@@ -256,14 +256,18 @@ static void free_pasid_state(struct pasid_state *pasid_state)static void put_pasid_state (struct pasid_state *pasid_state) { +spin_lock (&pasid_state->wq.lock);if (refcount_dec_and_test (&pasid_state->count)) -wake_up (&pasid_state->wq); +wake_up_locked (&pasid_state->wq); +spin_unlock (&pasid_state->wq.lock); }static void put_pasid_state_wait (struct pasid_state *pasid_state) { +spin_lock (&pasid_state->wq.lock);if (!refcount_dec_and_test (&pasid_state->count)) -wait_event (pasid_state->wq, !refcount_read (&pasid_state->count)); +wait_event_interruptible_locked (pasid_state->wq, !refcount_read (&pasid_state->count)); +spin_unlock (&pasid_state->wq.lock);free_pasid_state (pasid_state); }
Signature · 1 line
2.31.1
Re: [PATCH 2/3] iommu/amd/iommu_v2: Fix pasid_state->wq race
Quoted 37 lines · Vasant Hegde
From: Daniel Marcovitch <[email protected]>
pasid_state->wqis used to wait for all outstanding pri handling prior to pasid unbind.ppr handler calls the
wake_upAPI on thepasid_state->wqoncepasid_staterefcount has reached 0. Since refcount is initialized to 1, this will only occur during pasid unbind.However,
unbind_pasidmay reach refcount check at this point ofppr_handlerexecution. Since refcount is 0, it will not wait onwait_queue- but rather proceed withpasid_statedeallocation, includingwait_queue.This can cause unallocated memory access.
Fixed by adding spinlock to ensure atomicity between refcount check and event queue handling. Moved
wait_queueAPIs towake_up_locked/wake_event_interruptible_lockedaccordingly.Signed-off-by Daniel Marcovitch <[email protected]>Signed-off-by Vasant Hegde <[email protected]>--- drivers/iommu/amd/iommu_v2.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)drivers/iommu/amd/iommu_v2.c +2
-1@@ -256,14 +256,18 @@ static void free_pasid_state(struct pasid_state *pasid_state)static void put_pasid_state (struct pasid_state *pasid_state) { +spin_lock (&pasid_state->wq.lock);if (refcount_dec_and_test (&pasid_state->count)) -wake_up (&pasid_state->wq); +wake_up_locked (&pasid_state->wq);
drivers/iommu/amd/iommu_v2.c static void free_pasid_state(struct pasid_state *pasid_state) · line 270
Quoted 10 lines
drivers/iommu/amd/iommu_v2.c +4
-1+spin_unlock (&pasid_state->wq.lock); }static void put_pasid_state_wait (struct pasid_state *pasid_state) { +spin_lock (&pasid_state->wq.lock);if (!refcount_dec_and_test (&pasid_state->count)) -wait_event (pasid_state->wq, !refcount_read (&pasid_state->count)); +wait_event_interruptible_locked (pasid_state->wq, !refcount_read (&pasid_state->count)); +spin_unlock (&pasid_state->wq.lock);
You can't call wait_event_interruptible_locked() while holding a spinlock
What this is really trying to make is one of those open coded rwlocks that can be passed into a WQ.
struct refcount count
struct completion comp
init:
refcount_set(count, 1)
get:
if !refcount_inc_not_zero(count)
// Concurrent destruction, can't get the pasid state!
put:
if refcount_dec_and_test(count)
complete(&device->comp)
destroy:
put() // put back the refcount_set()
wait_for_completion()
// Now count == 0
clear_pasid_state()
free()
Jason
Vasant Hegde
maintainerPATCH 3/3 iommu/amd/iommu_v2: Prevent scheduling new ppr notifier during unbind_pasid
In reply to Vasant HegdeFrom: Daniel Marcovitch <[email protected]>
The pasid state wait_queue / ref_count mechanism allows unbind_pasid to wait for all outstanding ppr requests to be completed prior to freeing pasid_state.
However, we are still missing a mechanism to prevent new ppr_notifier being invoked after refcount has been decremented to 0, and prior to pasid_state deallocation.
This can cause unallocated memory access.
Fixed by changing ref_count_inc to ref_count_inc_not_zero to ensure no new ppr_handler starts after pasid has been unbound and NULL (invalid) pasid_state is returned on zero.
--- drivers/iommu/amd/iommu_v2.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/iommu/amd/iommu_v2.c +7 -2
@@ -240,8 +240,13 @@ static struct pasid_state *get_pasid_state(struct device_state *dev_state, goto out_unlock; ret = *ptr; -if (ret) -refcount_inc (&ret->count); + +/* +* Since pasid_state creation sets initial count to 1, +* a count of 0 indicates that unbind_pasid is in progress +*/ +if (ret && !refcount_inc_not_zero (&ret->count)) + ret =NULL ; out_unlock:spin_unlock_irqrestore (&dev_state->lock, flags);
Signature · 1 line
2.31.1
Re: [PATCH 3/3] iommu/amd/iommu_v2: Prevent scheduling new ppr notifier during unbind_pasid
Quoted 10 lines · Vasant Hegde
From: Daniel Marcovitch <[email protected]>
The pasid state
wait_queue/ref_countmechanism allowsunbind_pasidto wait for all outstanding ppr requests to be completed prior to freeingpasid_state.However, we are still missing a mechanism to prevent new
ppr_notifierbeing invoked after refcount has been decremented to 0, and prior topasid_statedeallocation.
Quoted 5 lines
This can cause unallocated memory access.
Fixed by changing
ref_count_inctoref_count_inc_not_zeroto ensure no newppr_handlerstarts after pasid has been unbound and NULL (invalid)pasid_stateis returned on zero.
It looks like this is prevented by clear_pasid_state() which will NULL the pasid entry under the spinlock before an attempt is made to zero the refcount?
Though maybe that is missed in the free_pasid_states() path?
Also this whole thing is just begging to be converted into an xarray..
Jason
Vasant Hegde
maintainerRe: [PATCH 3/3] iommu/amd/iommu_v2: Prevent scheduling new ppr notifier during unbind_pasid
Jason,
Quoted 17 lines · Jason Gunthorpe
Quoted 10 lines · Vasant Hegde
From: Daniel Marcovitch <[email protected]>
The pasid state
wait_queue/ref_countmechanism allowsunbind_pasidto wait for all outstanding ppr requests to be completed prior to freeingpasid_state.However, we are still missing a mechanism to prevent new
ppr_notifierbeing invoked after refcount has been decremented to 0, and prior topasid_statedeallocation.Quoted 5 lines
This can cause unallocated memory access.
Fixed by changing
ref_count_inctoref_count_inc_not_zeroto ensure no newppr_handlerstarts after pasid has been unbound and NULL (invalid)pasid_stateis returned on zero.
Quoted 5 lines
It looks like this is prevented by
clear_pasid_state()which will NULL the pasid entry under the spinlock before an attempt is made to zero the refcount?Though maybe that is missed in the
free_pasid_states()path?
You are right. We just need to add clear_pasid_start in above path (something like below).
----------
drivers/iommu/amd/iommu_v2.c +2 -0
@@ -336,6 +336,8 @@ static void free_pasid_states(struct device_state *dev_state)put_pasid_state (pasid_state); +clear_pasid_state (dev_state, pasid); +/* * This will call the mn_release function and * unbind the PASID
Quoted 2 lines
Also this whole thing is just begging to be converted into an xarray..
Agree. Makes sense. We will look into it later.
-Vasant