padata: fix liftime issues after ->serial() has completed
Subsystems (2)
cover padata: fix liftime issues after ->serial() has completed
Hi all,
this series is supposed to fix some lifetime issues all related to the fact that once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well with its right to tear down the associated padata_shell or parallel_data instance respectively.
Only the first one, addressed by patch [2/5], has actually been observed, namely on a (downstream) RT kernel under a very specific workload involving LTP's pcrypt_aead01. On non-RT, I've been unable to reproduce.
The remainder of this series, 3-5/5, fixes two more, somewhat related, but purely theoretical issues I spotted when scratching my head about possible reasons for the original Oops.
Thanks!
Nicolai
Nicolai Stange (5):
padata: introduce internal padata_get/put_pd() helpers
padata: make padata_free_shell() to respect pd's ->refcnt
padata: grab parallel_data refcnt for reorder
padata: split out dequeue operation from padata_find_next()
padata: avoid potential UAFs to the padata_shell from padata_reorder()
kernel/padata.c | 129 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 96 insertions(+), 33 deletions(-)
Signature · 1 line
2.37.3
PATCH 1/5 padata: introduce internal padata_get/put_pd() helpers
The next commit in this series will add yet another code site decrementing a struct parallel_data's refcount and invoking dealloction as appropriate.
With that, it's due to provide proper helper functions for managing parallel_datas' refcounts and to convert the existing open-coded refcount manipulation sites.
Implement padata_put_pd() as well as a padata_put_pd_many() for batched releases as needed in padata_serial_worker(). For symmetry reasons, also provide a padata_put_get(), even though its implementation is fairly trivial.
Convert the exisiting open-coded parallel_data ->refcnt manipulation code sites to these new helpers.
--- kernel/padata.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-)
kernel/padata.c +23 -5
@@ -45,6 +45,10 @@ struct padata_mt_job_state { };static void padata_free_pd (struct parallel_data *pd); +static inline void padata_get_pd (struct parallel_data *pd); +static void padata_put_pd_many (struct parallel_data *pd,int cnt); +static inline void padata_put_pd (struct parallel_data *pd); +static void __init padata_mt_helper (struct work_struct *work);static int padata_index_to_cpu (struct parallel_data *pd,int cpu_index)
@@ -198,7 +202,7 @@ int padata_do_parallel(struct padata_shell *ps,if ((pinst->flags &PADATA_RESET )) goto out; -refcount_inc (&pd->refcnt); +padata_get_pd (pd); padata->pd = pd; padata->cb_cpu = *cb_cpu;
@@ -370,8 +374,7 @@ static void padata_serial_worker(struct work_struct *serial_work) }local_bh_enable (); -if (refcount_sub_and_test (cnt, &pd->refcnt)) -padata_free_pd (pd); +padata_put_pd_many (pd, cnt); }/**
@@ -608,6 +611,22 @@ static void padata_free_pd(struct parallel_data *pd)kfree (pd); } +static inline void padata_get_pd (struct parallel_data *pd) +{ +refcount_inc (&pd->refcnt); +} + +static void padata_put_pd_many (struct parallel_data *pd,int cnt) +{ +if (refcount_sub_and_test (cnt, &pd->refcnt)) +padata_free_pd (pd); +} + +static inline void padata_put_pd (struct parallel_data *pd) +{ +padata_put_pd_many (pd,1 ); +} +static void __padata_start (struct padata_instance *pinst) { pinst->flags |=PADATA_INIT ;
@@ -654,8 +673,7 @@ static int padata_replace(struct padata_instance *pinst)synchronize_rcu ();list_for_each_entry_continue_reverse (ps, &pinst->pslist, list) -if (refcount_dec_and_test (&ps->opd->refcnt)) -padata_free_pd (ps->opd); +padata_put_pd (ps->opd); pinst->flags &= ~PADATA_RESET ;
Signature · 1 line
2.37.3
Re: [PATCH 1/5] padata: introduce internal padata_get/put_pd() helpers
Quoted 12 lines · Nicolai Stange
The next commit in this series will add yet another code site decrementing a struct parallel_data's refcount and invoking dealloction as appropriate.
With that, it's due to provide proper helper functions for managing
parallel_datas' refcounts and to convert the existing open-coded refcount manipulation sites.Implement
padata_put_pd()as well as apadata_put_pd_many()for batched releases as needed inpadata_serial_worker(). For symmetry reasons, also provide apadata_put_get(), even though its implementation is fairly trivial.
Quoted 4 lines
Convert the exisiting open-coded
parallel_data->refcnt manipulation code sites to these new helpers.Signed-off-by Nicolai Stange <[email protected]>
Thanks.
PATCH 2/5 padata: make padata_free_shell() to respect pd's ->refcnt
In reply to Nicolai StangeOn a PREEMPT kernel, the following has been observed while running pcrypt_aead01 from LTP:
[ ] general protection fault: 0000 [#1] PREEMPT_RT SMP PTI <...>
[ ] Workqueue: pdecrypt_parallel padata_parallel_worker [ ] RIP: 0010:padata_reorder+0x19/0x120
<...>
[ ] Call Trace:
[ ] padata_parallel_worker+0xa3/0xf0 [ ] process_one_work+0x1db/0x4a0 [ ] worker_thread+0x2d/0x3c0 [ ] ? process_one_work+0x4a0/0x4a0 [ ] kthread+0x159/0x180 [ ] ? kthread_park+0xb0/0xb0 [ ] ret_from_fork+0x35/0x40
The pcrypt_aead01 testcase basically runs a NEWALG/DELALG sequence for some fixed pcrypt instance in a loop, back to back.
The problem is that once the last ->serial() in padata_serial_worker() is getting invoked, the pcrypt requests from the selftests would signal completion, and pcrypt_aead01 can move on and subsequently issue a DELALG. Upon pcrypt instance deregistration, the associated padata_shell would get destroyed, which in turn would unconditionally free the associated parallel_data instance.
If padata_serial_worker() now resumes operation after e.g. having previously been preempted upon the return from the last of those ->serial() callbacks, its subsequent accesses to pd for managing the ->refcnt will all be UAFs. In particular, if the memory backing pd has meanwhile been reused for some new parallel_data allocation, e.g in the course of processing another subsequent NEWALG request, the padata_serial_worker() might find the initial ->refcnt of one and free pd from under that NEWALG or the associated selftests respectively, leading to "secondary" UAFs such as in the Oops above.
Note that as it currently stands, a padata_shell owns a reference on its associated parallel_data already. So fix the UAF in padata_serial_worker() by making padata_free_shell() to not unconditionally free the shell's associated parallel_data, but to properly drop that reference via padata_put_pd() instead.
--- kernel/padata.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/padata.c +1 -1
@@ -1112,7 +1112,7 @@ void padata_free_shell(struct padata_shell *ps)mutex_lock (&ps->pinst->lock);list_del (&ps->list); -padata_free_pd (rcu_dereference_protected (ps->pd,1 )); +padata_put_pd (rcu_dereference_protected (ps->pd,1 ));mutex_unlock (&ps->pinst->lock);kfree (ps);
Signature · 1 line
2.37.3
Re: [PATCH 2/5] padata: make padata_free_shell() to respect pd's ->refcnt
Quoted 37 lines · Nicolai Stange
On a PREEMPT kernel, the following has been observed while running
pcrypt_aead01from LTP:[ ] general protection fault: 0000 [#1]
PREEMPT_RTSMP PTI <...>
[ ] Workqueue:pdecrypt_parallelpadata_parallel_worker[ ] RIP: 0010:padata_reorder+0x19/0x120
<...>
[ ] Call Trace:[ ] padata_parallel_worker+0xa3/0xf0 [ ] process_one_work+0x1db/0x4a0 [ ] worker_thread+0x2d/0x3c0 [ ] ? process_one_work+0x4a0/0x4a0 [ ] kthread+0x159/0x180 [ ] ? kthread_park+0xb0/0xb0 [ ] ret_from_fork+0x35/0x40The
pcrypt_aead01testcase basically runs a NEWALG/DELALG sequence for some fixed pcrypt instance in a loop, back to back.The problem is that once the last ->serial() in
padata_serial_worker()is getting invoked, the pcrypt requests from the selftests would signal completion, andpcrypt_aead01can move on and subsequently issue a DELALG. Upon pcrypt instance deregistration, the associatedpadata_shellwould get destroyed, which in turn would unconditionally free the associatedparallel_datainstance.If
padata_serial_worker()now resumes operation after e.g. having previously been preempted upon the return from the last of those ->serial() callbacks, its subsequent accesses to pd for managing the ->refcnt will all be UAFs. In particular, if the memory backing pd has meanwhile been reused for some newparallel_dataallocation, e.g in the course of processing another subsequent NEWALG request, thepadata_serial_worker()might find the initial ->refcnt of one and free pd from under that NEWALG or the associated selftests respectively, leading to "secondary" UAFs such as in the Oops above.
Quoted 7 lines
Note that as it currently stands, a
padata_shellowns a reference on its associatedparallel_dataalready. So fix the UAF inpadata_serial_worker()by makingpadata_free_shell()to not unconditionally free the shell's associatedparallel_data, but to properly drop that reference viapadata_put_pd()instead.Fixes 07928d9bfc81 ("padata: Remove broken queue flushing")
It looks like this issue goes back to the first padata commit. For instance, pd->refcnt goes to zero after the last _priv is serialized, padata_free is called in another task, and a particularly sluggish padata_reorder call touches pd after.
So wouldn't it be
?
Otherwise,
Quoted 21 lines
Signed-off-by Nicolai Stange <[email protected]>--- kernel/padata.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/padata.c +1
-1@@ -1112,7 +1112,7 @@ void padata_free_shell(struct padata_shell *ps)mutex_lock (&ps->pinst->lock);list_del (&ps->list); -padata_free_pd (rcu_dereference_protected (ps->pd,1 )); +padata_put_pd (rcu_dereference_protected (ps->pd,1 ));mutex_unlock (&ps->pinst->lock);kfree (ps);Signature · 1 line
2.37.3
Daniel Jordan
maintainerRe: [PATCH 2/5] padata: make padata_free_shell() to respect pd's ->refcnt
Quoted 8 lines · Daniel Jordan
Quoted 1 line · Nicolai Stange
Fixes 07928d9bfc81 ("padata: Remove broken queue flushing")It looks like this issue goes back to the first padata commit. For instance,
pd->refcntgoes to zero after the last_privis serialized,padata_freeis called in another task, and a particularly sluggishpadata_reordercall touches pd after.
Quoted 5 lines
So wouldn't it be
Fixes 16295bec6398 ("padata: Generic parallelization/serialization interface")?
I guess not, by my own logic from 2/5. I think it might be
Re: [PATCH 2/5] padata: make padata_free_shell() to respect pd's ->refcnt
In reply to Daniel JordanQuoted 4 lines · Daniel Jordan
Quoted 2 lines · Nicolai Stange
Fixes 07928d9bfc81 ("padata: Remove broken queue flushing")
Quoted 8 lines
It looks like this issue goes back to the first padata commit. For instance,
pd->refcntgoes to zero after the last_privis serialized,padata_freeis called in another task, and a particularly sluggishpadata_reordercall touches pd after.So wouldn't it be
Fixes 16295bec6398 ("padata: Generic parallelization/serialization interface")
I chose 07928d9bfc81 ("padata: Remove broken queue flushing"), because that one reads like it fixed a couple of much more severe padata lifetime issues, it only missed the relatively minor one addressed here, in a sense.
Or to put it the other way around: if one were to backport this patch here, 07928d9bfc81 should probably get picked first, I think.
But I'd be fine with any Fixes tag, of course, I don't have a strong opinion on this matter.
Thanks!
Nicolai
Quoted 9 lines
?
Otherwise,
Acked-by Daniel Jordan <[email protected]>Quoted 2 lines
Signed-off-by Nicolai Stange <[email protected]>---
Signature · 3 lines
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
Daniel Jordan
maintainerRe: [PATCH 2/5] padata: make padata_free_shell() to respect pd's ->refcnt
Quoted 20 lines · Nicolai Stange
Quoted 4 lines · Daniel Jordan
Quoted 2 lines · Nicolai Stange
Fixes 07928d9bfc81 ("padata: Remove broken queue flushing")Quoted 8 lines
It looks like this issue goes back to the first padata commit. For instance,
pd->refcntgoes to zero after the last_privis serialized,padata_freeis called in another task, and a particularly sluggishpadata_reordercall touches pd after.So wouldn't it be
Fixes 16295bec6398 ("padata: Generic parallelization/serialization interface")I chose
07928d9bfc81("padata: Remove broken queue flushing"), because that one reads like it fixed a couple of much more severe padata lifetime issues, it only missed the relatively minor one addressed here, in a sense.
Quoted 5 lines
Or to put it the other way around: if one were to backport this patch here,
07928d9bfc81should probably get picked first, I think.But I'd be fine with any Fixes tag, of course, I don't have a strong opinion on this matter.
Ok, makes sense, your Fixes is fine then. Please put that justification in the changelog for the next version.
PATCH 3/5 padata: grab parallel_data refcnt for reorder
In reply to Nicolai StangeOn entry of padata_do_serial(), the in-flight padata_priv owns a reference to the associated parallel_data instance.
However, as soon as the padata_priv got enqueued on the reorder list, it can be completed from a different context, causing the reference to get released in the course.
This would potentially cause UAFs from the subsequent padata_reorder() operations invoked from the enqueueing padata_do_serial() or from the reorder work.
Note that this is a purely theroretical concern, the problem has never been actually observed -- it would require multiple pcrypt request submissions racing against each other, ultimately a pcrypt instance destruction (DELALG) short after request completions as well as unfortunate timing.
However, for the sake of correctness, it is still worth fixing.
Make padata_do_serial() grab a reference count on the parallel_data for the subsequent reorder operation(s). As long as the padata_priv has not been enqueued, this is safe, because as mentioned above, in-flight pdata_privs own a reference already.
Note that padata_reorder() might schedule another padata_reorder() work and thus, care must be taken to not prematurely release that "reorder refcount" from padata_do_serial() again in case that has happened. Make padata_reorder() return a bool for indicating whether or not a reorder work has been scheduled. Let padata_do_serial() drop its refcount only if this is not the case. Accordingly, make the reorder work handler, invoke_padata_reorder(), drop it then as appropriate.
A remark on the commit chosen for the Fixes tag reference below: before commit bbefa1dd6a6d ("crypto: pcrypt - Avoid deadlock by using per-instance padata queues"), the padata_parallel lifetime had been tied to the padata_instance. The padata_free() resp. padata_stop() issued a synchronize_rcu() before padata_free_pd() from the instance destruction path, rendering UAFs from the padata_do_serial()=>padata_reorder() invocations with BHs disabled impossible AFAICS. With that, the padata_reorder() work remains to be considered. Before
commit b128a3040935 ("padata: allocate workqueue internally"), the workqueue got destroyed (from pcrypt), hence drained, before the padata instance destruction, but this change moved that to after the padata_free_pd() invocation from __padata_free(). So, while the Fixes reference from below is most likely technically correct, I would still like to reiterate that this problem is probably hard to trigger in practice, even more so before commit bbefa1dd6a6d ("crypto: pcrypt - Avoid deadlock by using per-instance padata queues").
--- kernel/padata.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)
kernel/padata.c +21 -5
@@ -275,7 +275,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,return padata; } -static void padata_reorder (struct parallel_data *pd) +static bool padata_reorder (struct parallel_data *pd) {struct padata_instance *pinst = pd->ps->pinst;int cb_cpu;
@@ -294,7 +294,7 @@ static void padata_reorder(struct parallel_data *pd)* care for all the objects enqueued during the holdtime of the lock. */ if (!spin_trylock_bh (&pd->lock)) -return ; +return false;while (1 ) { padata =padata_find_next (pd, true);
@@ -331,17 +331,23 @@ static void padata_reorder(struct parallel_data *pd) reorder =per_cpu_ptr (pd->reorder_list, pd->cpu);if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +return queue_work (pinst->serial_wq, &pd->reorder_work); + +return false; }static void invoke_padata_reorder (struct work_struct *work) {struct parallel_data *pd; +bool keep_refcnt;local_bh_disable (); pd = container_of(work, struct parallel_data, reorder_work); -padata_reorder (pd); + keep_refcnt =padata_reorder (pd);local_bh_enable (); + +if (!keep_refcnt) +padata_put_pd (pd); }static void padata_serial_worker (struct work_struct *serial_work)
@@ -392,6 +398,15 @@ void padata_do_serial(struct padata_priv *padata)struct padata_list *reorder =per_cpu_ptr (pd->reorder_list, hashed_cpu);struct padata_priv *cur; +/* +* The in-flight padata owns a reference on pd. However, as +* soon as it's been enqueued on the reorder list, another +* task can dequeue and complete it, thereby dropping the +* reference. Grab another reference here, it will eventually +* be released from a reorder work, if any, or below. +*/ +padata_get_pd (pd); +spin_lock (&reorder->lock);/* Sort in ascending order of sequence number. */ list_for_each_entry_reverse (cur, &reorder->list, list)
@@ -407,7 +422,8 @@ void padata_do_serial(struct padata_priv *padata)*/ smp_mb (); -padata_reorder (pd); +if (!padata_reorder (pd)) +padata_put_pd (pd); }EXPORT_SYMBOL (padata_do_serial);
Signature · 1 line
2.37.3
Daniel Jordan
maintainerRe: [PATCH 3/5] padata: grab parallel_data refcnt for reorder
Quoted 119 lines · Nicolai Stange
On entry of
padata_do_serial(), the in-flightpadata_privowns a reference to the associatedparallel_datainstance.However, as soon as the
padata_privgot enqueued on the reorder list, it can be completed from a different context, causing the reference to get released in the course.This would potentially cause UAFs from the subsequent
padata_reorder()operations invoked from the enqueueingpadata_do_serial()or from the reorder work.Note that this is a purely theroretical concern, the problem has never been actually observed -- it would require multiple pcrypt request submissions racing against each other, ultimately a pcrypt instance destruction (DELALG) short after request completions as well as unfortunate timing.
However, for the sake of correctness, it is still worth fixing.
Make
padata_do_serial()grab a reference count on theparallel_datafor the subsequent reorder operation(s). As long as thepadata_privhas not been enqueued, this is safe, because as mentioned above, in-flightpdata_privsown a reference already.Note that
padata_reorder()might schedule anotherpadata_reorder()work and thus, care must be taken to not prematurely release that "reorder refcount" frompadata_do_serial()again in case that has happened. Makepadata_reorder()return a bool for indicating whether or not a reorder work has been scheduled. Letpadata_do_serial()drop its refcount only if this is not the case. Accordingly, make the reorder work handler,invoke_padata_reorder(), drop it then as appropriate.A remark on the commit chosen for the Fixes tag reference below: before commit
bbefa1dd6a6d("crypto: pcrypt - Avoid deadlock by using per-instance padata queues"), thepadata_parallellifetime had been tied to thepadata_instance. Thepadata_free()resp.padata_stop()issued asynchronize_rcu()beforepadata_free_pd()from the instance destruction path, rendering UAFs from the padata_do_serial()=>padata_reorder() invocations with BHs disabled impossible AFAICS. With that, thepadata_reorder()work remains to be considered. Before
commitb128a3040935("padata: allocate workqueue internally"), the workqueue got destroyed (from pcrypt), hence drained, before the padata instance destruction, but this change moved that to after thepadata_free_pd()invocation from__padata_free(). So, while the Fixes reference from below is most likely technically correct, I would still like to reiterate that this problem is probably hard to trigger in practice, even more so before commitbbefa1dd6a6d("crypto: pcrypt - Avoid deadlock by using per-instance padata queues").Fixes b128a3040935 ("padata: allocate workqueue internally")Signed-off-by Nicolai Stange <[email protected]>--- kernel/padata.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)kernel/padata.c +19
-4@@ -275,7 +275,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,return padata; } -static void padata_reorder (struct parallel_data *pd) +static bool padata_reorder (struct parallel_data *pd) {struct padata_instance *pinst = pd->ps->pinst;int cb_cpu;@@ -294,7 +294,7 @@ static void padata_reorder(struct parallel_data *pd)* care for all the objects enqueued during the holdtime of the lock. */ if (!spin_trylock_bh (&pd->lock)) -return ; +return false;while (1 ) { padata =padata_find_next (pd, true);@@ -331,17 +331,23 @@ static void padata_reorder(struct parallel_data *pd) reorder =per_cpu_ptr (pd->reorder_list, pd->cpu);if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +return queue_work (pinst->serial_wq, &pd->reorder_work); + +return false; }static void invoke_padata_reorder (struct work_struct *work) {struct parallel_data *pd; +bool keep_refcnt;local_bh_disable (); pd = container_of(work, struct parallel_data, reorder_work); -padata_reorder (pd); + keep_refcnt =padata_reorder (pd);local_bh_enable (); + +if (!keep_refcnt) +padata_put_pd (pd); }static void padata_serial_worker (struct work_struct *serial_work)@@ -392,6 +398,15 @@ void padata_do_serial(struct padata_priv *padata)struct padata_list *reorder =per_cpu_ptr (pd->reorder_list, hashed_cpu);struct padata_priv *cur; +/* +* The in-flight padata owns a reference on pd. However, as +* soon as it's been enqueued on the reorder list, another +* task can dequeue and complete it, thereby dropping the +* reference. Grab another reference here, it will eventually +* be released from a reorder work, if any, or below. +*/ +padata_get_pd (pd); +spin_lock (&reorder->lock);/* Sort in ascending order of sequence number. */ list_for_each_entry_reverse (cur, &reorder->list, list)
kernel/padata.c void padata_do_serial(struct padata_priv *padata) · line 426
Quoted 6 lines
kernel/padata.c +2
-1@@ -407,7 +422,8 @@ void padata_do_serial(struct padata_priv *padata)*/ smp_mb (); -padata_reorder (pd); +if (!padata_reorder (pd)) +padata_put_pd (pd);
do_serial is supposed to be called with BHs disabled and will be in every case after a fix for a separate issue that I plan to send this cycle. Given that it (will soon...) always happen under RCU protection, part of this issue could be addressed like this, which puts the expense of dealing with this rare problem in the slow path:
kernel/padata.c +6 -0
@@ -1110,6 +1110,12 @@ void padata_free_shell(struct padata_shell *ps)if (!ps)return ; +/* +* Wait for all _do_serial calls to finish to avoid touching freed pd's +* and ps's. +*/ +synchronize_rcu (); +mutex_lock (&ps->pinst->lock);list_del (&ps->list);padata_put_pd (rcu_dereference_protected (ps->pd,1 ));
pcrypt calls padata_free_shell() when all outstanding transforms (and thus requests, I think) have been freed/completed, so no new task can come into padata_reorder. synchronize_rcu() then flushes out any remaining _reorder calls.
This doesn't deal with pending reorder_work items, but we can fix it later in the series.
What do you think?
Re: [PATCH 3/5] padata: grab parallel_data refcnt for reorder
Quoted 12 lines · Daniel Jordan
Quoted 11 lines · Nicolai Stange
On entry of
padata_do_serial(), the in-flightpadata_privowns a reference to the associatedparallel_datainstance.However, as soon as the
padata_privgot enqueued on the reorder list, it can be completed from a different context, causing the reference to get released in the course.This would potentially cause UAFs from the subsequent
padata_reorder()operations invoked from the enqueueingpadata_do_serial()or from the reorder work.
<snip>
Quoted 105 lines
Quoted 69 lines
--- kernel/padata.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)kernel/padata.c +19
-4@@ -275,7 +275,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,return padata; } -static void padata_reorder (struct parallel_data *pd) +static bool padata_reorder (struct parallel_data *pd) {struct padata_instance *pinst = pd->ps->pinst;int cb_cpu;@@ -294,7 +294,7 @@ static void padata_reorder(struct parallel_data *pd)* care for all the objects enqueued during the holdtime of the lock. */ if (!spin_trylock_bh (&pd->lock)) -return ; +return false;while (1 ) { padata =padata_find_next (pd, true);@@ -331,17 +331,23 @@ static void padata_reorder(struct parallel_data *pd) reorder =per_cpu_ptr (pd->reorder_list, pd->cpu);if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +return queue_work (pinst->serial_wq, &pd->reorder_work); + +return false; }static void invoke_padata_reorder (struct work_struct *work) {struct parallel_data *pd; +bool keep_refcnt;local_bh_disable (); pd = container_of(work, struct parallel_data, reorder_work); -padata_reorder (pd); + keep_refcnt =padata_reorder (pd);local_bh_enable (); + +if (!keep_refcnt) +padata_put_pd (pd); }static void padata_serial_worker (struct work_struct *serial_work)@@ -392,6 +398,15 @@ void padata_do_serial(struct padata_priv *padata)struct padata_list *reorder =per_cpu_ptr (pd->reorder_list, hashed_cpu);struct padata_priv *cur; +/* +* The in-flight padata owns a reference on pd. However, as +* soon as it's been enqueued on the reorder list, another +* task can dequeue and complete it, thereby dropping the +* reference. Grab another reference here, it will eventually +* be released from a reorder work, if any, or below. +*/ +padata_get_pd (pd); +spin_lock (&reorder->lock);/* Sort in ascending order of sequence number. */ list_for_each_entry_reverse (cur, &reorder->list, list)kernel/padata.c void padata_do_serial(struct padata_priv *padata) · line 426
Quoted 6 lines
kernel/padata.c +2
-1@@ -407,7 +422,8 @@ void padata_do_serial(struct padata_priv *padata)*/ smp_mb (); -padata_reorder (pd); +if (!padata_reorder (pd)) +padata_put_pd (pd);
do_serialis supposed to be called with BHs disabled and will be in every case after a fix for a separate issue that I plan to send this cycle. Given that it (will soon...) always happen under RCU protection, part of this issue could be addressed like this, which puts the expense of dealing with this rare problem in the slow path:kernel/padata.c +6
-0@@ -1110,6 +1110,12 @@ void padata_free_shell(struct padata_shell *ps)if (!ps)return ; +/* +* Wait for all _do_serial calls to finish to avoid touching freed pd's +* and ps's. +*/ +synchronize_rcu (); +mutex_lock (&ps->pinst->lock);list_del (&ps->list);padata_put_pd (rcu_dereference_protected (ps->pd,1 ));pcrypt calls
padata_free_shell()when all outstanding transforms (and thus requests, I think) have been freed/completed, so no new task can come intopadata_reorder.synchronize_rcu()then flushes out any remaining_reordercalls.
Quoted 4 lines
This doesn't deal with pending
reorder_workitems, but we can fix it later in the series.What do you think?
Yes, I think that would work. Will you handle it alongside that fix for a separate issue you mentioned above? Or shall I once this fix has landed?
Thanks!
Nicolai
Signature · 3 lines
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
Daniel Jordan
maintainerRe: [PATCH 3/5] padata: grab parallel_data refcnt for reorder
Quoted 122 lines · Nicolai Stange
Quoted 12 lines · Daniel Jordan
Quoted 11 lines · Nicolai Stange
On entry of
padata_do_serial(), the in-flightpadata_privowns a reference to the associatedparallel_datainstance.However, as soon as the
padata_privgot enqueued on the reorder list, it can be completed from a different context, causing the reference to get released in the course.This would potentially cause UAFs from the subsequent
padata_reorder()operations invoked from the enqueueingpadata_do_serial()or from the reorder work.<snip>
Quoted 105 lines
Quoted 69 lines
--- kernel/padata.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)kernel/padata.c +19
-4@@ -275,7 +275,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,return padata; } -static void padata_reorder (struct parallel_data *pd) +static bool padata_reorder (struct parallel_data *pd) {struct padata_instance *pinst = pd->ps->pinst;int cb_cpu;@@ -294,7 +294,7 @@ static void padata_reorder(struct parallel_data *pd)* care for all the objects enqueued during the holdtime of the lock. */ if (!spin_trylock_bh (&pd->lock)) -return ; +return false;while (1 ) { padata =padata_find_next (pd, true);@@ -331,17 +331,23 @@ static void padata_reorder(struct parallel_data *pd) reorder =per_cpu_ptr (pd->reorder_list, pd->cpu);if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +return queue_work (pinst->serial_wq, &pd->reorder_work); + +return false; }static void invoke_padata_reorder (struct work_struct *work) {struct parallel_data *pd; +bool keep_refcnt;local_bh_disable (); pd = container_of(work, struct parallel_data, reorder_work); -padata_reorder (pd); + keep_refcnt =padata_reorder (pd);local_bh_enable (); + +if (!keep_refcnt) +padata_put_pd (pd); }static void padata_serial_worker (struct work_struct *serial_work)@@ -392,6 +398,15 @@ void padata_do_serial(struct padata_priv *padata)struct padata_list *reorder =per_cpu_ptr (pd->reorder_list, hashed_cpu);struct padata_priv *cur; +/* +* The in-flight padata owns a reference on pd. However, as +* soon as it's been enqueued on the reorder list, another +* task can dequeue and complete it, thereby dropping the +* reference. Grab another reference here, it will eventually +* be released from a reorder work, if any, or below. +*/ +padata_get_pd (pd); +spin_lock (&reorder->lock);/* Sort in ascending order of sequence number. */ list_for_each_entry_reverse (cur, &reorder->list, list)kernel/padata.c void padata_do_serial(struct padata_priv *padata) · line 426
Quoted 6 lines
kernel/padata.c +2
-1@@ -407,7 +422,8 @@ void padata_do_serial(struct padata_priv *padata)*/ smp_mb (); -padata_reorder (pd); +if (!padata_reorder (pd)) +padata_put_pd (pd);
do_serialis supposed to be called with BHs disabled and will be in every case after a fix for a separate issue that I plan to send this cycle. Given that it (will soon...) always happen under RCU protection, part of this issue could be addressed like this, which puts the expense of dealing with this rare problem in the slow path:kernel/padata.c +6
-0@@ -1110,6 +1110,12 @@ void padata_free_shell(struct padata_shell *ps)if (!ps)return ; +/* +* Wait for all _do_serial calls to finish to avoid touching freed pd's +* and ps's. +*/ +synchronize_rcu (); +mutex_lock (&ps->pinst->lock);list_del (&ps->list);padata_put_pd (rcu_dereference_protected (ps->pd,1 ));pcrypt calls
padata_free_shell()when all outstanding transforms (and thus requests, I think) have been freed/completed, so no new task can come intopadata_reorder.synchronize_rcu()then flushes out any remaining_reordercalls.
Quoted 8 lines
Quoted 4 lines
This doesn't deal with pending
reorder_workitems, but we can fix it later in the series.What do you think?
Yes, I think that would work. Will you handle it alongside that fix for a separate issue you mentioned above? Or shall I once this fix has landed?
Please go ahead and do it yourself. I'll send mine soon, I think it should be ok for them to go in in the same cycle.
Herbert Xu
maintainerRe: [PATCH 3/5] padata: grab parallel_data refcnt for reorder
In reply to Nicolai StangeQuoted 10 lines · Nicolai Stange
On entry of
padata_do_serial(), the in-flightpadata_privowns a reference to the associatedparallel_datainstance.However, as soon as the
padata_privgot enqueued on the reorder list, it can be completed from a different context, causing the reference to get released in the course.This would potentially cause UAFs from the subsequent
padata_reorder()operations invoked from the enqueueingpadata_do_serial()or from the reorder work.
I've just received yet another report of a UAF in padata on s390 while running the pcrypt_aead01 LTP test. The kernel in question contains an identical version of padata to upstream. Perhaps it's time to revisit the life-time issues in padata?
[ 9463.217163] ==================================================================
[ 9463.217339] BUG: KASAN: slab-use-after-free in work_is_static_object+0x2a/0x68
[ 9463.217678] Read of size 8 at addr 0000000098917c38 by task kworker/u4:2/59
[ 9463.217798]
[ 9463.217804] CPU: 0 PID: 59 Comm: kworker/u4:2 Kdump: loaded Not tainted 5.14.0-454.el9.s390x+debug #1
[ 9463.218051] Hardware name: IBM 8561 LT1 400 (KVM/Linux)
[ 9463.218058] Workqueue: pdecrypt_parallel padata_parallel_worker
[ 9463.218119] Call Trace:
[ 9463.218123] [<0000000001cbe122>] dump_stack_lvl+0xfa/0x150
[ 9463.218226] [<0000000001cada10>] print_address_description.constprop.0+0x180/0x430
[ 9463.218236] [<0000000001caddc2>] print_report+0x102/0x1f0
[ 9463.218243] [<00000000009c075a>] kasan_report+0xb2/0x1f8
[ 9463.218251] [<00000000009c1af2>] kasan_check_range+0x17a/0x1c0
[ 9463.218302] [<0000000000235b62>] work_is_static_object+0x2a/0x68
[ 9463.218308] [<0000000001002036>] lookup_object_or_alloc.part.0+0x12e/0x1a8
[ 9463.218373] [<0000000001002da2>] debug_object_activate+0x20a/0x450
[ 9463.218381] [<0000000000240456>] __queue_work+0x646/0x1128
[ 9463.218443] [<0000000000241080>] queue_work_on+0x148/0x150
[ 9463.218506] [<00000000007d707e>] padata_parallel_worker+0x76/0xc8
[ 9463.218988] [<0000000000242040>] process_one_work+0x978/0x1748
[ 9463.218994] [<00000000002433b4>] worker_thread+0x5a4/0xf58
[ 9463.219000] [<000000000025d60c>] kthread+0x2a4/0x358
[ 9463.219008] [<000000000010601a>] __ret_from_fork+0x8a/0xe8
[ 9463.219016] [<0000000001cf343a>] ret_from_fork+0xa/0x30
[ 9463.219086] 5 locks held by kworker/u4:2/59:
[ 9463.219143] #0: 00000000ef82a948 ((wq_completion)pdecrypt_parallel){+.+.}-{0:0}, at: process_one_work+0x830/0x1748
[ 9463.219204] #1: 001bff8000b0fcf8 ((work_completion)(&pw->pw_work)){+.+.}-{0:0}, at: process_one_work+0x830/0x1748
[ 9463.219215] #2: 0000000002a93160 (rcu_read_lock){....}-{1:2}, at: __queue_work+0xa0/0x1128
[ 9463.219337] #3: 000000017fa0f618 (&pool->lock){-.-.}-{2:2}, at: __queue_work+0x314/0x1128
[ 9463.219399] #4: 0000000005d7a0c0 (&obj_hash[i].lock){-.-.}-{2:2}, at: debug_object_activate+0x166/0x450
[ 9463.219446]
[ 9463.219448] Allocated by task 397970:
[ 9463.219452] kasan_save_stack+0x34/0x58
[ 9463.219457] kasan_set_track+0x36/0x48
[ 9463.219460] __kasan_kmalloc+0xaa/0xc0
[ 9463.219463] padata_alloc_pd+0x70/0xbc8
[ 9463.219466] padata_alloc_shell+0x8a/0x208
[ 9463.219469] pcrypt_create_aead.isra.0+0xba/0x698 [pcrypt]
[ 9463.219535] cryptomgr_probe+0x9e/0x1d8
[ 9463.219578] kthread+0x2a4/0x358
[ 9463.219582] __ret_from_fork+0x8a/0xe8
[ 9463.219586] ret_from_fork+0xa/0x30
[ 9463.219589]
[ 9463.219591] Freed by task 392422:
[ 9463.219594] kasan_save_stack+0x34/0x58
[ 9463.219598] kasan_set_track+0x36/0x48
[ 9463.219645] kasan_save_free_info+0x46/0x68
[ 9463.219651] __kasan_slab_free+0x102/0x178
[ 9463.219656] slab_free_freelist_hook+0x148/0x228
[ 9463.219685] __kmem_cache_free+0x92/0x2b8
[ 9463.219690] padata_free_shell+0x242/0x2f8
[ 9463.219697] pcrypt_free+0x50/0xa0 [pcrypt]
[ 9463.219703] crypto_destroy_instance+0x86/0xe0
[ 9463.219709] crypto_mod_put+0xa4/0xf0
[ 9463.219714] crypto_del_alg+0xc8/0x138
[ 9463.219720] crypto_user_rcv_msg+0x39a/0x490
[ 9463.219725] netlink_rcv_skb+0x21a/0x4a0
[ 9463.219773] crypto_netlink_rcv+0x40/0x58
[ 9463.219781] netlink_unicast+0x46a/0x6e0
[ 9463.219787] netlink_sendmsg+0x75c/0xbe8
[ 9463.219835] __sock_sendmsg+0xca/0x170
[ 9463.219843] ____sys_sendmsg+0x5d8/0x730
[ 9463.219848] ___sys_sendmsg+0x11e/0x178
[ 9463.219855] __sys_sendmsg+0xda/0x150
[ 9463.219915] __do_sys_socketcall+0x3e2/0x480
[ 9463.220092] do_syscall+0x22c/0x328
[ 9463.220097] __do_syscall+0xce/0xf0
[ 9463.220103] system_call+0x70/0x98
[ 9463.220109]
[ 9463.220148] Last potentially related work creation:
[ 9463.220152] kasan_save_stack+0x34/0x58
[ 9463.220157] __kasan_record_aux_stack+0xba/0xd0
[ 9463.221000] __call_rcu_common.constprop.0+0x104/0x998
[ 9463.221070] __key_link+0x124/0x320
[ 9463.221380] construct_alloc_key+0x53a/0x618
[ 9463.221386] construct_key_and_link+0x196/0x3a8
[ 9463.221506] request_key_and_link+0x38a/0x3e0
[ 9463.221512] __do_sys_request_key+0x164/0x268
[ 9463.221518] do_syscall+0x22c/0x328
[ 9463.221617] __do_syscall+0xce/0xf0
[ 9463.221622] system_call+0x70/0x98
[ 9463.221627]
[ 9463.221630] Second to last potentially related work creation:
[ 9463.221634] kasan_save_stack+0x34/0x58
[ 9463.221638] __kasan_record_aux_stack+0xba/0xd0
[ 9463.221642] __call_rcu_common.constprop.0+0x104/0x998
[ 9463.221743] assoc_array_gc+0xde6/0x10f8
[ 9463.221747] keyring_gc+0x170/0x210
[ 9463.221751] key_garbage_collector+0x598/0x760
[ 9463.221756] process_one_work+0x978/0x1748
[ 9463.221759] worker_thread+0x71a/0xf58
[ 9463.221763] kthread+0x2a4/0x358
[ 9463.221767] __ret_from_fork+0x8a/0xe8
[ 9463.221770] ret_from_fork+0xa/0x30
[ 9463.221774]
[ 9463.221836] The buggy address belongs to the object at 0000000098917c00
[ 9463.221836] which belongs to the cache kmalloc-512 of size 512
[ 9463.221887] The buggy address is located 56 bytes inside of
[ 9463.221887] freed 512-byte region [0000000098917c00, 0000000098917e00)
[ 9463.221891]
[ 9463.221894] The buggy address belongs to the physical page:
[ 9463.221898] page:00000000e2f31a4b refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x98914
[ 9463.222571] head:00000000e2f31a4b order:2 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[ 9463.222577] flags: 0x2000000000010200(slab|head|node=0|zone=1)
[ 9463.222585] page_type: 0xffffffff()
[ 9463.222592] raw: 2000000000010200 0000000080083400 0000000000000000 0000000100000122
[ 9463.222599] raw: 0000000000000000 0010002000000000 ffffffff00000001 0000000000000000
[ 9463.222604] page dumped because: kasan: bad access detected
[ 9463.222648]
[ 9463.222651] Memory state around the buggy address:
[ 9463.222660] 0000000098917b00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 9463.222665] 0000000098917b80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 9463.222670] >0000000098917c00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9463.222795] ^
[ 9463.222838] 0000000098917c80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9463.222843] 0000000098917d00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9463.222849] ==================================================================
[ 9463.222853] Disabling lock debugging due to kernel taint
[ 9463.222858] ------------[ cut here ]------------
[ 9463.222862] ODEBUG: activate not available (active state 0) object: 00000000990ad955 object type: work_struct hint: invoke_padata_reorder+0x0/0x58
[ 9463.223373] WARNING: CPU: 0 PID: 59 at lib/debugobjects.c:514 debug_print_object+0x1be/0x248
[ 9463.223549] Modules linked in: pcrypt zstd zstd_compress zsmalloc exfat vfat fat overlay ext4 mbcache jbd2 loop tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc vfio_ccw mdev vfio_iommu_type1 vfio iommufd drm i2c_core drm_panel_orientation_quirks fuse xfs libcrc32c ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 virtio_net net_failover failover virtio_blk sha3_256_s390 dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt [last unloaded: zram]
[ 9463.223743] CPU: 0 PID: 59 Comm: kworker/u4:2 Kdump: loaded Tainted: G B ------- --- 5.14.0-454.el9.s390x+debug #1
[ 9463.223747] Hardware name: IBM 8561 LT1 400 (KVM/Linux)
[ 9463.223748] Workqueue: pdecrypt_parallel padata_parallel_worker
[ 9463.223751] Krnl PSW : 0404e00180000000 00000000010004aa (debug_print_object+0x1c2/0x248)
[ 9463.223756] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
[ 9463.223789] Krnl GPRS: 0000000000000001 001c000000000000 0000000000000086 0000000000000004
[ 9463.223791] 0000000000000001 00000000006289f8 0000000098917c38 0000000001fccca0
[ 9463.223793] 00000000007d7dc8 00000000029243e0 0000000001fcc780 001bff8000b0f9c8
[ 9463.223795] 00000000028e3eb0 0000000000000000 00000000010004a6 001bff8000b0f868
[ 9463.223804] Krnl Code: 000000000100049a: c020007e5e13 larl %r2,0000000001fcc0c0
[ 9463.223804] 00000000010004a0: c0e5ff8ee004 brasl %r14,00000000001dc4a8
[ 9463.223804] #00000000010004a6: af000000 mc 0,0
[ 9463.223804] >00000000010004aa: c41d00c06113 lrl %r1,000000000280c6d0
[ 9463.223804] 00000000010004b0: a71a0001 ahi %r1,1
[ 9463.223804] 00000000010004b4: c41f00c0610e strl %r1,000000000280c6d0
[ 9463.223804] 00000000010004ba: eb6ff0b00004 lmg %r6,%r15,176(%r15)
[ 9463.223804] 00000000010004c0: 07fe bcr 15,%r14
[ 9463.223818] Call Trace:
[ 9463.223819] [<00000000010004aa>] debug_print_object+0x1c2/0x248
[ 9463.223823] ([<00000000010004a6>] debug_print_object+0x1be/0x248)
[ 9463.223852] [<0000000001002f5e>] debug_object_activate+0x3c6/0x450
[ 9463.223854] [<0000000000240456>] __queue_work+0x646/0x1128
[ 9463.223856] [<0000000000241080>] queue_work_on+0x148/0x150
[ 9463.223858] [<00000000007d707e>] padata_parallel_worker+0x76/0xc8
[ 9463.223861] [<0000000000242040>] process_one_work+0x978/0x1748
[ 9463.223864] [<00000000002433b4>] worker_thread+0x5a4/0xf58
[ 9463.223866] [<000000000025d60c>] kthread+0x2a4/0x358
[ 9463.223868] [<000000000010601a>] __ret_from_fork+0x8a/0xe8
[ 9463.223871] [<0000000001cf343a>] ret_from_fork+0xa/0x30
[ 9463.223873] INFO: lockdep is turned off.
[ 9463.223874] Last Breaking-Event-Address:
[ 9463.223875] [<00000000001dc64a>] __warn_printk+0x1a2/0x1a8
[ 9463.223878] irq event stamp: 24602747
[ 9463.223879] hardirqs last enabled at (24602746): [<00000000001f4ac0>] __local_bh_enable_ip+0x188/0x348
[ 9463.223884] hardirqs last disabled at (24602747): [<0000000000241042>] queue_work_on+0x10a/0x150
[ 9463.223886] softirqs last enabled at (24602712): [<00000000007d7072>] padata_parallel_worker+0x6a/0xc8
[ 9463.223917] softirqs last disabled at (24602720): [<00000000007d7042>] padata_parallel_worker+0x3a/0xc8
[ 9463.223920] ---[ end trace 0000000000000000 ]---
[ 9463.224330] ------------[ cut here ]------------
[ 9463.224394] ODEBUG: deactivate not available (active state 0) object: 00000000990ad955 object type: work_struct hint: invoke_padata_reorder+0x0/0x58
[ 9463.224520] WARNING: CPU: 0 PID: 306462 at lib/debugobjects.c:514 debug_print_object+0x1be/0x248
[ 9463.224526] Modules linked in: pcrypt zstd zstd_compress zsmalloc exfat vfat fat overlay ext4 mbcache jbd2 loop tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc vfio_ccw mdev vfio_iommu_type1 vfio iommufd drm i2c_core drm_panel_orientation_quirks fuse xfs libcrc32c ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 virtio_net net_failover failover virtio_blk sha3_256_s390 dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt [last unloaded: zram]
[ 9463.224781] CPU: 0 PID: 306462 Comm: kworker/0:0 Kdump: loaded Tainted: G B W ------- --- 5.14.0-454.el9.s390x+debug #1
[ 9463.224897] Hardware name: IBM 8561 LT1 400 (KVM/Linux)
[ 9463.224898] Workqueue: 0x0 (events)
[ 9463.224907] Krnl PSW : 0404e00180000000 00000000010004aa (debug_print_object+0x1c2/0x248)
[ 9463.224949] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
[ 9463.224953] Krnl GPRS: 0000000000000001 001c000000000000 0000000000000088 0000000000000004
[ 9463.224956] 0000000000000001 00000000006289f8 0000000098917c38 0000000001fccca0
[ 9463.224959] 00000000007d7dc8 00000000029243e0 0000000001fcc6c0 001bff8005b6fb58
[ 9463.224961] 00000000028e3eb0 0000000000000000 00000000010004a6 001bff8005b6fa10
[ 9463.224968] Krnl Code: 000000000100049a: c020007e5e13 larl %r2,0000000001fcc0c0
[ 9463.224968] 00000000010004a0: c0e5ff8ee004 brasl %r14,00000000001dc4a8
[ 9463.224968] #00000000010004a6: af000000 mc 0,0
[ 9463.224968] >00000000010004aa: c41d00c06113 lrl %r1,000000000280c6d0
[ 9463.224968] 00000000010004b0: a71a0001 ahi %r1,1
[ 9463.224968] 00000000010004b4: c41f00c0610e strl %r1,000000000280c6d0
[ 9463.224968] 00000000010004ba: eb6ff0b00004 lmg %r6,%r15,176(%r15)
[ 9463.224968] 00000000010004c0: 07fe bcr 15,%r14
[ 9463.225011] Call Trace:
[ 9463.225012] [<00000000010004aa>] debug_print_object+0x1c2/0x248
[ 9463.225022] ([<00000000010004a6>] debug_print_object+0x1be/0x248)
[ 9463.225025] [<0000000001002546>] debug_object_deactivate+0x1c6/0x2f8
[ 9463.225029] [<0000000000241a86>] process_one_work+0x3be/0x1748
[ 9463.225032] [<00000000002433b4>] worker_thread+0x5a4/0xf58
[ 9463.225035] [<000000000025d60c>] kthread+0x2a4/0x358
[ 9463.225038] [<000000000010601a>] __ret_from_fork+0x8a/0xe8
[ 9463.225101] [<0000000001cf343a>] ret_from_fork+0xa/0x30
[ 9463.225294] INFO: lockdep is turned off.
[ 9463.225357] Last Breaking-Event-Address:
[ 9463.225357] [<00000000001dc64a>] __warn_printk+0x1a2/0x1a8
[ 9463.225543] irq event stamp: 2298862
[ 9463.225544] hardirqs last enabled at (2298861): [<0000000001cf2cc2>] _raw_spin_unlock_irq+0x92/0xb0
[ 9463.225549] hardirqs last disabled at (2298862): [<0000000001cd7e60>] __schedule+0xb20/0x1478
[ 9463.225585] softirqs last enabled at (2292304): [<0000000001cf5c12>] __do_softirq+0xaaa/0x1040
[ 9463.225589] softirqs last disabled at (2291215): [<00000000001f3bf2>] __irq_exit_rcu+0x2fa/0x3b8
[ 9463.225593] ---[ end trace 0000000000000000 ]---
[ 9463.225606] Unable to handle kernel pointer dereference in virtual kernel address space
[ 9463.225847] Failing address: 0005fce7876cc000 TEID: 0005fce7876cc403
[ 9463.225853] Fault in home space mode while using kernel ASCE.
[ 9463.225863] AS:0000000003b2800b R2:0000000000000028
[ 9463.226167] Oops: 003a ilc:2 [#1] SMP
[ 9463.226174] Modules linked in: pcrypt zstd zstd_compress zsmalloc exfat vfat fat overlay ext4 mbcache jbd2 loop tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc vfio_ccw mdev vfio_iommu_type1 vfio iommufd drm i2c_core drm_panel_orientation_quirks fuse xfs libcrc32c ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 virtio_net net_failover failover virtio_blk sha3_256_s390 dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt [last unloaded: zram]
[ 9463.226373] CPU: 0 PID: 306462 Comm: kworker/0:0 Kdump: loaded Tainted: G B W ------- --- 5.14.0-454.el9.s390x+debug #1
[ 9463.226376] Hardware name: IBM 8561 LT1 400 (KVM/Linux)
[ 9463.226378] Workqueue: pdecrypt_serial invoke_padata_reorder
[ 9463.226381] Krnl PSW : 0704c00180000000 0000000000361e96 (do_raw_spin_trylock+0x46/0x148)
[ 9463.226427] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[ 9463.226430] Krnl GPRS: 0000000000000001 001c000000000000 0005fce700000001 001c000000000000
[ 9463.226433] 0000000000000000 00000000003583a8 0000000000000001 0005fce7876cc850
[ 9463.226435] 0000000098917c24 0005fce7876cc868 0005fce7876cc850 0005fce7876cc850
[ 9463.226438] 00000000028e3eb0 0000000013122f84 001bff8005b6f9b0 001bff8005b6f970
[ 9463.226445] Krnl Code: 0000000000361e8a: a7740065 brc 7,0000000000361f54
[ 9463.226445] 0000000000361e8e: 582003ac l %r2,940
[ 9463.226445] #0000000000361e92: a7180000 lhi %r1,0
[ 9463.226445] >0000000000361e96: ba12b000 cs %r1,%r2,0(%r11)
[ 9463.226445] 0000000000361e9a: a7a80001 lhi %r10,1
[ 9463.226445] 0000000000361e9e: ec160050007e cij %r1,0,6,0000000000361f3e
[ 9463.226445] 0000000000361ea4: a51c001c llihh %r1,28
[ 9463.226445] KernelAddressSanitizer initializedThanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
PATCH 4/5 padata: split out dequeue operation from padata_find_next()
In reply to Nicolai StangeCurrently, padata_find_next() takes a 'remove_object' argument for specifying whether the caller wants the returned patada_priv, if any, to get removed from the percpu reorder list it's been found on.
There are only two callsites, both from padata_reorder():
- one supposed to dequeue the padata_priv instances to be processed in a loop, i.e. it has the 'remove_object' set to true, and
- another one near the end of padata_reorder() with 'remove_object' set to false for checking whether the reorder work needs to get rescheduled.
In order to deal with lifetime issues, a future commit will need to move this latter reorder work scheduling operation to under the reorder->lock, where pd->ps is guaranteed to exist as long as there are any padata_privs to process. However this lock is currently taken within padata_find_next().
In order to be able to extend the reorder->lock to beyond the call to padata_find_next() from padata_reorder(), a variant where the caller may grab it for the callee shall be provided.
Split padata_find_next() into two parts:
- __padata_find_next(), which expects the caller to hold the reorder->lock and only returns the found padata_priv, if any, without removing it from the queue.
- padata_dequeue_next(), with functionality equivalent to the former padata_find_next(pd, remove_object=true) and implemented by means of the factored out __padata_find_next().
Adapt the two callsites in padata_reorder() as appropriate.
There is no change in functionality.
--- kernel/padata.c | 57 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 37 insertions(+), 20 deletions(-)
kernel/padata.c +37 -20
@@ -230,29 +230,24 @@ int padata_do_parallel(struct padata_shell *ps,EXPORT_SYMBOL (padata_do_parallel);/* -* padata_find_next - Find the next object that needs serialization. +* __padata_find_next - Find the next object that needs serialization. * * Return: * * A pointer to the control struct of the next object that needs -* serialization, if present in one of the percpu reorder queues. +* serialization, if already present on the given percpu reorder queue. * * NULL, if the next object that needs serialization will * be parallel processed by another cpu and is not yet present in -* the cpu's reorder queue. +* the reorder queue. */ -static struct padata_priv *padata_find_next (struct parallel_data *pd, -bool remove_object) +static struct padata_priv *__padata_find_next (struct parallel_data *pd, +struct padata_list *reorder) {struct padata_priv *padata; -struct padata_list *reorder; -int cpu = pd->cpu; - reorder =per_cpu_ptr (pd->reorder_list, cpu); +lockdep_assert_held (&reorder->lock); -spin_lock (&reorder->lock); -if (list_empty (&reorder->list)) { -spin_unlock (&reorder->lock); +if (list_empty (&reorder->list))return NULL ; - } padata = list_entry(reorder->list.next, struct padata_priv, list);
@@ -260,16 +255,30 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,* Checks the rare case where two or more parallel jobs have hashed to * the same CPU and one of the later ones finishes first. */ -if (padata->seq_nr != pd->processed) { +if (padata->seq_nr != pd->processed) +return NULL ; + +return padata; +} + +static struct padata_priv *padata_dequeue_next (struct parallel_data *pd) +{ +struct padata_priv *padata; +struct padata_list *reorder; +int cpu = pd->cpu; + + reorder =per_cpu_ptr (pd->reorder_list, cpu); +spin_lock (&reorder->lock); + + padata =__padata_find_next (pd, reorder); +if (!padata) {spin_unlock (&reorder->lock);return NULL ; } -if (remove_object) { -list_del_init (&padata->list); - ++pd->processed; - pd->cpu =cpumask_next_wrap (cpu, pd->cpumask.pcpu,-1 , false); - } +list_del_init (&padata->list); + ++pd->processed; + pd->cpu =cpumask_next_wrap (cpu, pd->cpumask.pcpu,-1 , false);spin_unlock (&reorder->lock);return padata;
@@ -297,7 +306,7 @@ static bool padata_reorder(struct parallel_data *pd)return false;while (1 ) { - padata =padata_find_next (pd, true); + padata =padata_dequeue_next (pd);/* * If the next object that needs serialization is parallel
@@ -330,8 +339,16 @@ static bool padata_reorder(struct parallel_data *pd)smp_mb (); reorder =per_cpu_ptr (pd->reorder_list, pd->cpu); -if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) +if (!list_empty (&reorder->list)) { +spin_lock (&reorder->lock); +if (!__padata_find_next (pd, reorder)) { +spin_unlock (&reorder->lock); +return false; + } +spin_unlock (&reorder->lock); +return queue_work (pinst->serial_wq, &pd->reorder_work); + }return false; }
Signature · 1 line
2.37.3
PATCH 5/5 padata: avoid potential UAFs to the padata_shell from padata_reorder()
In reply to Nicolai StangeEven though the parallel_data "pd" instance passed to padata_reorder() is guaranteed to exist as per the reference held by its callers, the same is not true for the associated padata_shell, pd->ps. More specifically, once the last padata_priv request has been completed, either at entry from padata_reorder() or concurrently to it, the padata API users are well within their right to free the padata_shell instance.
Note that this is a purely theoretical issue, it has not been actually observed. Yet it's worth fixing for the sake of robustness.
Exploit the fact that as long as there are any not yet completed padata_priv's around on any of the percpu reorder queues, pd->ps is guaranteed to exist. Make padata_reorder() to load from pd->ps only when it's known that there is at least one in-flight padata_priv object to reorder. Note that this involves moving pd->ps accesses to under the reorder->lock as appropriate, so that the found padata_priv object won't get dequeued and completed concurrently from a different context.
padata queues")
--- kernel/padata.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
kernel/padata.c +15 -3
@@ -286,7 +286,6 @@ static struct padata_priv *padata_dequeue_next(struct parallel_data *pd)static bool padata_reorder (struct parallel_data *pd) { -struct padata_instance *pinst = pd->ps->pinst;int cb_cpu;struct padata_priv *padata;struct padata_serial_queue *squeue;
@@ -323,7 +322,11 @@ static bool padata_reorder(struct parallel_data *pd)list_add_tail (&padata->list, &squeue->serial.list);spin_unlock (&squeue->serial.lock); -queue_work_on (cb_cpu, pinst->serial_wq, &squeue->work); +/* +* Note: as long as there are requests in-flight, +* pd->ps is guaranteed to exist. +*/ +queue_work_on (cb_cpu, pd->ps->pinst->serial_wq, &squeue->work); }spin_unlock_bh (&pd->lock);
@@ -340,14 +343,23 @@ static bool padata_reorder(struct parallel_data *pd) reorder =per_cpu_ptr (pd->reorder_list, pd->cpu);if (!list_empty (&reorder->list)) { +bool reenqueued; +spin_lock (&reorder->lock);if (!__padata_find_next (pd, reorder)) {spin_unlock (&reorder->lock);return false; } + +/* +* Note: as long as there are requests in-flight, +* pd->ps is guaranteed to exist. +*/ + reenqueued =queue_work (pd->ps->pinst->serial_wq, + &pd->reorder_work);spin_unlock (&reorder->lock); -return queue_work (pinst->serial_wq, &pd->reorder_work); +return reenqueued; }return false;
Signature · 1 line
2.37.3
Daniel Jordan
maintainerRe: [PATCH 5/5] padata: avoid potential UAFs to the padata_shell from padata_reorder()
Quoted 6 lines · Nicolai Stange
Even though the
parallel_data"pd" instance passed topadata_reorder()is guaranteed to exist as per the reference held by its callers, the same is not true for the associatedpadata_shell,pd->ps. More specifically, once the lastpadata_privrequest has been completed, either at entry frompadata_reorder()or concurrently to it, the padata API users are well within their right to free thepadata_shellinstance.
The synchronize_rcu change seems to make padata_reorder safe from freed ps's with the exception of a straggler reorder_work. For that, I think something like this hybrid of your code and mine is enough to plug the hole. It's on top of 1-2 and my hunk from 3. It has to take an extra ref on pd, but only in the rare case where the reorder work is used. Thoughts?
kernel/padata.c +7 -4
@@ -277,7 +277,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,static void padata_reorder (struct parallel_data *pd) { -struct padata_instance *pinst = pd->ps->pinst; +struct padata_instance *pinst;int cb_cpu;struct padata_priv *padata;struct padata_serial_queue *squeue;
@@ -314,7 +314,7 @@ static void padata_reorder(struct parallel_data *pd)list_add_tail (&padata->list, &squeue->serial.list);spin_unlock (&squeue->serial.lock); -queue_work_on (cb_cpu, pinst->serial_wq, &squeue->work); +queue_work_on (cb_cpu, pd->ps->pinst->serial_wq, &squeue->work); }spin_unlock_bh (&pd->lock);
@@ -330,8 +330,10 @@ static void padata_reorder(struct parallel_data *pd)smp_mb (); reorder =per_cpu_ptr (pd->reorder_list, pd->cpu); -if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) { +if (queue_work (pd->ps->pinst->serial_wq, &pd->reorder_work)) +padata_get_pd (pd); + } }static void invoke_padata_reorder (struct work_struct *work)
@@ -342,6 +344,7 @@ static void invoke_padata_reorder(struct work_struct *work) pd = container_of(work, struct parallel_data, reorder_work);padata_reorder (pd);local_bh_enable (); +padata_put_pd (pd); }static void padata_serial_worker (struct work_struct *serial_work)
Re: [PATCH 5/5] padata: avoid potential UAFs to the padata_shell from padata_reorder()
Quoted 38 lines · Daniel Jordan
Quoted 6 lines · Nicolai Stange
Even though the
parallel_data"pd" instance passed topadata_reorder()is guaranteed to exist as per the reference held by its callers, the same is not true for the associatedpadata_shell,pd->ps. More specifically, once the lastpadata_privrequest has been completed, either at entry frompadata_reorder()or concurrently to it, the padata API users are well within their right to free thepadata_shellinstance.The
synchronize_rcuchange seems to makepadata_reordersafe from freed ps's with the exception of a stragglerreorder_work. For that, I think something like this hybrid of your code and mine is enough to plug the hole. It's on top of 1-2 and my hunk from 3. It has to take an extra ref on pd, but only in the rare case where the reorder work is used. Thoughts?kernel/padata.c +2
-2@@ -277,7 +277,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,static void padata_reorder (struct parallel_data *pd) { -struct padata_instance *pinst = pd->ps->pinst; +struct padata_instance *pinst;int cb_cpu;struct padata_priv *padata;struct padata_serial_queue *squeue;@@ -314,7 +314,7 @@ static void padata_reorder(struct parallel_data *pd)list_add_tail (&padata->list, &squeue->serial.list);spin_unlock (&squeue->serial.lock); -queue_work_on (cb_cpu, pinst->serial_wq, &squeue->work); +queue_work_on (cb_cpu, pd->ps->pinst->serial_wq, &squeue->work); }spin_unlock_bh (&pd->lock);
kernel/padata.c static void padata_reorder(struct parallel_data *pd) · line 335
Quoted 8 lines
kernel/padata.c +3
-2@@ -330,8 +330,10 @@ static void padata_reorder(struct parallel_data *pd)smp_mb (); reorder =per_cpu_ptr (pd->reorder_list, pd->cpu); -if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) { +if (queue_work (pd->ps->pinst->serial_wq, &pd->reorder_work)) +padata_get_pd (pd);
As the reorder_work can start running immediately after having been submitted, wouldn't it be more correct to do something like
padata_get_pd(pd); if (!queue_work(pd->ps->pinst->serial_wq, &pd->reorder_work)) padata_put_pd(pd);
?
Otherwise the patch looks good to me.
Thanks!
Nicolai
Quoted 13 lines
kernel/padata.c +2
-0+ } }static void invoke_padata_reorder (struct work_struct *work)@@ -342,6 +344,7 @@ static void invoke_padata_reorder(struct work_struct *work) pd = container_of(work, struct parallel_data, reorder_work);padata_reorder (pd);local_bh_enable (); +padata_put_pd (pd); }static void padata_serial_worker (struct work_struct *serial_work)
Signature · 3 lines
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
Daniel Jordan
maintainerRe: [PATCH 5/5] padata: avoid potential UAFs to the padata_shell from padata_reorder()
Quoted 52 lines · Nicolai Stange
Quoted 38 lines · Daniel Jordan
Quoted 6 lines · Nicolai Stange
Even though the
parallel_data"pd" instance passed topadata_reorder()is guaranteed to exist as per the reference held by its callers, the same is not true for the associatedpadata_shell,pd->ps. More specifically, once the lastpadata_privrequest has been completed, either at entry frompadata_reorder()or concurrently to it, the padata API users are well within their right to free thepadata_shellinstance.The
synchronize_rcuchange seems to makepadata_reordersafe from freed ps's with the exception of a stragglerreorder_work. For that, I think something like this hybrid of your code and mine is enough to plug the hole. It's on top of 1-2 and my hunk from 3. It has to take an extra ref on pd, but only in the rare case where the reorder work is used. Thoughts?kernel/padata.c +2
-2@@ -277,7 +277,7 @@ static struct padata_priv *padata_find_next(struct parallel_data *pd,static void padata_reorder (struct parallel_data *pd) { -struct padata_instance *pinst = pd->ps->pinst; +struct padata_instance *pinst;int cb_cpu;struct padata_priv *padata;struct padata_serial_queue *squeue;@@ -314,7 +314,7 @@ static void padata_reorder(struct parallel_data *pd)list_add_tail (&padata->list, &squeue->serial.list);spin_unlock (&squeue->serial.lock); -queue_work_on (cb_cpu, pinst->serial_wq, &squeue->work); +queue_work_on (cb_cpu, pd->ps->pinst->serial_wq, &squeue->work); }spin_unlock_bh (&pd->lock);kernel/padata.c static void padata_reorder(struct parallel_data *pd) · line 335
Quoted 8 lines
kernel/padata.c +3
-2@@ -330,8 +330,10 @@ static void padata_reorder(struct parallel_data *pd)smp_mb (); reorder =per_cpu_ptr (pd->reorder_list, pd->cpu); -if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) -queue_work (pinst->serial_wq, &pd->reorder_work); +if (!list_empty (&reorder->list) &&padata_find_next (pd, false)) { +if (queue_work (pd->ps->pinst->serial_wq, &pd->reorder_work)) +padata_get_pd (pd);As the
reorder_workcan start running immediately after having been submitted, wouldn't it be more correct to do something like
Quoted 5 lines
padata_get_pd(pd); if (!queue_work(pd->ps->pinst->serial_wq, &pd->reorder_work)) padata_put_pd(pd);?
Yes, that's better, and all the above can go in your next version too.
Daniel Jordan
maintainerHi Nicolai,
Quoted 10 lines · Nicolai Stange
Hi all,
this series is supposed to fix some lifetime issues all related to the fact that once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well with its right to tear down the associated
padata_shellorparallel_datainstance respectively.Only the first one, addressed by patch [2/5], has actually been observed, namely on a (downstream) RT kernel under a very specific workload involving LTP's
pcrypt_aead01. On non-RT, I've been unable to reproduce.
I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an x86 vm. Were there any other things running on the system besides pcrypt_aead01? More details about your environment and your kernel config would be helpful.
The first two patches seem ok but I want to think about the second more next week. I'll look over the rest of the series then too.
Hi Daniel,
Quoted 6 lines · Daniel Jordan
Quoted 5 lines · Nicolai Stange
this series is supposed to fix some lifetime issues all related to the fact that once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well with its right to tear down the associated
padata_shellorparallel_datainstance respectively.
Quoted 8 lines
Quoted 3 lines
Only the first one, addressed by patch [2/5], has actually been observed, namely on a (downstream) RT kernel under a very specific workload involving LTP's
pcrypt_aead01. On non-RT, I've been unable to reproduce.I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an x86 vm. Were there any other things running on the system besides
pcrypt_aead01? More details about your environment and your kernel config would be helpful.
Right, the issue is indeed hard to reproduce, unfortunately. It has originally been reported internally by our QA Maintenance team, which -- for unknown reason -- suddenly started to hit the issue once every while in their testing environment. I did manage to reproduce it once or twice myself, but it took me several days running pcrypt_aead01 in a loop each time. AFAIR, I allocated a single cpu to the VM only and increased the priority of pcrypt_aead01 a bit, with the intent to make preemption of the ->serial() worker by DELALG more likely. But really, I cannot tell if that did in fact contribute to the likelihood of triggering the race or whether I've just been lucky.
Also, as mentioned in the cover letter, the RT kernel this has been observed on is a downstream one, based on 5.3.18 (source tree at [1], config at [2]), but with quite some additional patches on top. A backport of this patch series here had been subject to testing in the same environment the issue originally showed up in on a fairly regular basis and no new crashes have been observed since.
Let me know if I could provide you with any more details.
Thanks,
Nicolai
[1] https://github.com/SUSE/kernel/tree/SLE15-SP3-RT
[2] https://github.com/SUSE/kernel-source/blob/SLE15-SP3-RT/config/x86_64/rt
Signature · 3 lines
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)