srcu: Fix static initialization
Subsystems (2)
PATCH 1/1 srcu: Fix static initialization
init_srcu_struct_nodes() is called with is_static==true only internally and when this happens, the srcu->sda is not initialized in init_srcu_struct_fields() and we crash on dereferencing @sdp.
This fixes the crash by moving "if (is_static)" out of the loop which only does useful work for is_static=false case anyway.
Found by syzkaller.
--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/rcu/srcutree.c +3 -2
@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ; +/* * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree.
@@ -161,8 +164,6 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static)timer_setup (&sdp->delay_work, srcu_delay_timer,0 ); sdp->ssp = ssp; sdp->grpmask =1 << (cpu - sdp->mynode->grplo); -if (is_static) -continue ;/* Dynamically allocated, better be no srcu_read_locks()! */ for (i =0 ; i <ARRAY_SIZE (sdp->srcu_lock_count); i++) {
Signature · 1 line
2.17.1
Quoted 19 lines · Alexey Kardashevskiy
init_srcu_struct_nodes()is called with is_static==true only internally and when this happens, thesrcu->sdais not initialized ininit_srcu_struct_fields()and we crash on dereferencing @sdp.This fixes the crash by moving "if (
is_static)" out of the loop which only does useful work for is_static=false case anyway.Found by syzkaller.
Signed-off-by Alexey Kardashevskiy <[email protected]>--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/rcu/srcutree.c +0
-0
kernel/rcu/srcutree.c static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) · line 139
Quoted 5 lines
kernel/rcu/srcutree.c +2
-0@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ;
Actually, this is needed here too:
if (!ssp->sda)
return;
asssp->sda = alloc_percpu(struct srcu_data)
can fail if the process is killed too soon - it is quite easy to get this situation with syzkaller (syscalls fuzzer)
Makes sense?
Quoted 14 lines
kernel/rcu/srcutree.c +1
-2+/* * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. @@ -161,8 +164,6 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static)timer_setup (&sdp->delay_work, srcu_delay_timer,0 ); sdp->ssp = ssp; sdp->grpmask =1 << (cpu - sdp->mynode->grplo); -if (is_static) -continue ;/* Dynamically allocated, better be no srcu_read_locks()! */ for (i =0 ; i <ARRAY_SIZE (sdp->srcu_lock_count); i++) {
Signature · 1 line
Alexey
Paul E. McKenney
maintainerQuoted 33 lines · Alexey Kardashevskiy
Quoted 19 lines · Alexey Kardashevskiy
init_srcu_struct_nodes()is called with is_static==true only internally and when this happens, thesrcu->sdais not initialized ininit_srcu_struct_fields()and we crash on dereferencing @sdp.This fixes the crash by moving "if (
is_static)" out of the loop which only does useful work for is_static=false case anyway.Found by syzkaller.
Signed-off-by Alexey Kardashevskiy <[email protected]>--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) · line 139
Quoted 5 lines
kernel/rcu/srcutree.c +2
-0@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ;Actually, this is needed here too:
if (!ssp->sda)
return;
Quoted 7 lines
as
ssp->sda=alloc_percpu(struct srcu_data)can fail if the process is killed too soon - it is quite easy to get this situation with syzkaller (syscalls fuzzer)
Makes sense?
Just to make sure that I understand, these failures occur when the task running init_srcu_struct_nodes() is killed, correct?
Or has someone managed to invoke (say) synchronize_srcu() on a dynamically allocated srcu_struct before invoking init_srcu_struct() on that srcu_struct? This would be an SRCU usage bug. If you dynamically allocate your srcu_struct, you are absolutely required to invoke init_srcu_struct() on it before doing anything else with it.
Or am I missing something here?
(The rcutorture test suite does test both static and dynamic allocation of the srcu_struct, so I am expecting something a bit subtle here.)
Thanx, Paul
Quoted 17 lines
Quoted 14 lines
kernel/rcu/srcutree.c +1
-2+/* * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. @@ -161,8 +164,6 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static)timer_setup (&sdp->delay_work, srcu_delay_timer,0 ); sdp->ssp = ssp; sdp->grpmask =1 << (cpu - sdp->mynode->grplo); -if (is_static) -continue ;/* Dynamically allocated, better be no srcu_read_locks()! */ for (i =0 ; i <ARRAY_SIZE (sdp->srcu_lock_count); i++) {Signature · 1 line
Alexey
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Alexey Kardashevskiy
Quoted 19 lines · Alexey Kardashevskiy
init_srcu_struct_nodes()is called with is_static==true only internally and when this happens, thesrcu->sdais not initialized ininit_srcu_struct_fields()and we crash on dereferencing @sdp.This fixes the crash by moving "if (
is_static)" out of the loop which only does useful work for is_static=false case anyway.Found by syzkaller.
Signed-off-by Alexey Kardashevskiy <[email protected]>--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) · line 139
Quoted 5 lines
kernel/rcu/srcutree.c +2
-0@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ;Actually, this is needed here too:
if (!ssp->sda)
return;as
ssp->sda=alloc_percpu(struct srcu_data)
Quoted 7 lines
Quoted 4 lines
can fail if the process is killed too soon - it is quite easy to get this situation with syzkaller (syscalls fuzzer)
Makes sense?
Just to make sure that I understand, these failures occur when the task running
init_srcu_struct_nodes()is killed, correct?
There are multiple user tasks (8) which open /dev/kvm, do 1 ioctl - KVM_CREATE_VM - and terminate, running on 8 vcpus, 8 VMs, crashes every 20min or so, less tasks or vcpus - and the problem does not appear.
Quoted 4 lines
Or has someone managed to invoke (say)
synchronize_srcu()on a dynamically allocatedsrcu_structbefore invokinginit_srcu_struct()on thatsrcu_struct?
Nah, none of that :)
init_srcu_struct_nodes() assumes ssp->sda!=NULL but alloc_percpu() fails here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) {
pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size);
return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Quoted 28 lines
This would be an SRCU usage bug. If you dynamically allocate your
srcu_struct, you are absolutely required to invokeinit_srcu_struct()on it before doing anything else with it.Or am I missing something here?
(The rcutorture test suite does test both static and dynamic allocation of the
srcu_struct, so I am expecting something a bit subtle here.)Thanx, Paul
Quoted 17 lines
Quoted 14 lines
kernel/rcu/srcutree.c +1
-2+/* * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. @@ -161,8 +164,6 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static)timer_setup (&sdp->delay_work, srcu_delay_timer,0 ); sdp->ssp = ssp; sdp->grpmask =1 << (cpu - sdp->mynode->grplo); -if (is_static) -continue ;/* Dynamically allocated, better be no srcu_read_locks()! */ for (i =0 ; i <ARRAY_SIZE (sdp->srcu_lock_count); i++) {Signature · 1 line
Alexey
Signature · 1 line
Alexey
Paul E. McKenney
maintainerQuoted 65 lines · Alexey Kardashevskiy
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Alexey Kardashevskiy
Quoted 19 lines · Alexey Kardashevskiy
init_srcu_struct_nodes()is called with is_static==true only internally and when this happens, thesrcu->sdais not initialized ininit_srcu_struct_fields()and we crash on dereferencing @sdp.This fixes the crash by moving "if (
is_static)" out of the loop which only does useful work for is_static=false case anyway.Found by syzkaller.
Signed-off-by Alexey Kardashevskiy <[email protected]>--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) · line 139
Quoted 5 lines
kernel/rcu/srcutree.c +2
-0@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ;Actually, this is needed here too:
if (!ssp->sda)
return;as
ssp->sda=alloc_percpu(struct srcu_data)Quoted 7 lines
Quoted 4 lines
can fail if the process is killed too soon - it is quite easy to get this situation with syzkaller (syscalls fuzzer)
Makes sense?
Just to make sure that I understand, these failures occur when the task running
init_srcu_struct_nodes()is killed, correct?There are multiple user tasks (8) which open
/dev/kvm, do 1 ioctl -KVM_CREATE_VM- and terminate, running on 8 vcpus, 8 VMs, crashes every 20min or so, less tasks or vcpus - and the problem does not appear.Quoted 4 lines
Or has someone managed to invoke (say)
synchronize_srcu()on a dynamically allocatedsrcu_structbefore invokinginit_srcu_struct()on thatsrcu_struct?Nah, none of that :)
init_srcu_struct_nodes()assumes ssp->sda!=NULL butalloc_percpu()fails here:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===
Quoted 7 lines
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) { pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size); return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Ah, got it! Does the following patch help?
There will likely also need to be changes to cleanup_srcu_struct(), but first let's see if I understand the problem. ;-)
Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +3 -1
@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda) +return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */ -return ssp->sda ?0 : -ENOMEM ; +return 0 ; }#ifdef CONFIG_DEBUG_LOCK_ALLOC
Quoted 88 lines · Paul E. McKenney
Quoted 65 lines · Alexey Kardashevskiy
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Alexey Kardashevskiy
Quoted 19 lines · Alexey Kardashevskiy
init_srcu_struct_nodes()is called with is_static==true only internally and when this happens, thesrcu->sdais not initialized ininit_srcu_struct_fields()and we crash on dereferencing @sdp.This fixes the crash by moving "if (
is_static)" out of the loop which only does useful work for is_static=false case anyway.Found by syzkaller.
Signed-off-by Alexey Kardashevskiy <[email protected]>--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) · line 139
Quoted 5 lines
kernel/rcu/srcutree.c +2
-0@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ;Actually, this is needed here too:
if (!ssp->sda)
return;as
ssp->sda=alloc_percpu(struct srcu_data)Quoted 7 lines
Quoted 4 lines
can fail if the process is killed too soon - it is quite easy to get this situation with syzkaller (syscalls fuzzer)
Makes sense?
Just to make sure that I understand, these failures occur when the task running
init_srcu_struct_nodes()is killed, correct?There are multiple user tasks (8) which open
/dev/kvm, do 1 ioctl -KVM_CREATE_VM- and terminate, running on 8 vcpus, 8 VMs, crashes every 20min or so, less tasks or vcpus - and the problem does not appear.Quoted 4 lines
Or has someone managed to invoke (say)
synchronize_srcu()on a dynamically allocatedsrcu_structbefore invokinginit_srcu_struct()on thatsrcu_struct?Nah, none of that :)
init_srcu_struct_nodes()assumes ssp->sda!=NULL butalloc_percpu()fails here:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===Quoted 7 lines
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) { pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size); return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Ah, got it! Does the following patch help?
There will likely also need to be changes to
cleanup_srcu_struct(), but first let's see if I understand the problem. ;-)Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +0
-0
kernel/rcu/srcutree.c static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) · line 185
Quoted 9 lines
kernel/rcu/srcutree.c +2
-0@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda) +return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */
The line above confuses me a bit. What you propose returns without smp_store_release() called which should not matter I suppose.
Otherwise it should work, although I cannot verify right now as my box went down and since it is across Pacific - it may take time to power cycle it :) Thanks,
Quoted 6 lines
kernel/rcu/srcutree.c +1
-1-return ssp->sda ?0 : -ENOMEM ; +return 0 ; }#ifdef CONFIG_DEBUG_LOCK_ALLOC
Signature · 1 line
Alexey
Paul E. McKenney
maintainerQuoted 95 lines · Alexey Kardashevskiy
Quoted 92 lines · Paul E. McKenney
Quoted 65 lines · Alexey Kardashevskiy
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Alexey Kardashevskiy
Quoted 19 lines · Alexey Kardashevskiy
init_srcu_struct_nodes()is called with is_static==true only internally and when this happens, thesrcu->sdais not initialized ininit_srcu_struct_fields()and we crash on dereferencing @sdp.This fixes the crash by moving "if (
is_static)" out of the loop which only does useful work for is_static=false case anyway.Found by syzkaller.
Signed-off-by Alexey Kardashevskiy <[email protected]>--- kernel/rcu/srcutree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/rcu/srcutree.c +0
-0kernel/rcu/srcutree.c static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) · line 139
Quoted 5 lines
kernel/rcu/srcutree.c +2
-0@@ -135,6 +135,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *ssp, bool is_static) levelspread[level -1 ]; } +if (is_static) +return ;Actually, this is needed here too:
if (!ssp->sda)
return;as
ssp->sda=alloc_percpu(struct srcu_data)Quoted 7 lines
Quoted 4 lines
can fail if the process is killed too soon - it is quite easy to get this situation with syzkaller (syscalls fuzzer)
Makes sense?
Just to make sure that I understand, these failures occur when the task running
init_srcu_struct_nodes()is killed, correct?There are multiple user tasks (8) which open
/dev/kvm, do 1 ioctl -KVM_CREATE_VM- and terminate, running on 8 vcpus, 8 VMs, crashes every 20min or so, less tasks or vcpus - and the problem does not appear.Quoted 4 lines
Or has someone managed to invoke (say)
synchronize_srcu()on a dynamically allocatedsrcu_structbefore invokinginit_srcu_struct()on thatsrcu_struct?Nah, none of that :)
init_srcu_struct_nodes()assumes ssp->sda!=NULL butalloc_percpu()fails here:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===Quoted 7 lines
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) { pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size); return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Ah, got it! Does the following patch help?
There will likely also need to be changes to
cleanup_srcu_struct(), but first let's see if I understand the problem. ;-)Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +1
-0@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda)
Quoted 8 lines
kernel/rcu/srcutree.c static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) · line 185
Quoted 5 lines
kernel/rcu/srcutree.c +1
-0+return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */ The line above confuses me a bit. What you propose returns without
smp_store_release()called which should not matter I suppose.
The idea is that if init_srcu_struct() returns -ENOMEM, the structure has not been initialized and had better not be used. If the calling code cannot handle that outcome, then the calling code needs to do something to insulate init_srcu_struct() from signals. One thing that it could do would be to invoke init_srcu_struct() from a workqueue handler and wait for this handler to complete.
Please keep in mind that there is nothing init_srcu_struct() can do about this: The srcu_struct is useless unless alloc_percpu() succeeds.
And yes, I do need to update the header comments to make this clear.
Quoted 3 lines
Otherwise it should work, although I cannot verify right now as my box went down and since it is across Pacific - it may take time to power cycle it :) Thanks,
I know that feeling! And here is hoping that the box is out of reach of the local hot spots. ;-)
Thanx, Paul
Quoted 9 lines
Quoted 6 lines
kernel/rcu/srcutree.c +1
-1-return ssp->sda ?0 : -ENOMEM ; +return 0 ; }#ifdef CONFIG_DEBUG_LOCK_ALLOC Signature · 1 line
Alexey
Paul E. McKenney
maintainerQuoted 3 lines · Paul E. McKenney
Quoted 2 lines · Alexey Kardashevskiy
Quoted 1 line · Paul E. McKenney
On Wed, Sep 09, 2020 at 10:31:03PM +1000, Alexey Kardashevskiy wrote:
[ . . . ]
Quoted 50 lines
Quoted 31 lines
Quoted 31 lines
Quoted 5 lines
init_srcu_struct_nodes()assumes ssp->sda!=NULL butalloc_percpu()fails here:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===Quoted 7 lines
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) { pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size); return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Ah, got it! Does the following patch help?
There will likely also need to be changes to
cleanup_srcu_struct(), but first let's see if I understand the problem. ;-)Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +1
-0@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda)Quoted 8 lines
kernel/rcu/srcutree.c static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) · line 185
Quoted 5 lines
kernel/rcu/srcutree.c +1
-0+return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */ The line above confuses me a bit. What you propose returns without
smp_store_release()called which should not matter I suppose.The idea is that if
init_srcu_struct()returns-ENOMEM, the structure has not been initialized and had better not be used. If the calling code cannot handle that outcome, then the calling code needs to do something to insulateinit_srcu_struct()from signals. One thing that it could do would be to invokeinit_srcu_struct()from a workqueue handler and wait for this handler to complete.Please keep in mind that there is nothing
init_srcu_struct()can do about this: Thesrcu_structis useless unlessalloc_percpu()succeeds.
Quoted 8 lines
And yes, I do need to update the header comments to make this clear.
Quoted 3 lines
Otherwise it should work, although I cannot verify right now as my box went down and since it is across Pacific - it may take time to power cycle it :) Thanks,
I know that feeling! And here is hoping that the box is out of reach of the local hot spots. ;-)
Just following up... Did that patch help?
Thanx, Paul
Alexey Kardashevskiy
Tested-byQuoted 59 lines · Paul E. McKenney
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines · Alexey Kardashevskiy
Quoted 1 line · Paul E. McKenney
On Wed, Sep 09, 2020 at 10:31:03PM +1000, Alexey Kardashevskiy wrote:
[ . . . ]
Quoted 52 lines
Quoted 31 lines
Quoted 31 lines
Quoted 5 lines
init_srcu_struct_nodes()assumes ssp->sda!=NULL butalloc_percpu()fails here:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===Quoted 7 lines
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) { pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size); return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Ah, got it! Does the following patch help?
There will likely also need to be changes to
cleanup_srcu_struct(), but first let's see if I understand the problem. ;-)Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +1
-0@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda)Quoted 8 lines
kernel/rcu/srcutree.c static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) · line 185
Quoted 5 lines
kernel/rcu/srcutree.c +1
-0+return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */ The line above confuses me a bit. What you propose returns without
smp_store_release()called which should not matter I suppose.The idea is that if
init_srcu_struct()returns-ENOMEM, the structure has not been initialized and had better not be used. If the calling code cannot handle that outcome, then the calling code needs to do something to insulateinit_srcu_struct()from signals. One thing that it could do would be to invokeinit_srcu_struct()from a workqueue handler and wait for this handler to complete.Please keep in mind that there is nothing
init_srcu_struct()can do about this: Thesrcu_structis useless unlessalloc_percpu()succeeds.And yes, I do need to update the header comments to make this clear.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Otherwise it should work, although I cannot verify right now as my box went down and since it is across Pacific - it may take time to power cycle it :) Thanks,
I know that feeling! And here is hoping that the box is out of reach of the local hot spots. ;-)
Just following up... Did that patch help?
Yes it did.
Quoted 3 lines
Thanx, Paul
Signature · 1 line
Alexey
Paul E. McKenney
maintainerQuoted 66 lines · Alexey Kardashevskiy
Quoted 63 lines · Paul E. McKenney
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines · Alexey Kardashevskiy
Quoted 1 line · Paul E. McKenney
On Wed, Sep 09, 2020 at 10:31:03PM +1000, Alexey Kardashevskiy wrote:
[ . . . ]
Quoted 56 lines
Quoted 31 lines
Quoted 31 lines
Quoted 5 lines
init_srcu_struct_nodes()assumes ssp->sda!=NULL butalloc_percpu()fails here:https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/percpu.c#n1734
===Quoted 7 lines
} else if (mutex_lock_killable(&pcpu_alloc_mutex)) { pcpu_memcg_post_alloc_hook(objcg, NULL, 0, size); return NULL;===
I am still up to reading that osr-rcuusage.pdf to provide better analysis :) Thanks,
Ah, got it! Does the following patch help?
There will likely also need to be changes to
cleanup_srcu_struct(), but first let's see if I understand the problem. ;-)Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +1
-0@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda)Quoted 8 lines
kernel/rcu/srcutree.c static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) · line 185
Quoted 5 lines
kernel/rcu/srcutree.c +1
-0+return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */ The line above confuses me a bit. What you propose returns without
smp_store_release()called which should not matter I suppose.The idea is that if
init_srcu_struct()returns-ENOMEM, the structure has not been initialized and had better not be used. If the calling code cannot handle that outcome, then the calling code needs to do something to insulateinit_srcu_struct()from signals. One thing that it could do would be to invokeinit_srcu_struct()from a workqueue handler and wait for this handler to complete.Please keep in mind that there is nothing
init_srcu_struct()can do about this: Thesrcu_structis useless unlessalloc_percpu()succeeds.And yes, I do need to update the header comments to make this clear.
Quoted 3 lines
Otherwise it should work, although I cannot verify right now as my box went down and since it is across Pacific - it may take time to power cycle it :) Thanks,
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I know that feeling! And here is hoping that the box is out of reach of the local hot spots. ;-)
Just following up... Did that patch help?
Yes it did.
Tested-by Alexey Kardashevskiy <[email protected]>
Thank you, and I will apply on the next rebase.
Thanx, Paul