batman-adv: Fix general protection fault in batadv_tt_global_del_orig()
PATCH 1/1 batman-adv: Fix general protection fault in batadv_tt_global_del_orig()
On shutdown a race condition where we access a just freed global TT hash might occure:
batadv_mesh_free()->batadv_originator_free() schedules the batadv_orig_node_free_rcu().
Before batadv_orig_node_free_rcu() is executed (which happens on the rcu_barrier() call in batadv_exit() the latest), batadv_mesh_free()->batadv_tt_free()->batadv_tt_global_table_free()-> batadv_hash_destroy(hash)->kfree(hash)
is called, freeing the global tt hash.
When batadv_orig_node_free_rcu()->batadv_tt_global_del_orig() now gets executed it tries to access this just freed global tt hash, causing a kernel panic.
This patch tries to fix this by waiting for any just scheduled batadv_orig_node_free_rcu() to finish via an extra rcu_barrier() call before freeing the global TT hash.
---
Ref: #169
main.c | 5 +++++ 1 file changed, 5 insertions(+)
main.c +5 -0
@@ -164,6 +164,11 @@ void batadv_mesh_free(struct net_device *soft_iface)batadv_gw_node_purge (bat_priv);batadv_originator_free (bat_priv); + +/* Wait for any batadv_orig_node_free_rcu() to finish, +* they access the soon to be freed global TT hash */ +rcu_barrier (); +batadv_nc_free (bat_priv);batadv_tt_free (bat_priv);
Signature · 1 line
1.7.10.4
It'd probably be nicer to refactor the way we are freeing the TT things instead of this extra rcu_barrier() which isn't that self explanatory, even with a comment.
However this simple extra rcu_barrier() call should be an easy way to fix this issue for now, it's probably better to refactor afterwards.
Cheers, Linus
Hrm, forget it, this patch isn't sufficient...
batadv_nc_free() is scheduling batadv_orig_node_free_rcu()s, too. And actually, batadv_tt_free() itself does so, too, before it frees its global TT hash... which can backfire after the global TT hash freeing...
Ok, a more invasive patch to properly fix this issue is needed.
Quoted 53 lines
Gesendet: Sonntag, 17. März 2013 um 03:30 Uhr
Von: "Linus Lüssing" <[email protected]>
An: [email protected]
Cc: "Linus Lüssing" <[email protected]>
Betreff: [PATCH] batman-adv: Fix general protection fault inbatadv_tt_global_del_orig()On shutdown a race condition where we access a just freed global TT hash might occure:
batadv_mesh_free()->batadv_originator_free() schedules the
batadv_orig_node_free_rcu().Before
batadv_orig_node_free_rcu()is executed (which happens on thercu_barrier()call inbatadv_exit()the latest), batadv_mesh_free()->batadv_tt_free()->batadv_tt_global_table_free()-> batadv_hash_destroy(hash)->kfree(hash)
is called, freeing the global tt hash.When batadv_orig_node_free_rcu()->batadv_tt_global_del_orig() now gets executed it tries to access this just freed global tt hash, causing a kernel panic.
This patch tries to fix this by waiting for any just scheduled
batadv_orig_node_free_rcu()to finish via an extrarcu_barrier()call before freeing the global TT hash.Signed-off-by Linus Lüssing <[email protected]>---
Ref: #169main.c | 5 +++++ 1 file changed, 5 insertions(+)main.c +5
-1@@ -164,6 +164,11 @@ void batadv_mesh_free(struct net_device *soft_iface)batadv_gw_node_purge (bat_priv);batadv_originator_free (bat_priv); + +/* Wait for any batadv_orig_node_free_rcu() to finish, +* they access the soon to be freed global TT hash */ +rcu_barrier (); +batadv_nc_free (bat_priv);batadv_tt_free (bat_priv); --1.7.10.4
PATCH 1/2 batman-adv: Fix rcu_barrier() miss due to double call_rcu() in TT code
rcu_barrier() only waits for the currently scheduled rcu functions to finish - it won't wait for any function scheduled via another call_rcu() within an rcu scheduled function.
Unfortunately our batadv_tt_orig_list_entry_free_ref() does just that, via a batadv_orig_node_free_ref() call, leading to our rcu_barrier() call potentially missing such a batadv_orig_node_free_ref().
This patch fixes this issue by calling the batadv_orig_node_free_rcu() directly from the rcu callback, removing the unnecessary, additional call_rcu() layer here.
originator.c +1 -1
@@ -117,7 +117,7 @@ out:return neigh_node; } -static void batadv_orig_node_free_rcu (struct rcu_head *rcu) +void batadv_orig_node_free_rcu (struct rcu_head *rcu) {struct hlist_node *node_tmp;struct batadv_neigh_node *neigh_node, *tmp_neigh_node;
originator.h +1 -0
@@ -25,6 +25,7 @@int batadv_originator_init (struct batadv_priv *bat_priv);void batadv_originator_free (struct batadv_priv *bat_priv);void batadv_purge_orig_ref (struct batadv_priv *bat_priv); +void batadv_orig_node_free_rcu (struct rcu_head *rcu);void batadv_orig_node_free_ref (struct batadv_orig_node *orig_node);struct batadv_orig_node *batadv_get_orig_node (struct batadv_priv *bat_priv,const uint8_t *addr);
translation-table.c +4 -1
@@ -144,7 +144,10 @@ static void batadv_tt_orig_list_entry_free_rcu(struct rcu_head *rcu)struct batadv_tt_orig_list_entry *orig_entry; orig_entry = container_of(rcu, struct batadv_tt_orig_list_entry, rcu); -batadv_orig_node_free_ref (orig_entry->orig_node); + +if (atomic_dec_and_test (&orig_entry->orig_node->refcount)) +batadv_orig_node_free_rcu (&orig_entry->orig_node->rcu); +kfree (orig_entry); }
Signature · 1 line
1.7.10.4
PATCH 2/2 batman-adv: Fix general protection fault in batadv_tt_global_del_orig()
On shutdown a race condition where we access a just freed global TT hash might occure. batadv_orig_node_free_rcu() callbacks might have been scheduled (especially during the shutdown procedure) and unfortunately batadv_tt_global_table_free() does not wait for them to finish first before freeing the global TT hash.
This potentially results in a general protection fault in batadv_tt_global_del_orig(), called via a batadv_orig_node_free_rcu() callback, which tries to access the just freed global TT hash.
This patch tries to fix this by waiting for any just scheduled batadv_orig_node_free_rcu() to finish via an extra rcu_barrier() call before freeing the global TT hash. And by moving the TT freeing call to the end of the batman cleanup routines.
main.c +3 -2
@@ -166,12 +166,13 @@ void batadv_mesh_free(struct net_device *soft_iface)batadv_originator_free (bat_priv);batadv_nc_free (bat_priv); -batadv_tt_free (bat_priv); -batadv_bla_free (bat_priv);batadv_dat_free (bat_priv); +/* Don't call any batadv_orig_node_free_ref() after me */ +batadv_tt_free (bat_priv); +free_percpu (bat_priv->bat_counters);atomic_set (&bat_priv->mesh_state,BATADV_MESH_INACTIVE );
translation-table.c +4 -0
@@ -1315,6 +1315,10 @@ static void batadv_tt_global_table_free(struct batadv_priv *bat_priv)spin_unlock_bh (list_lock); } +/* Wait for any batadv_orig_node_free_rcu() to finish, +* they access the to be freed global TT hash */ +rcu_barrier (); +batadv_hash_destroy (hash); bat_priv->tt.global_hash =NULL ;
Signature · 1 line
1.7.10.4
Quoted 10 lines · Linus Lüssing
On shutdown a race condition where we access a just freed global TT hash might occure.
batadv_orig_node_free_rcu()callbacks might have been scheduled (especially during the shutdown procedure) and unfortunatelybatadv_tt_global_table_free()does not wait for them to finish first before freeing the global TT hash.This potentially results in a general protection fault in
batadv_tt_global_del_orig(), called via abatadv_orig_node_free_rcu()callback, which tries to access the just freed global TT hash.
Quoted 6 lines
This patch tries to fix this by waiting for any just scheduled
batadv_orig_node_free_rcu()to finish via an extrarcu_barrier()call before freeing the global TT hash. And by moving the TT freeing call to the end of the batman cleanup routines.Signed-off-by Linus Lüssing <[email protected]>
@Marek: when you will merge this commit, can you please reword "tries to fix" in "fixes" ? :)
Actually this patch is fixing the problem :)
However, as I discussed with Linus on IRC, this is only a temporary fix, which aims to remove the problem, but still we will need a redesign of the TT clean up routine in order to cleanly get rid of this race condition.
Cheers,
Signature · 4 lines
Antonio Quartulli ..each of us alone is worth nothing.. Ernesto "Che" Guevara
Quoted 11 lines · Antonio Quartulli
Quoted 10 lines · Linus Lüssing
On shutdown a race condition where we access a just freed global TT hash might occure.
batadv_orig_node_free_rcu()callbacks might have been scheduled (especially during the shutdown procedure) and unfortunatelybatadv_tt_global_table_free()does not wait for them to finish first before freeing the global TT hash.This potentially results in a general protection fault in
batadv_tt_global_del_orig(), called via abatadv_orig_node_free_rcu()callback, which tries to access the just freed global TT hash.
Quoted 8 lines
Quoted 6 lines
This patch tries to fix this by waiting for any just scheduled
batadv_orig_node_free_rcu()to finish via an extrarcu_barrier()call before freeing the global TT hash. And by moving the TT freeing call to the end of the batman cleanup routines.Signed-off-by Linus Lüssing <[email protected]>
NACK.
This patch is solving one problem but creating a new one:
by using rcu_barrier we avoid the crash but we will leak memory, because batadv_orig_node_free_rcu()->batadv_tt_global_del_orig() will access an empty global table and so will not be able to free the global entries.
Patch
("batman-adv: avoid race conditions on TT global table by counting references") is fixing the problem by redesigning the TT clean up routine.
Cheers,
Signature · 4 lines
Antonio Quartulli ..each of us alone is worth nothing.. Ernesto "Che" Guevara
Re: [B.A.T.M.A.N.] [PATCH 1/2] batman-adv: Fix rcu_barrier() miss due to double call_rcu() in TT code
In reply to Linus LüssingQuoted 44 lines · Linus Lüssing
rcu_barrier()only waits for the currently scheduled rcu functions to finish - it won't wait for any function scheduled via anothercall_rcu()within an rcu scheduled function.Unfortunately our
batadv_tt_orig_list_entry_free_ref()does just that, via abatadv_orig_node_free_ref()call, leading to ourrcu_barrier()call potentially missing such abatadv_orig_node_free_ref().This patch fixes this issue by calling the
batadv_orig_node_free_rcu()directly from the rcu callback, removing the unnecessary, additionalcall_rcu()layer here.Signed-off-by Linus Lüssing <[email protected]>originator.c +1
-1@@ -117,7 +117,7 @@ out:return neigh_node; } -static void batadv_orig_node_free_rcu (struct rcu_head *rcu) +void batadv_orig_node_free_rcu (struct rcu_head *rcu) {struct hlist_node *node_tmp;struct batadv_neigh_node *neigh_node, *tmp_neigh_node;originator.h +1
-0@@ -25,6 +25,7 @@int batadv_originator_init (struct batadv_priv *bat_priv);void batadv_originator_free (struct batadv_priv *bat_priv);void batadv_purge_orig_ref (struct batadv_priv *bat_priv); +void batadv_orig_node_free_rcu (struct rcu_head *rcu);void batadv_orig_node_free_ref (struct batadv_orig_node *orig_node);struct batadv_orig_node *batadv_get_orig_node (struct batadv_priv *bat_priv,const uint8_t *addr);translation-table.c +0
-0
translation-table.c static void batadv_tt_orig_list_entry_free_rcu(struct rcu_head *rcu) · line 152
Quoted 10 lines
translation-table.c +4
-1@@ -144,7 +144,10 @@ static void batadv_tt_orig_list_entry_free_rcu(struct rcu_head *rcu)struct batadv_tt_orig_list_entry *orig_entry; orig_entry = container_of(rcu, struct batadv_tt_orig_list_entry, rcu); -batadv_orig_node_free_ref (orig_entry->orig_node); + +if (atomic_dec_and_test (&orig_entry->orig_node->refcount)) +batadv_orig_node_free_rcu (&orig_entry->orig_node->rcu); +kfree (orig_entry); }
Hi Linus,
I was just re-reading this patch: the code can be beautified later (I think it would be worth defining a new function, e.g. batadv_orig_node_free(orig_node), which can then be invoked both in batadv_orig_node_free_rcu() and here - in this way we avoid to export batadv_orig_node_free_rcu() which should remain private to the originator.c module and we avoid this forced fake rcu invocation), but I think that putting a comment here to explain why we don't invoke call_rcu is definitely needed. In the future somebody else may ask why we don't use it and will try to re-add it.
The rest looks good.
Thanks!
Signature · 4 lines
Antonio Quartulli ..each of us alone is worth nothing.. Ernesto "Che" Guevara
PATCH v2 1/2 batman-adv: Fix rcu_barrier() miss due to double call_rcu() in TT code
rcu_barrier() only waits for the currently scheduled rcu functions to finish - it won't wait for any function scheduled via another call_rcu() within an rcu scheduled function.
Unfortunately our batadv_tt_orig_list_entry_free_ref() does just that, via a batadv_orig_node_free_ref() call, leading to our rcu_barrier() call potentially missing such a batadv_orig_node_free_ref().
This patch fixes this issue by calling the batadv_orig_node_free_rcu() directly from the rcu callback, removing the unnecessary, additional call_rcu() layer here.
---
* v2: Added a code comment as discussed on IRC:
To avoid forgetting about it, to avoid accidentally changing things back in the future.
originator.c | 2 +- originator.h | 1 + translation-table.c | 8 +++++++- 3 files changed, 9 insertions(+), 2 deletions(-)
originator.c +1 -1
@@ -117,7 +117,7 @@ out:return neigh_node; } -static void batadv_orig_node_free_rcu (struct rcu_head *rcu) +void batadv_orig_node_free_rcu (struct rcu_head *rcu) {struct hlist_node *node_tmp;struct batadv_neigh_node *neigh_node, *tmp_neigh_node;
originator.h +1 -0
@@ -25,6 +25,7 @@int batadv_originator_init (struct batadv_priv *bat_priv);void batadv_originator_free (struct batadv_priv *bat_priv);void batadv_purge_orig_ref (struct batadv_priv *bat_priv); +void batadv_orig_node_free_rcu (struct rcu_head *rcu);void batadv_orig_node_free_ref (struct batadv_orig_node *orig_node);struct batadv_orig_node *batadv_get_orig_node (struct batadv_priv *bat_priv,const uint8_t *addr);
translation-table.c +7 -1
@@ -144,7 +144,13 @@ static void batadv_tt_orig_list_entry_free_rcu(struct rcu_head *rcu)struct batadv_tt_orig_list_entry *orig_entry; orig_entry = container_of(rcu, struct batadv_tt_orig_list_entry, rcu); -batadv_orig_node_free_ref (orig_entry->orig_node); + +/* We are in an rcu callback here, therefore we cannot use +* batadv_orig_node_free_ref() and its call_rcu(): +* An rcu_barrier() wouldn't wait for that to finish */ +if (atomic_dec_and_test (&orig_entry->orig_node->refcount)) +batadv_orig_node_free_rcu (&orig_entry->orig_node->rcu); +kfree (orig_entry); }
Signature · 1 line
1.7.10.4
Re: [B.A.T.M.A.N.] [PATCHv2 1/2] batman-adv: Fix rcu_barrier() miss due to double call_rcu() in TT code
Quoted 8 lines · Linus Lüssing
rcu_barrier()only waits for the currently scheduled rcu functions to finish - it won't wait for any function scheduled via anothercall_rcu()within an rcu scheduled function.Unfortunately our
batadv_tt_orig_list_entry_free_ref()does just that, via abatadv_orig_node_free_ref()call, leading to ourrcu_barrier()call potentially missing such abatadv_orig_node_free_ref().
Quoted 5 lines
This patch fixes this issue by calling the
batadv_orig_node_free_rcu()directly from the rcu callback, removing the unnecessary, additionalcall_rcu()layer here.Signed-off-by Linus Lüssing <[email protected]>
Thanks a lot
Signature · 4 lines
Antonio Quartulli ..each of us alone is worth nothing.. Ernesto "Che" Guevara
Marek Lindner
AppliedRe: [B.A.T.M.A.N.] [PATCHv2 1/2] batman-adv: Fix rcu_barrier() miss due to double call_rcu() in TT code
Quoted 7 lines · On Wednesday, April 03, 2013 16:11:45 Antonio Quartulli wrote:
Quoted 5 lines
This patch fixes this issue by calling the
batadv_orig_node_free_rcu()directly from the rcu callback, removing the unnecessary, additionalcall_rcu()layer here.Signed-off-by Linus Lüssing <[email protected]>
Applied in revision c8bda21.
Cheers,
Marek
PS: I did some style adjustments to the patch for the sake of clarity.