adapt page_lock_anon_vma() to PREEMPT_RCU
Subsystems (2)
PATCH 1/1 adapt page_lock_anon_vma() to PREEMPT_RCU
page_lock_anon_vma() uses spin_lock() to block RCU. This doesn't work with PREEMPT_RCU, we have to do rcu_read_lock() explicitely. Otherwise, it is theoretically possible that slab returns anon_vma's memory to the system before we do spin_unlock(&anon_vma->lock).
mm/rmap.c renamed from mm/rmap.c~ +13 -4
@@ -183,7 +183,7 @@ void __init anon_vma_init(void)*/ static struct anon_vma *page_lock_anon_vma (struct page *page) { -struct anon_vma *anon_vma =NULL ; +struct anon_vma *anon_vma;unsigned long anon_mapping;rcu_read_lock ();
@@ -195,9 +195,16 @@ static struct anon_vma *page_lock_anon_v anon_vma = (struct anon_vma *) (anon_mapping -PAGE_MAPPING_ANON );spin_lock (&anon_vma->lock); +return anon_vma; out:rcu_read_unlock (); -return anon_vma; +return NULL ; +} + +static void page_unlock_anon_vma (struct anon_vma *anon_vma) +{ +spin_unlock (&anon_vma->lock); +rcu_read_unlock (); }/*
@@ -333,7 +340,8 @@ static int page_referenced_anon(struct pif (!mapcount)break ; } -spin_unlock (&anon_vma->lock); + +page_unlock_anon_vma (anon_vma);return referenced; }
@@ -809,7 +817,8 @@ static int try_to_unmap_anon(struct page !page_mapped(page))break ; } -spin_unlock (&anon_vma->lock); + +page_unlock_anon_vma (anon_vma);return ret; }
Andrew Morton
maintainermm/rmap.c static struct anon_vma *page_lock_anon_v · line 208
Quoted 13 lines · On Sun, 25 Feb 2007 23:06:21 +0300 Oleg Nesterov <[email protected]> wrote:
page_lock_anon_vma()usesspin_lock()to block RCU. This doesn't work withPREEMPT_RCU, we have to dorcu_read_lock()explicitely. Otherwise, it is theoretically possible that slab returns anon_vma's memory to the system before we dospin_unlock(&anon_vma->lock)....
mm/rmap.c renamed from mm/rmap.c~ +4
-0+static void page_unlock_anon_vma (struct anon_vma *anon_vma) +{ +spin_unlock (&anon_vma->lock); +rcu_read_unlock (); }
It's a bit sad doing a double preempt_disable() for non-PREEMPT_RCU builds.
Perhaps we would benefit from a new rcu_read_lock_preempt_rcu() which is a no-op if !PREEMPT_RCU.
Quoted 9 lines · Andrew Morton
Quoted 9 lines · On Sun, 25 Feb 2007 23:06:21 +0300 Oleg Nesterov <[email protected]> wrote:
page_lock_anon_vma()usesspin_lock()to block RCU. This doesn't work withPREEMPT_RCU, we have to dorcu_read_lock()explicitely. Otherwise, it is theoretically possible that slab returns anon_vma's memory to the system before we dospin_unlock(&anon_vma->lock)....
Quoted 7 lines
mm/rmap.c static struct anon_vma *page_lock_anon_v · line 208
Quoted 5 lines
mm/rmap.c renamed from mm/rmap.c~ +4
-0+static void page_unlock_anon_vma (struct anon_vma *anon_vma) +{ +spin_unlock (&anon_vma->lock); +rcu_read_unlock (); }It's a bit sad doing a double
preempt_disable()for non-PREEMPT_RCU builds.
Actually, we don't in this case. This patch in essence moves "preempt_enable" from "lock" to "unlock" side. Zero impact for non-PREEMPT_RCU builds, except .text grows a bit.
Before this patch, page_lock_anon_vma() does preempt_enable() before return, but this can't help because ->preempt_count was incremented by spin_lock().
Quoted 2 lines
Perhaps we would benefit from a new
rcu_read_lock_preempt_rcu()which is a no-op if !PREEMPT_RCU.
I also thought about things like
rcu_read_lock_when_we_know_that_preemption_disabled() rcu_read_lock_when_we_know_that_irqs_disabled()
which are noops when !PREEMPT_RCU.
Oleg.
Quoted 16 lines · Andrew Morton
mm/rmap.c static struct anon_vma *page_lock_anon_v · line 208
Quoted 13 lines · On Sun, 25 Feb 2007 23:06:21 +0300 Oleg Nesterov <[email protected]> wrote:
page_lock_anon_vma()usesspin_lock()to block RCU. This doesn't work withPREEMPT_RCU, we have to dorcu_read_lock()explicitely. Otherwise, it is theoretically possible that slab returns anon_vma's memory to the system before we dospin_unlock(&anon_vma->lock)....
mm/rmap.c renamed from mm/rmap.c~ +4
-0+static void page_unlock_anon_vma (struct anon_vma *anon_vma) +{ +spin_unlock (&anon_vma->lock); +rcu_read_unlock (); }It's a bit sad doing a double
preempt_disable()for non-PREEMPT_RCU builds.
Quoted 2 lines
Perhaps we would benefit from a new
rcu_read_lock_preempt_rcu()which is a no-op if !PREEMPT_RCU.
We were doing double preempt_disable() before as well. The only difference is that we moved RCU preempt_enable() (it used to be inside the critical section, and now it is after the corresponding spin_unlock()).
I hope to keep RCU API proliferation down to a dull roar, but if we need more APIs, we need more APIs. This example does not demonstrate that need to me, however.
Thanx, Paul
Quoted 6 lines · Oleg Nesterov
page_lock_anon_vma()usesspin_lock()to block RCU. This doesn't work withPREEMPT_RCU, we have to dorcu_read_lock()explicitely. Otherwise, it is theoretically possible that slab returns anon_vma's memory to the system before we dospin_unlock(&anon_vma->lock).Signed-off-by Oleg Nesterov <[email protected]>
Thanks for doing this, and sorry for my delay.
Hugh
Quoted 51 lines
mm/rmap.c renamed from mm/rmap.c~ +13
-4@@ -183,7 +183,7 @@ void __init anon_vma_init(void)*/ static struct anon_vma *page_lock_anon_vma (struct page *page) { -struct anon_vma *anon_vma =NULL ; +struct anon_vma *anon_vma;unsigned long anon_mapping;rcu_read_lock ();@@ -195,9 +195,16 @@ static struct anon_vma *page_lock_anon_v anon_vma = (struct anon_vma *) (anon_mapping -PAGE_MAPPING_ANON );spin_lock (&anon_vma->lock); +return anon_vma; out:rcu_read_unlock (); -return anon_vma; +return NULL ; +} + +static void page_unlock_anon_vma (struct anon_vma *anon_vma) +{ +spin_unlock (&anon_vma->lock); +rcu_read_unlock (); }/* @@ -333,7 +340,8 @@ static int page_referenced_anon(struct pif (!mapcount)break ; } -spin_unlock (&anon_vma->lock); + +page_unlock_anon_vma (anon_vma);return referenced; }@@ -809,7 +817,8 @@ static int try_to_unmap_anon(struct page !page_mapped(page))break ; } -spin_unlock (&anon_vma->lock); + +page_unlock_anon_vma (anon_vma);return ret; }