Skip to contentSkip to conversation

PREEMPT_RCU breaks anon_vma locking ?

DISCUSSION Oleg Nesterov 4 participants 10 messages Feb 23, 2007 – Mar 2, 2007 Memory management

lore
Share & tools
lore
#0 lore raw

If my understanding correct, vmscan can find a page which lives in a already anon_vma_unlink'ed vma. This is ok, the page is pinned, and page->mapping is not cleared until free_hot_cold_page().

So page_lock_anon_vma() works correctly due to SLAB_DESTROY_BY_RCU even if anon_vma_unlink() has already freed anon_vma. In that case we should see list_empty(&anon_vma->head), we are safe.

However, we are doing spin_unlock(anon_vma->lock) after page_lock_anon_vma(), and this looks unsafe to me because page_lock_anon_vma() does rcu_read_unlock() on return.

This worked before because spin_lock() implied rcu_read_lock(), so rcu was blocked if page_lock_anon_vma() returns !NULL. With CONFIG_PREEMPT_RCU this is not true (yes?), so it is possible that the slab returns the memory to the system and it is re-used when we write to anon_vma->lock.

IOW, don't we need something like this

	static struct anon_vma *page_lock_anon_vma(struct page *page)
	{
		struct anon_vma *anon_vma;
		unsigned long anon_mapping;

		rcu_read_lock();
		anon_mapping = (unsigned long) page->mapping;
		if (!(anon_mapping & PAGE_MAPPING_ANON))
			goto out;
		if (!page_mapped(page))
			goto out;

		anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
		spin_lock(&anon_vma->lock);
		return anon_vma;

	out:
		rcu_read_unlock();
		return NULL;
	}

	static inline void page_lock_anon_vma(struct anon_vma *anon_vma)
	{
		spin_unlock(&anon_vma->lock);
		rcu_read_unlock();
	}

?

Oleg.

#1 lore raw
Quoted 11 lines · Oleg Nesterov

If my understanding correct, vmscan can find a page which lives in a already anon_vma_unlink'ed vma. This is ok, the page is pinned, and page->mapping is not cleared until free_hot_cold_page().

So page_lock_anon_vma() works correctly due to SLAB_DESTROY_BY_RCU even if anon_vma_unlink() has already freed anon_vma. In that case we should see list_empty(&anon_vma->head), we are safe.

However, we are doing spin_unlock(anon_vma->lock) after page_lock_anon_vma(), and this looks unsafe to me because page_lock_anon_vma() does rcu_read_unlock() on return.

This would indeed be bad when using CONFIG_PREEMPT_RCU! Good catch!!!

Quoted 33 lines

This worked before because spin_lock() implied rcu_read_lock(), so rcu was blocked if page_lock_anon_vma() returns !NULL. With CONFIG_PREEMPT_RCU this is not true (yes?), so it is possible that the slab returns the memory to the system and it is re-used when we write to anon_vma->lock.

IOW, don't we need something like this

	static struct anon_vma *page_lock_anon_vma(struct page *page)
	{
		struct anon_vma *anon_vma;
		unsigned long anon_mapping;

		rcu_read_lock();
		anon_mapping = (unsigned long) page->mapping;
		if (!(anon_mapping & PAGE_MAPPING_ANON))
			goto out;
		if (!page_mapped(page))
			goto out;

		anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
		spin_lock(&anon_vma->lock);
		return anon_vma;

	out:
		rcu_read_unlock();
		return NULL;
	}

	static inline void page_lock_anon_vma(struct anon_vma *anon_vma)
	{
		spin_unlock(&anon_vma->lock);
		rcu_read_unlock();
	}
Quoted 1 line

?

This look like a valid fix to me, at least as long as the lock is never dropped in the meantime (e.g., to do I/O). If the lock -is- dropped in the meantime, then presumably whatever is done to keep the page from vanishing should allow an rcu_read_unlock() to be placed after each spin_unlock(&...->lock) and an rcu_read_lock() to be placed before each spin_lock(&...->lock).

Thanx, Paul

#2 lore raw
Quoted 7 lines · Paul E. McKenney

This look like a valid fix to me, at least as long as the lock is never dropped in the meantime (e.g., to do I/O). If the lock -is- dropped in the meantime, then presumably whatever is done to keep the page from vanishing should allow an rcu_read_unlock() to be placed after each spin_unlock(&...->lock) and an rcu_read_lock() to be placed before each spin_lock(&...->lock).

Thankfully no complications of that kind, page_lock_anon_vma is static to mm/rmap.c, and only used to hold the spin lock while examining page tables of the vmas in the list, never a need to drop that lock at all. (Until the day when someone reports such a long list that we start to worry about the latency.)

Hugh

#3 lore raw
Quoted 9 lines · Hugh Dickins
Quoted 7 lines · Paul E. McKenney

This look like a valid fix to me, at least as long as the lock is never dropped in the meantime (e.g., to do I/O). If the lock -is- dropped in the meantime, then presumably whatever is done to keep the page from vanishing should allow an rcu_read_unlock() to be placed after each spin_unlock(&...->lock) and an rcu_read_lock() to be placed before each spin_lock(&...->lock).

Quoted 5 lines

Thankfully no complications of that kind, page_lock_anon_vma is static to mm/rmap.c, and only used to hold the spin lock while examining page tables of the vmas in the list, never a need to drop that lock at all. (Until the day when someone reports such a long list that we start to worry about the latency.)

Whew!!! For now, anyway! ;-)

Thanx, Paul

#4 lore raw
In reply to Oleg Nesterov
Quoted 3 lines · Oleg Nesterov

If my understanding correct, vmscan can find a page which lives in a already anon_vma_unlink'ed vma. This is ok, the page is pinned, and page->mapping is not cleared until free_hot_cold_page().

That's about right. The page_mapped checks, at several levels, make it very hard to hit this case in practice; but it is possible for the page to be unmapped and the anon_vma unlinked and kmem_cache_freed while vmscan is racing through page_lock_anon_vma.

Quoted 4 lines

So page_lock_anon_vma() works correctly due to SLAB_DESTROY_BY_RCU even if anon_vma_unlink() has already freed anon_vma. In that case we should see list_empty(&anon_vma->head), we are safe.

(It doesn't affect your argument, but we won't necessarily see list_empty
there: the anon_vma slot may already have got reused for a different bundle of vmas completely; but its lock remains a lock and its list remains a list of vmas, and the worst that happens is that page_referenced_anon or try_to_unmap_anon wanders through an irrelevant bundle of vmas, looking for a page that cannot be found there.)

Quoted 9 lines

However, we are doing spin_unlock(anon_vma->lock) after page_lock_anon_vma(), and this looks unsafe to me because page_lock_anon_vma() does rcu_read_unlock() on return.

This worked before because spin_lock() implied rcu_read_lock(), so rcu was blocked if page_lock_anon_vma() returns !NULL. With CONFIG_PREEMPT_RCU this is not true (yes?), so it is possible that the slab returns the memory to the system and it is re-used when we write to anon_vma->lock.

I had been meaning to take a fresh look at page_lock_anon_vma, to see whether recent RCU developments in -mm affected it. Thanks for saving me the trouble.

You and Paul know infinitely more about CONFIG_PREEMPT_RCU than I do, so if you believe that the change below is enough, that's great, it's much simpler than I'd feared might be needed. CONFIG_PREEMPT_RCU rather scares me, but perhaps it's less worrying than I'd imagined.

Have you checked through the SLAB_DESTROY_BY_RCU end in slab.c? Is what that's doing still valid?

Quoted 17 lines

IOW, don't we need something like this

	static struct anon_vma *page_lock_anon_vma(struct page *page)
	{
		struct anon_vma *anon_vma;
		unsigned long anon_mapping;

		rcu_read_lock();
		anon_mapping = (unsigned long) page->mapping;
		if (!(anon_mapping & PAGE_MAPPING_ANON))
			goto out;
		if (!page_mapped(page))
			goto out;

		anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
		spin_lock(&anon_vma->lock);
Quoted 8 lines
		return anon_vma;

	out:
		rcu_read_unlock();
		return NULL;
	}

	static inline void page_lock_anon_vma(struct anon_vma *anon_vma)

It might be wiser to call that one page_unlock_anon_vma ;)

(I'm slightly disgruntled that page_lock_anon_vma takes a struct page *, but page_unlock_anon_vma no struct page *. But it would be silly to do it differently, or mess with the naming: besides, it's a static function and the prototype guards against error anyway.)

Quoted 5 lines
	{
		spin_unlock(&anon_vma->lock);
		rcu_read_unlock();
	}

?

Looks fine to me: please go ahead a make a patch for -mm, Oleg: thanks.

I've CC'ed Christoph for several reasons.

One, I think he was trying to persuade me a year ago to change page_lock_anon_vma as you're now proposing; but I resisted, preferring to keep the RCU stuff self-contained within page_lock_anon_vma: let's credit him for prescience, and admit you've proved me wrong.

Two, in his SLUB thread it sounds like he's toying with mixing up kmem caches of similar sizes: that seems a really bad idea to me (for reasons Andi and others have made) and I expect it'll get dropped; but in case not, let's note that the anon_vma cache is one which would fare very badly from getting mixed in with others (lock would no longer remain a lock, list would no longer remain a list, across that race).

Three, he has a recurring interest in this unusual page_lock_anon_vma, mainly for page migration reasons, so let's keep him informed.

Hugh

#5 lore raw
Quoted 4 lines · Hugh Dickins

On Sat, 24 Feb 2007, Oleg Nesterov wrote:

Have you checked through the SLAB_DESTROY_BY_RCU end in slab.c? Is what that's doing still valid?

The only thing I see needed due to PREEMPT_RCU is the following comment change.

For a terrified few minutes, I thought that the code assumed that struct rcu_head was the same size as struct list_head, but it turns out to only assume that struct slab is at least as large as struct slab_rcu.

Thanx, Paul

---

mm/slab.c +1 -1
diff -urpNa -X dontdiff linux-2.6.20/mm/slab.c linux-2.6.20-slabrcufix/mm/slab.c
@@ -238,7 +238,7 @@ struct slab {
  * other kind of object (which our subsystem's lock might corrupt).
  *
  * rcu_read_lock before reading the address, then rcu_read_unlock after
- * taking the spinlock within the structure expected at that address.
+ * releasing the spinlock within the structure expected at that address.
  *
  * We assume struct slab_rcu can overlay struct slab when destroying.
  */
Acked-by
#6 lore raw
Quoted 5 lines · Paul E. McKenney
Quoted 3 lines · Hugh Dickins

Have you checked through the SLAB_DESTROY_BY_RCU end in slab.c? Is what that's doing still valid?

Quoted 8 lines

The only thing I see needed due to PREEMPT_RCU is the following comment change.

For a terrified few minutes, I thought that the code assumed that struct rcu_head was the same size as struct list_head, but it turns out to only assume that struct slab is at least as large as struct slab_rcu.

Thanx, Paul

Thanks for enduring the terror, checking it out, and arriving at such a reassuring conclusion. Andrew, please add this to your -mm collection after (or folded into) Paul's rcu-preemptible-rcu.patch.

PREEMPT_RCU has stricter needs: updated comment on SLAB_DESTROY_BY_RCU.

---

mm/slab.c +1 -1
diff -urpNa -X dontdiff linux-2.6.20/mm/slab.c linux-2.6.20-slabrcufix/mm/slab.c
@@ -238,7 +238,7 @@ struct slab {
  * other kind of object (which our subsystem's lock might corrupt).
  *
  * rcu_read_lock before reading the address, then rcu_read_unlock after
- * taking the spinlock within the structure expected at that address.
+ * releasing the spinlock within the structure expected at that address.
  *
  * We assume struct slab_rcu can overlay struct slab when destroying.
  */
#7 lore raw
In reply to Hugh Dickins
Quoted 6 lines · Hugh Dickins

Two, in his SLUB thread it sounds like he's toying with mixing up kmem caches of similar sizes: that seems a really bad idea to me (for reasons Andi and others have made) and I expect it'll get dropped; but in case not, let's note that the anon_vma cache is one which would fare very badly from getting mixed in with others (lock would no longer remain a lock, list would no longer remain a list, across that race).

Dont worry DESTROY_BY_RCU slabs wont be mixed with those not marked DESTROY_BY_RCU.

#8 lore raw
In reply to Hugh Dickins
Quoted 7 lines · Hugh Dickins
Quoted 3 lines · Oleg Nesterov

So page_lock_anon_vma() works correctly due to SLAB_DESTROY_BY_RCU even if anon_vma_unlink() has already freed anon_vma. In that case we should see list_empty(&anon_vma->head), we are safe.

Quoted 6 lines

(It doesn't affect your argument, but we won't necessarily see list_empty
there: the anon_vma slot may already have got reused for a different bundle of vmas completely; but its lock remains a lock and its list remains a list of vmas, and the worst that happens is that page_referenced_anon or try_to_unmap_anon wanders through an irrelevant bundle of vmas, looking for a page that cannot be found there.)

Yes, but in that case we are safe, right? We hold the lock, anon_vma can't be freed. But thanks for clarification! Somehow I missed that not only unlock() is unsafe (in theory). If anon_vma's memory was re-used for something else, we can't assume that we will see list_empty(&anon_vma->head).

Quoted 3 lines
Quoted 1 line

static inline void page_lock_anon_vma(struct anon_vma *anon_vma)

It might be wiser to call that one page_unlock_anon_vma ;)

Congratulations, you passed the test! Paul didn't :)

Quoted 4 lines

(I'm slightly disgruntled that page_lock_anon_vma takes a struct page *, but page_unlock_anon_vma no struct page *. But it would be silly to do it differently, or mess with the naming: besides, it's a static function and the prototype guards against error anyway.)

OK. I thought about "unlock_anon_vma", but symmetry is good indeed.

Oleg.

#9 lore raw
Quoted 20 lines · Oleg Nesterov
Quoted 7 lines · Hugh Dickins
Quoted 3 lines · Oleg Nesterov

So page_lock_anon_vma() works correctly due to SLAB_DESTROY_BY_RCU even if anon_vma_unlink() has already freed anon_vma. In that case we should see list_empty(&anon_vma->head), we are safe.

Quoted 6 lines

(It doesn't affect your argument, but we won't necessarily see list_empty
there: the anon_vma slot may already have got reused for a different bundle of vmas completely; but its lock remains a lock and its list remains a list of vmas, and the worst that happens is that page_referenced_anon or try_to_unmap_anon wanders through an irrelevant bundle of vmas, looking for a page that cannot be found there.)

Yes, but in that case we are safe, right? We hold the lock, anon_vma can't be freed. But thanks for clarification! Somehow I missed that not only unlock() is unsafe (in theory). If anon_vma's memory was re-used for something else, we can't assume that we will see list_empty(&anon_vma->head).

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

static inline void page_lock_anon_vma(struct anon_vma *anon_vma)

It might be wiser to call that one page_unlock_anon_vma ;)

Congratulations, you passed the test! Paul didn't :)

What is in a name? ;-)

Thanx, Paul

Quoted 9 lines
Quoted 4 lines

(I'm slightly disgruntled that page_lock_anon_vma takes a struct page *, but page_unlock_anon_vma no struct page *. But it would be silly to do it differently, or mess with the naming: besides, it's a static function and the prototype guards against error anyway.)

OK. I thought about "unlock_anon_vma", but symmetry is good indeed.

Oleg.