Skip to contentSkip to conversation

mm, swap: Remove unnecessary smp_rmb() in swap_type_to_swap_info()

PATCH In review Huang Ying 5 participants 12 messages May 13, 2021 – May 14, 2021

Subsystems (2)
lore
Share & tools
lore

Huang Ying

author
#0 lore raw

PATCH 1/1 mm, swap: Remove unnecessary smp_rmb() in swap_type_to_swap_info()

Before commit c10d38cc8d3e ("mm, swap: bounds check swap_info array accesses to avoid NULL derefs"), the typical code to reference the swap_info[] is as follows,

  type = swp_type(swp_entry);
  if (type >= nr_swapfiles)
          /* handle invalid swp_entry */;
  p = swap_info[type];
  /* access fields of *p.  OOPS! p may be NULL! */

Because the ordering isn't guaranteed, it's possible that "p" is read before checking "type". And that may result in NULL pointer dereference.

So in commit c10d38cc8d3e, the code becomes,

struct swap_info_struct *swap_type_to_swap_info(int type)

  {
	  if (type >= READ_ONCE(nr_swapfiles))
		  return NULL;
	  smp_rmb();
	  return READ_ONCE(swap_info[type]);

}

/* users */
type = swp_type(swp_entry);
p = swap_type_to_swap_info(type);
if (!p)

	  /* handle invalid swp_entry */;

/* access fields of *p */

Because "p" is checked to be non-zero before dereference, smp_rmb() isn't needed anymore.

We still need to guarantee swap_info[type] is read before dereference. That can be satisfied via the data dependency ordering of READ_ONCE(swap_info[type]). The corresponding smp_wmb() is adjusted in alloc_swap_info() too.

And, we don't need to read "nr_swapfiles" too. Because if "type >= nr_swapfiles", swap_info[type] will be NULL. We just need to make sure we will not access out of the boundary of the array. With that change, nr_swapfiles will only be accessed with swap_lock held, except in swapcache_free_entries(). Where the absolute correctness of the value isn't needed, as described in the comments.

Cc Daniel Jordan <[email protected]>
Cc Dan Carpenter <[email protected]>
Cc Andrea Parri <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Omar Sandoval <[email protected]>
Cc Paul McKenney <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Will Deacon <[email protected]>
Cc Miaohe Lin <[email protected]>
---
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +9 -9
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
+	/*
+	 * The data dependency ordering from the READ_ONCE() pairs
+	 * with smp_wmb() in alloc_swap_info() to guarantee the
+	 * swap_info_struct fields are read after swap_info[type].
+	 */
 	return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);
-		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
-		 */
+		/* Paired with READ_ONCE() in swap_type_to_swap_info() */
 		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		WRITE_ONCE(swap_info[type], p);
+		nr_swapfiles++;
 	} else {
 		defer = p;
 		p = swap_info[type];
Signature · 1 line
2.30.2
#1 lore raw
Quoted 85 lines · Huang Ying

Before commit c10d38cc8d3e ("mm, swap: bounds check swap_info array accesses to avoid NULL derefs"), the typical code to reference the swap_info[] is as follows,

  type = swp_type(swp_entry);
  if (type >= nr_swapfiles)
          /* handle invalid swp_entry */;
  p = swap_info[type];
  /* access fields of *p.  OOPS! p may be NULL! */

Because the ordering isn't guaranteed, it's possible that "p" is read before checking "type". And that may result in NULL pointer dereference.

So in commit c10d38cc8d3e, the code becomes,

struct swap_info_struct *swap_type_to_swap_info(int type)

  {
	  if (type >= READ_ONCE(nr_swapfiles))
		  return NULL;
	  smp_rmb();
	  return READ_ONCE(swap_info[type]);

}

/* users */
type = swp_type(swp_entry);
p = swap_type_to_swap_info(type);
if (!p)

	  /* handle invalid swp_entry */;

/* access fields of *p */

Because "p" is checked to be non-zero before dereference, smp_rmb() isn't needed anymore.

We still need to guarantee swap_info[type] is read before dereference. That can be satisfied via the data dependency ordering of READ_ONCE(swap_info[type]). The corresponding smp_wmb() is adjusted in alloc_swap_info() too.

And, we don't need to read "nr_swapfiles" too. Because if "type >= nr_swapfiles", swap_info[type] will be NULL. We just need to make sure we will not access out of the boundary of the array. With that change, nr_swapfiles will only be accessed with swap_lock held, except in swapcache_free_entries(). Where the absolute correctness of the value isn't needed, as described in the comments.

Cc Daniel Jordan <[email protected]>
Cc Dan Carpenter <[email protected]>
Cc Andrea Parri <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Omar Sandoval <[email protected]>
Cc Paul McKenney <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Will Deacon <[email protected]>
Cc Miaohe Lin <[email protected]>
---
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +6 -2
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
+	/*
+	 * The data dependency ordering from the READ_ONCE() pairs
+	 * with smp_wmb() in alloc_swap_info() to guarantee the
+	 * swap_info_struct fields are read after swap_info[type].
+	 */
 	return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2892

Quoted 10 lines
mm/swapfile.c +1 -6
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);
-		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
-		 */
+		/* Paired with READ_ONCE() in swap_type_to_swap_info() */
 		smp_wmb();

Many thank for your patch. The patch looks fine to me. There is one question:

There is no smp_rmb() paired with above smp_wmb(). What is this smp_wmb() used for ? Could you please have a explanation ?

Thanks again!

Quoted 7 lines
mm/swapfile.c +2 -1
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		WRITE_ONCE(swap_info[type], p);
+		nr_swapfiles++;
 	} else {
 		defer = p;
 		p = swap_info[type];
#2 lore raw
Quoted 100 lines · Miaohe Lin
Quoted 85 lines · Huang Ying

Before commit c10d38cc8d3e ("mm, swap: bounds check swap_info array accesses to avoid NULL derefs"), the typical code to reference the swap_info[] is as follows,

  type = swp_type(swp_entry);
  if (type >= nr_swapfiles)
          /* handle invalid swp_entry */;
  p = swap_info[type];
  /* access fields of *p.  OOPS! p may be NULL! */

Because the ordering isn't guaranteed, it's possible that "p" is read before checking "type". And that may result in NULL pointer dereference.

So in commit c10d38cc8d3e, the code becomes,

  struct swap_info_struct *swap_type_to_swap_info(int type)
  {
        if (type >= READ_ONCE(nr_swapfiles))
                return NULL;
        smp_rmb();
        return READ_ONCE(swap_info[type]);
  }

  /* users */
  type = swp_type(swp_entry);
  p = swap_type_to_swap_info(type);
  if (!p)
        /* handle invalid swp_entry */;
  /* access fields of *p */

Because "p" is checked to be non-zero before dereference, smp_rmb() isn't needed anymore.

We still need to guarantee swap_info[type] is read before dereference. That can be satisfied via the data dependency ordering of READ_ONCE(swap_info[type]). The corresponding smp_wmb() is adjusted in alloc_swap_info() too.

And, we don't need to read "nr_swapfiles" too. Because if "type >= nr_swapfiles", swap_info[type] will be NULL. We just need to make sure we will not access out of the boundary of the array. With that change, nr_swapfiles will only be accessed with swap_lock held, except in swapcache_free_entries(). Where the absolute correctness of the value isn't needed, as described in the comments.

Cc Daniel Jordan <[email protected]>
Cc Dan Carpenter <[email protected]>
Cc Andrea Parri <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Omar Sandoval <[email protected]>
Cc Paul McKenney <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Will Deacon <[email protected]>
Cc Miaohe Lin <[email protected]>
---
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +6 -2
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-     if (type >= READ_ONCE(nr_swapfiles))
+     if (type >= MAX_SWAPFILES)
              return NULL;
 
-     smp_rmb();      /* Pairs with smp_wmb in alloc_swap_info. */
+     /*
+      * The data dependency ordering from the READ_ONCE() pairs
+      * with smp_wmb() in alloc_swap_info() to guarantee the
+      * swap_info_struct fields are read after swap_info[type].
+      */
      return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
      }

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2892

Quoted 10 lines
mm/swapfile.c +1 -6
      if (type >= nr_swapfiles) {
              p->type = type;
-             WRITE_ONCE(swap_info[type], p);
-             /*
-              * Write swap_info[type] before nr_swapfiles, in case a
-              * racing procfs swap_start() or swap_next() is reading them.
-              * (We never shrink nr_swapfiles, we never free this entry.)
-              */
+             /* Paired with READ_ONCE() in swap_type_to_swap_info() */
              smp_wmb();

Many thank for your patch. The patch looks fine to me. There is one question:

Quoted 2 lines

There is no smp_rmb() paired with above smp_wmb(). What is this smp_wmb() used for ? Could you please have a explanation ?

The comment is very clear, it matches READ_ONCE() which implies a data dependence barrier on some archs.

Thanks.

Quoted 11 lines

Thanks again!

Quoted 7 lines
mm/swapfile.c +2 -1
-             WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+             WRITE_ONCE(swap_info[type], p);
+             nr_swapfiles++;
      } else {
              defer = p;
              p = swap_info[type];
#3 lore raw
Quoted 101 lines · Muchun Song
Quoted 100 lines · Miaohe Lin
Quoted 85 lines · Huang Ying

Before commit c10d38cc8d3e ("mm, swap: bounds check swap_info array accesses to avoid NULL derefs"), the typical code to reference the swap_info[] is as follows,

  type = swp_type(swp_entry);
  if (type >= nr_swapfiles)
          /* handle invalid swp_entry */;
  p = swap_info[type];
  /* access fields of *p.  OOPS! p may be NULL! */

Because the ordering isn't guaranteed, it's possible that "p" is read before checking "type". And that may result in NULL pointer dereference.

So in commit c10d38cc8d3e, the code becomes,

  struct swap_info_struct *swap_type_to_swap_info(int type)
  {
        if (type >= READ_ONCE(nr_swapfiles))
                return NULL;
        smp_rmb();
        return READ_ONCE(swap_info[type]);
  }

  /* users */
  type = swp_type(swp_entry);
  p = swap_type_to_swap_info(type);
  if (!p)
        /* handle invalid swp_entry */;
  /* access fields of *p */

Because "p" is checked to be non-zero before dereference, smp_rmb() isn't needed anymore.

We still need to guarantee swap_info[type] is read before dereference. That can be satisfied via the data dependency ordering of READ_ONCE(swap_info[type]). The corresponding smp_wmb() is adjusted in alloc_swap_info() too.

And, we don't need to read "nr_swapfiles" too. Because if "type >= nr_swapfiles", swap_info[type] will be NULL. We just need to make sure we will not access out of the boundary of the array. With that change, nr_swapfiles will only be accessed with swap_lock held, except in swapcache_free_entries(). Where the absolute correctness of the value isn't needed, as described in the comments.

Cc Daniel Jordan <[email protected]>
Cc Dan Carpenter <[email protected]>
Cc Andrea Parri <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Omar Sandoval <[email protected]>
Cc Paul McKenney <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Will Deacon <[email protected]>
Cc Miaohe Lin <[email protected]>
---
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +6 -2
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-     if (type >= READ_ONCE(nr_swapfiles))
+     if (type >= MAX_SWAPFILES)
              return NULL;
 
-     smp_rmb();      /* Pairs with smp_wmb in alloc_swap_info. */
+     /*
+      * The data dependency ordering from the READ_ONCE() pairs
+      * with smp_wmb() in alloc_swap_info() to guarantee the
+      * swap_info_struct fields are read after swap_info[type].
+      */
      return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
      }

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2892

Quoted 10 lines
mm/swapfile.c +1 -6
      if (type >= nr_swapfiles) {
              p->type = type;
-             WRITE_ONCE(swap_info[type], p);
-             /*
-              * Write swap_info[type] before nr_swapfiles, in case a
-              * racing procfs swap_start() or swap_next() is reading them.
-              * (We never shrink nr_swapfiles, we never free this entry.)
-              */
+             /* Paired with READ_ONCE() in swap_type_to_swap_info() */
              smp_wmb();

Many thank for your patch. The patch looks fine to me. There is one question:

Quoted 7 lines
Quoted 2 lines

There is no smp_rmb() paired with above smp_wmb(). What is this smp_wmb() used for ? Could you please have a explanation ?

The comment is very clear, it matches READ_ONCE() which implies a data dependence barrier on some archs.

Thanks.

Got it. I misunderstood it... Thanks!

Quoted 14 lines
Quoted 12 lines

Thanks again!

Quoted 7 lines
mm/swapfile.c +2 -1
-             WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+             WRITE_ONCE(swap_info[type], p);
+             nr_swapfiles++;
      } else {
              defer = p;
              p = swap_info[type];

.

#4 lore raw
In reply to Muchun Song
Quoted 42 lines · Muchun Song
Quoted 41 lines · Miaohe Lin
Quoted 26 lines · Huang Ying
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +6 -2
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-     if (type >= READ_ONCE(nr_swapfiles))
+     if (type >= MAX_SWAPFILES)
              return NULL;
 
-     smp_rmb();      /* Pairs with smp_wmb in alloc_swap_info. */
+     /*
+      * The data dependency ordering from the READ_ONCE() pairs
+      * with smp_wmb() in alloc_swap_info() to guarantee the
+      * swap_info_struct fields are read after swap_info[type].
+      */
      return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
      }

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2892

Quoted 10 lines
mm/swapfile.c +1 -6
      if (type >= nr_swapfiles) {
              p->type = type;
-             WRITE_ONCE(swap_info[type], p);
-             /*
-              * Write swap_info[type] before nr_swapfiles, in case a
-              * racing procfs swap_start() or swap_next() is reading them.
-              * (We never shrink nr_swapfiles, we never free this entry.)
-              */
+             /* Paired with READ_ONCE() in swap_type_to_swap_info() */
              smp_wmb();

Many thank for your patch. The patch looks fine to me. There is one question:

Quoted 5 lines
Quoted 2 lines

There is no smp_rmb() paired with above smp_wmb(). What is this smp_wmb() used for ? Could you please have a explanation ?

The comment is very clear, it matches READ_ONCE() which implies a data dependence barrier on some archs.

This statement doesn't make sense; this isn't code that needs to be correct on 'some' archs, it needs to be unconditionally correct.

Also, you cannot pair with a single memop, there is no order in a set of one element.

And if you depend on a data dependency, you need a store order; but you just removed the store order. in which case the data dependency is also moot.

All of this is utter confusion. Possibly correct, but a complete trainwreck non-the-less.

Either you say ordering is irrelevant, because we only ever increase the number of swapfiles and therefore any load is either NULL or the correct pointer, as guaranteed by WRITE_ONCE()/READ_ONCE() avoiding load/store tearing.

Or you need the data dependency, but then you also need the store order like:

	CPU0					CPU1

	if (type >= READ_ONCE(nr_swapfiles))	WRITE_ONCE(swap_info[type], p);
		return NULL;
	/* data-dependency on type */		smp_wmb();
	return READ_ONCE(swap_info[type]);	WRITE_ONCE(nr_swapfiles, nr_swapfiles+1);

But you cannot have half of both and expect any of it to make sense.

#5 lore raw
In reply to Huang Ying
Quoted 29 lines · Huang Ying
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +6 -3
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
+	/*
+	 * The data dependency ordering from the READ_ONCE() pairs
+	 * with smp_wmb() in alloc_swap_info() to guarantee the
+	 * swap_info_struct fields are read after swap_info[type].
+	 */
 	return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2894

Quoted 10 lines
mm/swapfile.c +3 -6
-		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
-		 */
+		/* Paired with READ_ONCE() in swap_type_to_swap_info() */
 		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		WRITE_ONCE(swap_info[type], p);
+		nr_swapfiles++;

Ah, I think I see what you meant to say, it would perhaps help if you write it like so:

mm/swapfile.c +5 -9
index 149e77454e3c..94735248dcd2 100644
@@ -99,11 +99,10 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
-	return READ_ONCE(swap_info[type]);
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline unsigned char swap_count(unsigned char ent)
@@ -2869,14 +2868,11 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);
 		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
+		 * Publish the swap_info_struct.
 		 */
-		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		smp_store_release(&swap_info[type], p); /* rcu_assign_pointer() */
+		nr_swapfiles++;
 	} else {
 		defer = p;
 		p = swap_info[type];
#6 lore raw
Quoted 27 lines · Peter Zijlstra

Ah, I think I see what you meant to say, it would perhaps help if you write it like so:

mm/swapfile.c +2 -4
index 149e77454e3c..94735248dcd2 100644
@@ -99,11 +99,10 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
-	return READ_ONCE(swap_info[type]);
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline unsigned char swap_count(unsigned char ent)
@@ -2869,14 +2868,11 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2875

Quoted 10 lines
mm/swapfile.c +3 -5
 		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
+		 * Publish the swap_info_struct.
 		 */
-		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		smp_store_release(&swap_info[type], p); /* rcu_assign_pointer() */
+		nr_swapfiles++;

Yes, this does help, I didn't understand why smp_wmb stayed around in the original post.

I think the only access smp_store_release() orders is p->type. Wouldn't it be kinda inconsistent to only initialize that one field before publishing when many others would be done at the end of alloc_swap_info() after the fact? p->type doesn't seem special. For instance, get_swap_page_of_type() touches si->lock soon after it calls swap_type_to_swap_info(), so there could be a small window where there's a non-NULL si with an uninitialized lock.

It's not as if this is likely to be a problem in practice, it would just make it harder to understand why smp_store_release is there. Maybe all we need is a WRITE_ONCE, or if it's really necessary for certain fields to be set before publication then move them up and explain?

#7 lore raw
Quoted 42 lines · Daniel Jordan
Quoted 27 lines · Peter Zijlstra

Ah, I think I see what you meant to say, it would perhaps help if you write it like so:

mm/swapfile.c +2 -4
index 149e77454e3c..94735248dcd2 100644
@@ -99,11 +99,10 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
-	return READ_ONCE(swap_info[type]);
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline unsigned char swap_count(unsigned char ent)
@@ -2869,14 +2868,11 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2875

Quoted 10 lines
mm/swapfile.c +3 -5
 		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
+		 * Publish the swap_info_struct.
 		 */
-		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		smp_store_release(&swap_info[type], p); /* rcu_assign_pointer() */
+		nr_swapfiles++;

Yes, this does help, I didn't understand why smp_wmb stayed around in the original post.

Quoted 4 lines

I think the only access smp_store_release() orders is p->type. Wouldn't it be kinda inconsistent to only initialize that one field before publishing when many others would be done at the end of alloc_swap_info() after the fact?

In addition to p->type, *p is zeroed via kvzalloc().

Quoted 4 lines

p->type doesn't seem special. For
instance, get_swap_page_of_type() touches si->lock soon after it calls swap_type_to_swap_info(), so there could be a small window where there's a non-NULL si with an uninitialized lock.

We usually check the state of swap_info_struct before other operations. For example, we check si->swap_map in swap_start().

Quoted 4 lines

It's not as if this is likely to be a problem in practice, it would just make it harder to understand why smp_store_release is there. Maybe all we need is a WRITE_ONCE, or if it's really necessary for certain fields to be set before publication then move them up and explain?

I think we have initialized all fields before publication :-).

Best Regards,
Huang, Ying

#8 lore raw
Quoted 10 lines · Ying
Quoted 7 lines · Daniel Jordan

Yes, this does help, I didn't understand why smp_wmb stayed around in the original post.

I think the only access smp_store_release() orders is p->type. Wouldn't it be kinda inconsistent to only initialize that one field before publishing when many others would be done at the end of alloc_swap_info() after the fact?

In addition to p->type, *p is zeroed via kvzalloc().

So it is, good point.

Quoted 7 lines
Quoted 4 lines

p->type doesn't seem special. For
instance, get_swap_page_of_type() touches si->lock soon after it calls swap_type_to_swap_info(), so there could be a small window where there's a non-NULL si with an uninitialized lock.

We usually check the state of swap_info_struct before other operations. For example, we check si->swap_map in swap_start().

Yes, we usually do.

Quoted 6 lines
Quoted 4 lines

It's not as if this is likely to be a problem in practice, it would just make it harder to understand why smp_store_release is there. Maybe all we need is a WRITE_ONCE, or if it's really necessary for certain fields to be set before publication then move them up and explain?

I think we have initialized all fields before publication :-).

Probably all the ones that matter in practice, yes :-)

Still feeling slightly uneasy about the theoretical p->lock, but that was possible before this change too so it's out of scope.

A comment explaining the pairing and that we care mostly about the zero init would be nice.

#9 lore raw
In reply to Daniel Jordan
Quoted 50 lines · Daniel Jordan
Quoted 27 lines · Peter Zijlstra

Ah, I think I see what you meant to say, it would perhaps help if you write it like so:

mm/swapfile.c +2 -4
index 149e77454e3c..94735248dcd2 100644
@@ -99,11 +99,10 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
-	return READ_ONCE(swap_info[type]);
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline unsigned char swap_count(unsigned char ent)
@@ -2869,14 +2868,11 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2875

Quoted 10 lines
mm/swapfile.c +3 -5
 		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
+		 * Publish the swap_info_struct.
 		 */
-		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		smp_store_release(&swap_info[type], p); /* rcu_assign_pointer() */
+		nr_swapfiles++;

Yes, this does help, I didn't understand why smp_wmb stayed around in the original post.

I think the only access smp_store_release() orders is p->type. Wouldn't it be kinda inconsistent to only initialize that one field before publishing when many others would be done at the end of alloc_swap_info() after the fact? p->type doesn't seem special. For instance, get_swap_page_of_type() touches si->lock soon after it calls swap_type_to_swap_info(), so there could be a small window where there's a non-NULL si with an uninitialized lock.

Quoted 4 lines

It's not as if this is likely to be a problem in practice, it would just make it harder to understand why smp_store_release is there. Maybe all we need is a WRITE_ONCE, or if it's really necessary for certain fields to be set before publication then move them up and explain?

You also care about the zero fill from kvzalloc(). Without the smp_store_release() the zero-fill from the memset() might only be visible 'late'.

Unless that also isn't a problem?

#10 lore raw
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Daniel Jordan

Yes, this does help, I didn't understand why smp_wmb stayed around in the original post.

I think the only access smp_store_release() orders is p->type. Wouldn't it be kinda inconsistent to only initialize that one field before publishing when many others would be done at the end of alloc_swap_info() after the fact? p->type doesn't seem special. For instance, get_swap_page_of_type() touches si->lock soon after it calls swap_type_to_swap_info(), so there could be a small window where there's a non-NULL si with an uninitialized lock.

Quoted 8 lines
Quoted 4 lines

It's not as if this is likely to be a problem in practice, it would just make it harder to understand why smp_store_release is there. Maybe all we need is a WRITE_ONCE, or if it's really necessary for certain fields to be set before publication then move them up and explain?

You also care about the zero fill from kvzalloc(). Without the smp_store_release() the zero-fill from the memset() might only be visible 'late'.

Aha, yes, didn't consider that!

Quoted 1 line

Unless that also isn't a problem?

No, you're right, we need that for p->flags at least.

#11 lore raw
In reply to Peter Zijlstra
Quoted 71 lines · Peter Zijlstra
Quoted 29 lines · Huang Ying
 mm/swapfile.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
mm/swapfile.c +6 -3
index 2aad85751991..4c1fb28bbe0e 100644
@@ -100,10 +100,14 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
+	/*
+	 * The data dependency ordering from the READ_ONCE() pairs
+	 * with smp_wmb() in alloc_swap_info() to guarantee the
+	 * swap_info_struct fields are read after swap_info[type].
+	 */
 	return READ_ONCE(swap_info[type]);
 }
 
@@ -2884,14 +2888,10 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2894

Quoted 10 lines
mm/swapfile.c +3 -6
-		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.
-		 * (We never shrink nr_swapfiles, we never free this entry.)
-		 */
+		/* Paired with READ_ONCE() in swap_type_to_swap_info() */
 		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		WRITE_ONCE(swap_info[type], p);
+		nr_swapfiles++;

Ah, I think I see what you meant to say, it would perhaps help if you write it like so:

mm/swapfile.c +2 -6
index 149e77454e3c..94735248dcd2 100644
@@ -99,11 +99,10 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
 static struct swap_info_struct *swap_type_to_swap_info(int type)
 {
-	if (type >= READ_ONCE(nr_swapfiles))
+	if (type >= MAX_SWAPFILES)
 		return NULL;
 
-	smp_rmb();	/* Pairs with smp_wmb in alloc_swap_info. */
-	return READ_ONCE(swap_info[type]);
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline unsigned char swap_count(unsigned char ent)
@@ -2869,14 +2868,11 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	if (type >= nr_swapfiles) {
 		p->type = type;
-		WRITE_ONCE(swap_info[type], p);
 		/*
-		 * Write swap_info[type] before nr_swapfiles, in case a
-		 * racing procfs swap_start() or swap_next() is reading them.

mm/swapfile.c static struct swap_info_struct *alloc_swap_info(void) · line 2878

Quoted 10 lines
mm/swapfile.c +3 -3
-		 * (We never shrink nr_swapfiles, we never free this entry.)
+		 * Publish the swap_info_struct.
 		 */
-		smp_wmb();
-		WRITE_ONCE(nr_swapfiles, nr_swapfiles + 1);
+		smp_store_release(&swap_info[type], p); /* rcu_assign_pointer() */
+		nr_swapfiles++;
 	} else {
 		defer = p;
 		p = swap_info[type];

OK. It seems that this helps people to understand. I will use this in the next version.

Best Regards,
Huang, Ying