Bug fix in aio ring page migration.

PATCH Superseded Tang Chen 7 participants 13 messages Feb 27, 2014 – Mar 5, 2014

Subsystems (2)

VFS Aio

lore

Share & tools

lore

Tang Chen

author

2014-02-27 10:38 UTC #0 lore raw

cover Bug fix in aio ring page migration.

This patch-set fixes the following two problems:

1. Need to use ctx->completion_lock to protect ring pages from being mis-written while migration.

2. Need memory barrier to ensure memory copy is done before ctx->ring_pages[] is updated.

NOTE: AIO ring page migration was implemented since Linux 3.12.
So we need to merge these two patches into 3.12 stable tree.

Tang Chen (2):
aio, memory-hotplug: Fix confliction when migrating and accessing ring
pages.
aio, mem-hotplug: Add memory barrier to aio ring page migration.

 fs/aio.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

Signature · 1 line

1.8.3.1

Tang Chen

author

2014-02-27 10:38 UTC #1 lore raw

PATCH 1/2 aio, memory-hotplug: Fix confliction when migrating and accessing ring pages.

AIO ring page migration has been implemented by the following patch:

        https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/aio.c?id=36bc08cc01709b4a9bb563b35aa530241ddc63e3

In this patch, ctx->completion_lock is used to prevent other processes from accessing the ring page being migrated.

But in aio_setup_ring(), ioctx_add_table() and aio_read_events_ring(), when writing to the ring page, they didn't take ctx->completion_lock.

As a result, for example, we have the following problem:

            thread 1                      |              thread 2
                                          |
aio_migratepage()                         |
 |-> take ctx->completion_lock            |
 |-> migrate_page_copy(new, old)          |
 |   *NOW*, ctx->ring_pages[idx] == old   |
                                          |
                                          |    *NOW*, ctx->ring_pages[idx] == old
                                          |    aio_read_events_ring()
                                          |     |-> ring = kmap_atomic(ctx->ring_pages[0])
                                          |     |-> ring->head = head;          *HERE, write to the old ring page*
                                          |     |-> kunmap_atomic(ring);
                                          |
 |-> ctx->ring_pages[idx] = new           |
 |   *BUT NOW*, the content of            |
 |    ring_pages[idx] is old.             |
 |-> release ctx->completion_lock         |

As above, the new ring page will not be updated.

The solution is taking ctx->completion_lock in thread 2, which means, in aio_setup_ring(), ioctx_add_table() and aio_read_events_ring() when writing to ring pages.

Reported-by Yasuaki Ishimatsu <[email protected]>

Signed-off-by Tang Chen <[email protected]>

---
 fs/aio.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

fs/aio.c +33 -0

index 062a5f6..50c089c 100644
@@ -366,6 +366,7 @@ static int aio_setup_ring(struct kioctx *ctx)
 	int nr_pages;
 	int i;
 	struct file *file;
+	unsigned long flags;
 
 	/* Compensate for the ring buffer's head/tail overlap entry */
 	nr_events += 2;	/* 1 is required, 2 for good luck */

@@ -437,6 +438,14 @@ static int aio_setup_ring(struct kioctx *ctx)
 	ctx->user_id = ctx->mmap_base;
 	ctx->nr_events = nr_events; /* trusted copy */
 
+	/*
+	 * The aio ring pages are user space pages, so they can be migrated.
+	 * When writing to an aio ring page, we should ensure the page is not
+	 * being migrated. Aio page migration procedure is protected by
+	 * ctx->completion_lock, so we add this lock here.
+	 */
+	spin_lock_irqsave(&ctx->completion_lock, flags);
+
 	ring = kmap_atomic(ctx->ring_pages[0]);
 	ring->nr = nr_events;	/* user copy */
 	ring->id = ~0U;

@@ -448,6 +457,8 @@ static int aio_setup_ring(struct kioctx *ctx)
 	kunmap_atomic(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
+	spin_unlock_irqrestore(&ctx->completion_lock, flags);
+
 	return 0;
 }

@@ -542,6 +553,7 @@ static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm)
 	unsigned i, new_nr;
 	struct kioctx_table *table, *old;
 	struct aio_ring *ring;
+	unsigned long flags;
 
 	spin_lock(&mm->ioctx_lock);
 	rcu_read_lock();

@@ -556,9 +568,19 @@ static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm)
 					rcu_read_unlock();
 					spin_unlock(&mm->ioctx_lock);
 
+					/*
+					 * Accessing ring pages must be done
+					 * holding ctx->completion_lock to
+					 * prevent aio ring page migration
+					 * procedure from migrating ring pages.
+					 */
+					spin_lock_irqsave(&ctx->completion_lock,
+							  flags);
 					ring = kmap_atomic(ctx->ring_pages[0]);
 					ring->id = ctx->id;
 					kunmap_atomic(ring);
+					spin_unlock_irqrestore(
+						&ctx->completion_lock, flags);
 					return 0;
 				}

@@ -1021,6 +1043,7 @@ static long aio_read_events_ring(struct kioctx *ctx,
 	unsigned head, tail, pos;
 	long ret = 0;
 	int copy_ret;
+	unsigned long flags;
 
 	mutex_lock(&ctx->ring_lock);

@@ -1066,11 +1089,21 @@ static long aio_read_events_ring(struct kioctx *ctx,
 		head %= ctx->nr_events;
 	}
 
+	/*
+	 * The aio ring pages are user space pages, so they can be migrated.
+	 * When writing to an aio ring page, we should ensure the page is not
+	 * being migrated. Aio page migration procedure is protected by
+	 * ctx->completion_lock, so we add this lock here.
+	 */
+	spin_lock_irqsave(&ctx->completion_lock, flags);
+
 	ring = kmap_atomic(ctx->ring_pages[0]);
 	ring->head = head;
 	kunmap_atomic(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
+	spin_unlock_irqrestore(&ctx->completion_lock, flags);
+
 	pr_debug("%li  h%u t%u\n", ret, head, tail);
 
 	put_reqs_available(ctx, ret);

Signature · 1 line

1.8.3.1

Jeff Moyer

Reviewed-by on v1 1/2

2014-03-05 19:23 UTC #2 lore raw

Re: [PATCH 1/2] aio, memory-hotplug: Fix confliction when migrating and accessing ring pages.

Quoted 30 lines · Tang Chen

AIO ring page migration has been implemented by the following patch:

        https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/aio.c?id=36bc08cc01709b4a9bb563b35aa530241ddc63e3

In this patch, ctx->completion_lock is used to prevent other processes from accessing the ring page being migrated.

But in aio_setup_ring(), ioctx_add_table() and aio_read_events_ring(), when writing to the ring page, they didn't take ctx->completion_lock.

As a result, for example, we have the following problem:

            thread 1                      |              thread 2
                                          |
aio_migratepage()                         |
 |-> take ctx->completion_lock            |
 |-> migrate_page_copy(new, old)          |
 |   *NOW*, ctx->ring_pages[idx] == old   |
                                          |
                                          |    *NOW*, ctx->ring_pages[idx] == old
                                          |    aio_read_events_ring()
                                          |     |-> ring = kmap_atomic(ctx->ring_pages[0])
                                          |     |-> ring->head = head;          *HERE, write to the old ring page*
                                          |     |-> kunmap_atomic(ring);
                                          |
 |-> ctx->ring_pages[idx] = new           |
 |   *BUT NOW*, the content of            |
 |    ring_pages[idx] is old.             |
 |-> release ctx->completion_lock         |

Quoted 5 lines

As above, the new ring page will not be updated.
The solution is taking ctx->completion_lock in thread 2, which means, in aio_setup_ring(), ioctx_add_table() and aio_read_events_ring() when writing to ring pages.

Thanks for the good explanation and for adding comments to the code. This looks right to me. I guess the only issue I have with it is that the code paths changed don't run in interrupt context, so I think you could just use spin_lock_irq. That's not a big deal, though.

Reviewed-by Jeff Moyer <[email protected]>

Quoted 90 lines

Reported-by Yasuaki Ishimatsu <[email protected]>

Signed-off-by Tang Chen <[email protected]>

---
 fs/aio.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

fs/aio.c +31 -0

index 062a5f6..50c089c 100644
@@ -366,6 +366,7 @@ static int aio_setup_ring(struct kioctx *ctx)
 	int nr_pages;
 	int i;
 	struct file *file;
+	unsigned long flags;
 
 	/* Compensate for the ring buffer's head/tail overlap entry */
 	nr_events += 2;	/* 1 is required, 2 for good luck */

@@ -437,6 +438,14 @@ static int aio_setup_ring(struct kioctx *ctx)
 	ctx->user_id = ctx->mmap_base;
 	ctx->nr_events = nr_events; /* trusted copy */
 
+	/*
+	 * The aio ring pages are user space pages, so they can be migrated.
+	 * When writing to an aio ring page, we should ensure the page is not
+	 * being migrated. Aio page migration procedure is protected by
+	 * ctx->completion_lock, so we add this lock here.
+	 */
+	spin_lock_irqsave(&ctx->completion_lock, flags);
+
 	ring = kmap_atomic(ctx->ring_pages[0]);
 	ring->nr = nr_events;	/* user copy */
 	ring->id = ~0U;

@@ -448,6 +457,8 @@ static int aio_setup_ring(struct kioctx *ctx)
 	kunmap_atomic(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
+	spin_unlock_irqrestore(&ctx->completion_lock, flags);
+
 	return 0;
 }

@@ -542,6 +553,7 @@ static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm)
 	unsigned i, new_nr;
 	struct kioctx_table *table, *old;
 	struct aio_ring *ring;
+	unsigned long flags;
 
 	spin_lock(&mm->ioctx_lock);
 	rcu_read_lock();

@@ -556,9 +568,19 @@ static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm)
 					rcu_read_unlock();
 					spin_unlock(&mm->ioctx_lock);
 
+					/*
+					 * Accessing ring pages must be done
+					 * holding ctx->completion_lock to
+					 * prevent aio ring page migration
+					 * procedure from migrating ring pages.
+					 */
+					spin_lock_irqsave(&ctx->completion_lock,
+							  flags);
 					ring = kmap_atomic(ctx->ring_pages[0]);
 					ring->id = ctx->id;
 					kunmap_atomic(ring);
+					spin_unlock_irqrestore(
+						&ctx->completion_lock, flags);
 					return 0;
 				}

@@ -1021,6 +1043,7 @@ static long aio_read_events_ring(struct kioctx *ctx,
 	unsigned head, tail, pos;
 	long ret = 0;
 	int copy_ret;
+	unsigned long flags;
 
 	mutex_lock(&ctx->ring_lock);

@@ -1066,11 +1089,21 @@ static long aio_read_events_ring(struct kioctx *ctx,
 		head %= ctx->nr_events;
 	}
 
+	/*
+	 * The aio ring pages are user space pages, so they can be migrated.
+	 * When writing to an aio ring page, we should ensure the page is not
+	 * being migrated. Aio page migration procedure is protected by
+	 * ctx->completion_lock, so we add this lock here.
+	 */
+	spin_lock_irqsave(&ctx->completion_lock, flags);
+

fs/aio.c static long aio_read_events_ring(struct kioctx *ctx, · line 1109

Quoted 10 lines

fs/aio.c +2 -0

 	ring = kmap_atomic(ctx->ring_pages[0]);
 	ring->head = head;
 	kunmap_atomic(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
+	spin_unlock_irqrestore(&ctx->completion_lock, flags);
+
 	pr_debug("%li  h%u t%u\n", ret, head, tail);
 
 	put_reqs_available(ctx, ret);

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

Tang Chen

author

2014-02-27 10:38 UTC #3 lore raw

PATCH 2/2 aio, mem-hotplug: Add memory barrier to aio ring page migration.

In reply to Tang Chen

When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:

aio_migratepage()

 |-> migrate_page_copy(new, old)
 |   ......				/* Need barrier here */
 |-> ctx->ring_pages[idx] = new

Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.

So add a wmb to synchronize them.

Reported-by Yasuaki Ishimatsu <[email protected]>

Signed-off-by Tang Chen <[email protected]>

---
 fs/aio.c | 8 ++++++++
 1 file changed, 8 insertions(+)

fs/aio.c +8 -0

index 50c089c..f0ed838 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 		pgoff_t idx;
 		spin_lock_irqsave(&ctx->completion_lock, flags);
 		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
 		idx = old->index;
 		if (idx < (pgoff_t)ctx->nr_pages) {
 			/* And only do the move if things haven't changed */

Signature · 1 line

1.8.3.1

Yasuaki Ishimatsu

2014-02-27 12:06 UTC #4 lore raw

Re: [PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

Hi Tang,

Quoted 27 lines · Tang Chen

When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:
aio_migratepage()
  |-> migrate_page_copy(new, old)
  |   ......				/* Need barrier here */
  |-> ctx->ring_pages[idx] = new
Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.
So add a wmb to synchronize them.
Reported-by Yasuaki Ishimatsu <[email protected]>
Signed-off-by Tang Chen <[email protected]>
---
  fs/aio.c | 8 ++++++++
  1 file changed, 8 insertions(+)
fs/aio.c +0 -0
index 50c089c..f0ed838 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
  		pgoff_t idx;

fs/aio.c static int aio_migratepage(struct address_space *mapping, struct page *new, · line 337

Quoted 10 lines

fs/aio.c +8 -0

  		spin_lock_irqsave(&ctx->completion_lock, flags);
  		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+

If you put smp_wmb() here, you should put smp_rmb() before kmap() in aio_read_events_ring().

Thanks,
Yasuaki Ishimatsu

fs/aio.c static int aio_migratepage(struct address_space *mapping, struct page *new, · line 340

Quoted 4 lines

fs/aio.c +0 -0

  		idx = old->index;
  		if (idx < (pgoff_t)ctx->nr_pages) {
  			/* And only do the move if things haven't changed */

Yasuaki Ishimatsu

2014-02-27 12:44 UTC #5 lore raw

PATCH 2/2 aio, mem-hotplug: Add memory barrier to aio ring page migration.

When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:

aio_migratepage()

 |-> migrate_page_copy(new, old)
 |   ......				/* Need barrier here */
 |-> ctx->ring_pages[idx] = new

So add a wmb and rmb to synchronize them.

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Yasuaki Ishimatsu <[email protected]>

---
 fs/aio.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

fs/aio.c +14 -0

index 50c089c..8d9b82b 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 		pgoff_t idx;
 		spin_lock_irqsave(&ctx->completion_lock, flags);
 		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
 		idx = old->index;
 		if (idx < (pgoff_t)ctx->nr_pages) {
 			/* And only do the move if things haven't changed */

@@ -1074,6 +1082,12 @@ static long aio_read_events_ring(struct kioctx *ctx,
 		page = ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE];
 		pos %= AIO_EVENTS_PER_PAGE;
 
+		/*
+		 * Ensure that the page's data was copied from old one by
+		 * aio_migratepage().
+		 */
+		smp_rmb();
+
 		ev = kmap(page);
 		copy_ret = copy_to_user(event + ret, ev + pos,
 					sizeof(*ev) * avail);

Miao Xie

2014-03-04 05:35 UTC #6 lore raw

Re: [Update PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

Quoted 42 lines · Yasuaki Ishimatsu

When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:
aio_migratepage()
 |-> migrate_page_copy(new, old)
 |   ......				/* Need barrier here */
 |-> ctx->ring_pages[idx] = new
Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.
So add a wmb and rmb to synchronize them.
Signed-off-by Tang Chen <[email protected]>
Signed-off-by Yasuaki Ishimatsu <[email protected]>
---
 fs/aio.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
fs/aio.c +8 -0
index 50c089c..8d9b82b 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 		pgoff_t idx;
 		spin_lock_irqsave(&ctx->completion_lock, flags);
 		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
 		idx = old->index;
 		if (idx < (pgoff_t)ctx->nr_pages) {
 			/* And only do the move if things haven't changed */

fs/aio.c static long aio_read_events_ring(struct kioctx *ctx, · line 1090

Quoted 9 lines

fs/aio.c +6 -0

@@ -1074,6 +1082,12 @@ static long aio_read_events_ring(struct kioctx *ctx,
 		page = ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE];
 		pos %= AIO_EVENTS_PER_PAGE;
 
+		/*
+		 * Ensure that the page's data was copied from old one by
+		 * aio_migratepage().
+		 */
+		smp_rmb();
+

smp_read_barrier_depends() is better.

"One could place an A smp_rmb() primitive between the pointer fetch and
dereference. However, this imposes unneeded overhead on systems (such as
i386, IA64, PPC, and SPARC) that respect data dependencies on the read side.
A smp_read_barrier_depends() primitive has been added to the Linux 2.6 kernel to eliminate overhead on these systems."
-- From Chapter 7.1 of <Memory Barriers: a Hardware View for Software Hackers>
Written by Paul E. McKenney

Thanks
Miao

Quoted 10 lines

fs/aio.c +0 -1
 		ev = kmap(page);
 		copy_ret = copy_to_user(event + ret, ev + pos,
 					sizeof(*ev) * avail);
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

KOSAKI Motohiro

2014-03-05 03:04 UTC #7 lore raw

Re: [Update PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

Quoted 9 lines

fs/aio.c static long aio_read_events_ring(struct kioctx *ctx, · line 1090

Quoted 6 lines

fs/aio.c +6 -0

+             /*
+              * Ensure that the page's data was copied from old one by
+              * aio_migratepage().
+              */
+             smp_rmb();
+

smp_read_barrier_depends() is better.

Quoted 8 lines

"One could place an A smp_rmb() primitive between the pointer fetch and
dereference. However, this imposes unneeded overhead on systems (such as
i386, IA64, PPC, and SPARC) that respect data dependencies on the read side.
A smp_read_barrier_depends() primitive has been added to the Linux 2.6 kernel to eliminate overhead on these systems."
-- From Chapter 7.1 of <Memory Barriers: a Hardware View for Software Hackers>
Written by Paul E. McKenney

Right.
The document of memory barrier described this situation.

see https://www.kernel.org/doc/Documentation/memory-barriers.txt

CPU 1 CPU 2
=============== ===============
{ M[0] == 1, M[1] == 2, M[3] = 3, P == 0, Q == 3 } M[1] = 4;
<write barrier>
ACCESS_ONCE(P) = 1

                                     Q = ACCESS_ONCE(P);
                                     <data dependency barrier>
                                     D = M[Q];

Yasuaki Ishimatsu

2014-03-05 06:59 UTC #8 lore raw

Re: [Update PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

In reply to Miao Xie

Quoted 63 lines · Miao Xie

Quoted 42 lines · Yasuaki Ishimatsu
When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:
aio_migratepage()
  |-> migrate_page_copy(new, old)
  |   ......				/* Need barrier here */
  |-> ctx->ring_pages[idx] = new
Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.
So add a wmb and rmb to synchronize them.
Signed-off-by Tang Chen <[email protected]>
Signed-off-by Yasuaki Ishimatsu <[email protected]>
---
  fs/aio.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
fs/aio.c +8 -0
index 50c089c..8d9b82b 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
  		pgoff_t idx;
  		spin_lock_irqsave(&ctx->completion_lock, flags);
  		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
  		idx = old->index;
  		if (idx < (pgoff_t)ctx->nr_pages) {
  			/* And only do the move if things haven't changed */
fs/aio.c static long aio_read_events_ring(struct kioctx *ctx, · line 1090
Quoted 9 lines
fs/aio.c +6 -0
@@ -1074,6 +1082,12 @@ static long aio_read_events_ring(struct kioctx *ctx,
  		page = ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE];
  		pos %= AIO_EVENTS_PER_PAGE;
 
+		/*
+		 * Ensure that the page's data was copied from old one by
+		 * aio_migratepage().
+		 */
+		smp_rmb();
+
smp_read_barrier_depends() is better.
"One could place an A smp_rmb() primitive between the pointer fetch and
dereference. However, this imposes unneeded overhead on systems (such as i386, IA64, PPC, and SPARC) that respect data dependencies on the read side. A smp_read_barrier_depends() primitive has been added to the Linux 2.6 kernel to eliminate overhead on these systems."
-- From Chapter 7.1 of <Memory Barriers: a Hardware View for Software Hackers>
Written by Paul E. McKenney

Quoted 2 lines

Thanks
Miao

Thank you for your comment.
I'll update soon.

Thanks,
Yasauaki Ishimatsu

Quoted 5 lines

fs/aio.c +0 -0

  		ev = kmap(page);
  		copy_ret = copy_to_user(event + ret, ev + pos,
  					sizeof(*ev) * avail);

Quoted 7 lines

Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Yasuaki Ishimatsu

2014-03-05 07:17 UTC #9 lore raw

PATCH v2 2/2 aio, mem-hotplug: Add memory barrier to aio ring page migration.

In reply to Yasuaki Ishimatsu

When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:

aio_migratepage()

 |-> migrate_page_copy(new, old)
 |   ......				/* Need barrier here */
 |-> ctx->ring_pages[idx] = new

So add a wmb and rmb to synchronize them.

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Yasuaki Ishimatsu <[email protected]>

---
v2: change smp_rmb() to smp_read_barrier_depends(). Thanks Miao.

---
 fs/aio.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

fs/aio.c +14 -0

index 50c089c..98c7f2d 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 		pgoff_t idx;
 		spin_lock_irqsave(&ctx->completion_lock, flags);
 		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
 		idx = old->index;
 		if (idx < (pgoff_t)ctx->nr_pages) {
 			/* And only do the move if things haven't changed */

@@ -1074,6 +1082,12 @@ static long aio_read_events_ring(struct kioctx *ctx,
 		page = ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE];
 		pos %= AIO_EVENTS_PER_PAGE;
 
+		/*
+		 * Ensure that the page's data was copied from old one by
+		 * aio_migratepage().
+		 */
+		smp_read_barrier_depends();
+
 		ev = kmap(page);
 		copy_ret = copy_to_user(event + ret, ev + pos,
 					sizeof(*ev) * avail);

Benjamin LaHaise

maintainer

2014-02-27 14:57 UTC #10 lore raw

Re: [PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

In reply to Tang Chen

Quoted 8 lines · Tang Chen

When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:
aio_migratepage()
 |-> migrate_page_copy(new, old)
 |   ......				/* Need barrier here */
 |-> ctx->ring_pages[idx] = new

Quoted 6 lines

Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.
So add a wmb to synchronize them.

The smp_wmb() is not needed after you added the taking of ctx->completion_lock lock since all accesses to ring_pages is then protected by the spinlock. Why are you adding this then? Or have you missed adding the lock somewhere? Also, if you've changed the patch, it is customary to add a "v2" somewhere in the patch title so that I have some idea what version of the patch should be applied.

-ben

Quoted 27 lines

Reported-by Yasuaki Ishimatsu <[email protected]>

Signed-off-by Tang Chen <[email protected]>

---
 fs/aio.c | 8 ++++++++
 1 file changed, 8 insertions(+)

fs/aio.c +8 -0

index 50c089c..f0ed838 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 		pgoff_t idx;
 		spin_lock_irqsave(&ctx->completion_lock, flags);
 		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
 		idx = old->index;
 		if (idx < (pgoff_t)ctx->nr_pages) {
 			/* And only do the move if things haven't changed */

Signature · 1 line

1.8.3.1

Signature · 7 lines

"Thought is the essence of where you are now."

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

Gu Zheng

2014-02-28 01:29 UTC #11 lore raw

Re: [PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

Hi Ben,

Quoted 14 lines · Benjamin LaHaise

Quoted 13 lines · Tang Chen
When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:
aio_migratepage()
 |-> migrate_page_copy(new, old)
 |   ......				/* Need barrier here */
 |-> ctx->ring_pages[idx] = new
Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.

Quoted 8 lines

Quoted 1 line
So add a wmb to synchronize them.
The smp_wmb() is not needed after you added the taking of ctx->completion_lock lock since all accesses to ring_pages is then protected by the spinlock. Why are you adding this then? Or have you missed adding the lock somewhere? Also, if you've changed the patch, it is customary to add a "v2" somewhere in the patch title so that I have some idea what version of the patch should be applied.

The completion_lock just protects updating ring->head when reading events, so wmb is still needed here.

Regards,
Gu

Quoted 31 lines

-ben

Quoted 27 lines

Reported-by Yasuaki Ishimatsu <[email protected]>

Signed-off-by Tang Chen <[email protected]>

---
 fs/aio.c | 8 ++++++++
 1 file changed, 8 insertions(+)

fs/aio.c +8 -0

index 50c089c..f0ed838 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 		pgoff_t idx;
 		spin_lock_irqsave(&ctx->completion_lock, flags);
 		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.
+		 */
+		smp_wmb();
+
 		idx = old->index;
 		if (idx < (pgoff_t)ctx->nr_pages) {
 			/* And only do the move if things haven't changed */

Signature · 1 line

1.8.3.1

Yasuaki Ishimatsu

2014-02-28 07:25 UTC #12 lore raw

Re: [PATCH 2/2] aio, mem-hotplug: Add memory barrier to aio ring page migration.

In reply to Benjamin LaHaise

Quoted 14 lines · Benjamin LaHaise

Quoted 13 lines · Tang Chen
When doing aio ring page migration, we migrated the page, and update ctx->ring_pages[]. Like the following:
aio_migratepage()
  |-> migrate_page_copy(new, old)
  |   ......				/* Need barrier here */
  |-> ctx->ring_pages[idx] = new
Actually, we need a memory barrier between these two operations. Otherwise, if ctx->ring_pages[] is updated before memory copy due to the compiler optimization, other processes may have an opportunity to access to the not fully initialized new ring page.

Quoted 5 lines

Quoted 1 line
So add a wmb to synchronize them.
The smp_wmb() is not needed after you added the taking of ctx->completion_lock lock since all accesses to ring_pages is then protected by the spinlock. Why are you adding this then? Or have you missed adding the lock somewhere?

Pleaes see following thread. It's a updated patch.

https://lkml.org/lkml/2014/2/27/237

aio_read_events_ring() accesses ring_pages without locking ctx-completion_lock. If copying old page'date to new page runs after new page was set to ctx->ring_pages by changing order, aio_read_events_ring() may not work correctly.

So smp_wmb() and smp_rmb() is needed.

Thanks,
Yasuaki Ishimatsu

Quoted 28 lines

Also, if you've changed the patch, it is customary to add a "v2" somewhere in the patch title so that I have some idea what version of the patch should be applied.
-ben
Quoted 19 lines
Reported-by Yasuaki Ishimatsu <[email protected]>
Signed-off-by Tang Chen <[email protected]>
---
  fs/aio.c | 8 ++++++++
  1 file changed, 8 insertions(+)
fs/aio.c +5 -0
index 50c089c..f0ed838 100644
@@ -327,6 +327,14 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
  		pgoff_t idx;
  		spin_lock_irqsave(&ctx->completion_lock, flags);
  		migrate_page_copy(new, old);
+
+		/*
+		 * Ensure memory copy is finished before updating
+		 * ctx->ring_pages[]. Otherwise other processes may access to
+		 * new ring pages which are not fully initialized.

Quoted 9 lines

Quoted 8 lines

fs/aio.c +3 -1

+		 */
+		smp_wmb();
+
  		idx = old->index;
  		if (idx < (pgoff_t)ctx->nr_pages) {
  			/* And only do the move if things haven't changed */
--

1.8.3.1