Fix aio performance regression for database caused by THP

RFC PATCH In review Khalid Aziz 5 participants 7 messages Aug 15, 2013 – Aug 16, 2013

Subsystems (2)

Stable branch Memory management

lore

Share & tools

lore

Khalid Aziz

author

2013-08-15 18:13 UTC #0 lore raw

RFC PATCH 1/1 Fix aio performance regression for database caused by THP

I am working with a tool that simulates oracle database I/O workload.
This tool (orion to be specific -
<http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#autoId24>) allocates hugetlbfs pages using shmget() with SHM_HUGETLB flag. It then does aio into these pages from flash disks using various common block sizes used by database. I am looking at performance with two of the most common block sizes - 1M and 64K. aio performance with these two block sizes plunged after Transparent HugePages was introduced in the kernel. Here are performance numbers:

		pre-THP		2.6.39		3.11-rc5

1M read 8384 MB/s 5629 MB/s 6501 MB/s
64K read 7867 MB/s 4576 MB/s 4251 MB/s

I have narrowed the performance impact down to the overheads introduced by THP in __get_page_tail() and put_compound_page() routines. perf top shows >40% of cycles being spent in these two routines. Every time direct I/O to hugetlbfs pages starts, kernel calls get_page() to grab a reference to the pages and calls put_page() when I/O completes to put the reference away. THP introduced significant amount of locking overhead to get_page() and put_page() when dealing with compound pages because hugepages can be split underneath get_page() and put_page(). It added this overhead irrespective of whether it is dealing with hugetlbfs pages or transparent hugepages. This resulted in 20%-45% drop in aio performance when using hugetlbfs pages.

Since hugetlbfs pages can not be split, there is no reason to go through all the locking overhead for these pages from what I can see. I added code to __get_page_tail() and put_compound_page() to bypass all the locking code when working with hugetlbfs pages. This improved performance significantly. Performance numbers with this patch:

		pre-THP		3.11-rc5	3.11-rc5 + Patch

1M read 8384 MB/s 6501 MB/s 8371 MB/s
64K read 7867 MB/s 4251 MB/s 6510 MB/s

Performance with 64K read is still lower than what it was before THP, but still a 53% improvement. It does mean there is more work to be done but I will take a 53% improvement for now.

Please take a look at the following patch and let me know if it looks reasonable.

Signed-off-by Khalid Aziz <[email protected]>

---
 mm/swap.c |   77 +++++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 52 insertions(+), 25 deletions(-)

mm/swap.c +52 ~~-25~~

index 62b78a6..cc8326f 100644
@@ -31,6 +31,7 @@
 #include <linux/memcontrol.h>
 #include <linux/gfp.h>
 #include <linux/uio.h>
+#include <linux/hugetlb.h>
 
 #include "internal.h"

@@ -81,6 +82,19 @@ static void __put_compound_page(struct page *page)
 
 static void put_compound_page(struct page *page)
 {
+	/*
+	 * hugetlbfs pages can not be split from under us. If this
+	 * is a hugetlbfs page, check refcount on head page and release
+	 * the page if refcount is zero.
+	 */
+	if (PageHuge(page)) {
+		page = compound_head(page);
+		if (put_page_testzero(page))
+			__put_compound_page(page);
+
+		return;
+	}
+
 	if (unlikely(PageTail(page))) {
 		/* __split_huge_page_refcount can run under us */
 		struct page *page_head = compound_trans_head(page);

@@ -184,38 +198,51 @@ bool __get_page_tail(struct page *page)
 	 * proper PT lock that already serializes against
 	 * split_huge_page().
 	 */
-	unsigned long flags;
 	bool got = false;
-	struct page *page_head = compound_trans_head(page);
+	struct page *page_head;
 
-	if (likely(page != page_head && get_page_unless_zero(page_head))) {
+	/*
+	 * If this is a hugetlbfs page, it can not be split under
+	 * us. Simply increment refcount for head page
+	 */
+	if (PageHuge(page)) {
+		page_head = compound_head(page);
+		atomic_inc(&page_head->_count);
+		got = true;
+	} else {
+		unsigned long flags;
+
+		page_head = compound_trans_head(page);
+		if (likely(page != page_head &&
+					get_page_unless_zero(page_head))) {
+
+			/* Ref to put_compound_page() comment. */
+			if (PageSlab(page_head)) {
+				if (likely(PageTail(page))) {
+					__get_page_tail_foll(page, false);
+					return true;
+				} else {
+					put_page(page_head);
+					return false;
+				}
+			}
 
-		/* Ref to put_compound_page() comment. */
-		if (PageSlab(page_head)) {
+			/*
+			 * page_head wasn't a dangling pointer but it
+			 * may not be a head page anymore by the time
+			 * we obtain the lock. That is ok as long as it
+			 * can't be freed from under us.
+			 */
+			flags = compound_lock_irqsave(page_head);
+			/* here __split_huge_page_refcount won't run anymore */
 			if (likely(PageTail(page))) {
 				__get_page_tail_foll(page, false);
-				return true;
-			} else {
-				put_page(page_head);
-				return false;
+				got = true;
 			}
+			compound_unlock_irqrestore(page_head, flags);
+			if (unlikely(!got))
+				put_page(page_head);
 		}
-
-		/*
-		 * page_head wasn't a dangling pointer but it
-		 * may not be a head page anymore by the time
-		 * we obtain the lock. That is ok as long as it
-		 * can't be freed from under us.
-		 */
-		flags = compound_lock_irqsave(page_head);
-		/* here __split_huge_page_refcount won't run anymore */
-		if (likely(PageTail(page))) {
-			__get_page_tail_foll(page, false);
-			got = true;
-		}
-		compound_unlock_irqrestore(page_head, flags);
-		if (unlikely(!got))
-			put_page(page_head);
 	}
 	return got;
 }

Signature · 9 lines

1.7.10.4



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andi Kleen

2013-08-15 19:34 UTC #1 lore raw

Quoted 4 lines · Khalid Aziz

I am working with a tool that simulates oracle database I/O workload. This tool (orion to be specific -
<http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#autoId24>)
allocates hugetlbfs pages using shmget() with SHM_HUGETLB flag.

Is this tool available for download?

I would rather prefer to address the locking overhead in THP too.

The fundamental problem is that we have to touch all the pages?

-Andi

Signature · 7 lines

[email protected] -- Speaking for myself only

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Khalid Aziz

author

2013-08-15 20:17 UTC #2 lore raw

Quoted 8 lines · Andi Kleen

Quoted 4 lines · Khalid Aziz
I am working with a tool that simulates oracle database I/O workload. This tool (orion to be specific -
<http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#autoId24>)
allocates hugetlbfs pages using shmget() with SHM_HUGETLB flag.
Is this tool available for download?

I am not sure if it is available separately. It is part of Oracle 11gR2 release.

Quoted 4 lines

I would rather prefer to address the locking overhead in THP too.
The fundamental problem is that we have to touch all the pages?

Not so much that THP has to touch all the pages, rather it checks the head and tail flags multiple times with the assumption they could change underneath. Then it executes memory barriers (in compound_trans_head()) to force any updates to these flags from all cores. It also locks the head page (in put_compound_page()) which stops any other thread trying to put reference to one of the pages in the compound page. I see a lot of cycles being spent in compound_trans_head() and various atomic operations in the path added by THP.

--
Khalid

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

maintainer

2013-08-15 21:16 UTC #3 lore raw

In reply to Khalid Aziz

Quoted 30 lines · On Thu, 15 Aug 2013 12:13:09 -0600 Khalid Aziz <[email protected]> wrote:

I am working with a tool that simulates oracle database I/O workload.
This tool (orion to be specific -
<http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#autoId24>) allocates hugetlbfs pages using shmget() with SHM_HUGETLB flag. It then does aio into these pages from flash disks using various common block sizes used by database. I am looking at performance with two of the most common block sizes - 1M and 64K. aio performance with these two block sizes plunged after Transparent HugePages was introduced in the kernel. Here are performance numbers:
		pre-THP		2.6.39		3.11-rc5
1M read 8384 MB/s 5629 MB/s 6501 MB/s
64K read 7867 MB/s 4576 MB/s 4251 MB/s
I have narrowed the performance impact down to the overheads introduced by THP in __get_page_tail() and put_compound_page() routines. perf top shows >40% of cycles being spent in these two routines. Every time direct I/O to hugetlbfs pages starts, kernel calls get_page() to grab a reference to the pages and calls put_page() when I/O completes to put the reference away. THP introduced significant amount of locking overhead to get_page() and put_page() when dealing with compound pages because hugepages can be split underneath get_page() and put_page(). It added this overhead irrespective of whether it is dealing with hugetlbfs pages or transparent hugepages. This resulted in 20%-45% drop in aio performance when using hugetlbfs pages.
Since hugetlbfs pages can not be split, there is no reason to go through all the locking overhead for these pages from what I can see. I added code to __get_page_tail() and put_compound_page() to bypass all the locking code when working with hugetlbfs pages. This improved performance significantly. Performance numbers with this patch:
		pre-THP		3.11-rc5	3.11-rc5 + Patch
1M read 8384 MB/s 6501 MB/s 8371 MB/s
64K read 7867 MB/s 4251 MB/s 6510 MB/s

Quoted 6 lines

Performance with 64K read is still lower than what it was before THP, but still a 53% improvement. It does mean there is more work to be done but I will take a 53% improvement for now.
Please take a look at the following patch and let me know if it looks reasonable.

Pretty convincing.

I tagged this for a -stable backport. To allow time for review and testing I'll plan to merge the patch into 3.12-rc1, so it should materialize in 3.11.x (and hopefully earlier) stable kernels after that.

To facilitate backporting the patch could have been quite a bit smaller, with some simple restructuring. It applies OK to 3.10, but not 3.9. Hopefully that's good enough...

Greg KH

maintainer

2013-08-15 23:20 UTC #4 lore raw

Quoted 7 lines · Andrew Morton

I tagged this for a -stable backport. To allow time for review and testing I'll plan to merge the patch into 3.12-rc1, so it should materialize in 3.11.x (and hopefully earlier) stable kernels after that.
To facilitate backporting the patch could have been quite a bit smaller, with some simple restructuring. It applies OK to 3.10, but not 3.9. Hopefully that's good enough...

3.9 is end-of-life, and I do not know anyone who is wishing to try to keep it going, so don't worry about it anymore. 3.10 is great to have if possible.

thanks,

greg k-h

Kirill A. Shutemov

2013-08-16 09:03 UTC #5 lore raw

In reply to Khalid Aziz

Quoted 83 lines · Khalid Aziz

I am working with a tool that simulates oracle database I/O workload.
This tool (orion to be specific -
<http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#autoId24>) allocates hugetlbfs pages using shmget() with SHM_HUGETLB flag. It then does aio into these pages from flash disks using various common block sizes used by database. I am looking at performance with two of the most common block sizes - 1M and 64K. aio performance with these two block sizes plunged after Transparent HugePages was introduced in the kernel. Here are performance numbers:
		pre-THP		2.6.39		3.11-rc5
1M read 8384 MB/s 5629 MB/s 6501 MB/s
64K read 7867 MB/s 4576 MB/s 4251 MB/s
I have narrowed the performance impact down to the overheads introduced by THP in __get_page_tail() and put_compound_page() routines. perf top shows >40% of cycles being spent in these two routines. Every time direct I/O to hugetlbfs pages starts, kernel calls get_page() to grab a reference to the pages and calls put_page() when I/O completes to put the reference away. THP introduced significant amount of locking overhead to get_page() and put_page() when dealing with compound pages because hugepages can be split underneath get_page() and put_page(). It added this overhead irrespective of whether it is dealing with hugetlbfs pages or transparent hugepages. This resulted in 20%-45% drop in aio performance when using hugetlbfs pages.
Since hugetlbfs pages can not be split, there is no reason to go through all the locking overhead for these pages from what I can see. I added code to __get_page_tail() and put_compound_page() to bypass all the locking code when working with hugetlbfs pages. This improved performance significantly. Performance numbers with this patch:
		pre-THP		3.11-rc5	3.11-rc5 + Patch
1M read 8384 MB/s 6501 MB/s 8371 MB/s
64K read 7867 MB/s 4251 MB/s 6510 MB/s
Performance with 64K read is still lower than what it was before THP, but still a 53% improvement. It does mean there is more work to be done but I will take a 53% improvement for now.
Please take a look at the following patch and let me know if it looks reasonable.
Signed-off-by Khalid Aziz <[email protected]>
---
 mm/swap.c |   77 +++++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 52 insertions(+), 25 deletions(-)
mm/swap.c +15 -2
index 62b78a6..cc8326f 100644
@@ -31,6 +31,7 @@
 #include <linux/memcontrol.h>
 #include <linux/gfp.h>
 #include <linux/uio.h>
+#include <linux/hugetlb.h>
 
 #include "internal.h"
 
@@ -81,6 +82,19 @@ static void __put_compound_page(struct page *page)
 
 static void put_compound_page(struct page *page)
 {
+	/*
+	 * hugetlbfs pages can not be split from under us. If this
+	 * is a hugetlbfs page, check refcount on head page and release
+	 * the page if refcount is zero.
+	 */
+	if (PageHuge(page)) {
+		page = compound_head(page);
+		if (put_page_testzero(page))
+			__put_compound_page(page);
+
+		return;
+	}
+
 	if (unlikely(PageTail(page))) {
 		/* __split_huge_page_refcount can run under us */
 		struct page *page_head = compound_trans_head(page);
@@ -184,38 +198,51 @@ bool __get_page_tail(struct page *page)
 	 * proper PT lock that already serializes against
 	 * split_huge_page().
 	 */
-	unsigned long flags;
 	bool got = false;
-	struct page *page_head = compound_trans_head(page);
+	struct page *page_head;

mm/swap.c bool __get_page_tail(struct page *page) · line 211

Quoted 10 lines

mm/swap.c +8 -1

 
-	if (likely(page != page_head && get_page_unless_zero(page_head))) {
+	/*
+	 * If this is a hugetlbfs page, it can not be split under
+	 * us. Simply increment refcount for head page
+	 */
+	if (PageHuge(page)) {
+		page_head = compound_head(page);
+		atomic_inc(&page_head->_count);
+		got = true;

Why not just return here and don't increase indentantion level for rest of the function?

Signature · 7 lines

 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Khalid Aziz

author

2013-08-16 14:14 UTC #6 lore raw

Quoted 15 lines · Kirill A. Shutemov

mm/swap.c bool __get_page_tail(struct page *page) · line 211
Quoted 10 lines · Khalid Aziz
mm/swap.c +8 -1
-	if (likely(page != page_head && get_page_unless_zero(page_head))) {
+	/*
+	 * If this is a hugetlbfs page, it can not be split under
+	 * us. Simply increment refcount for head page
+	 */
+	if (PageHuge(page)) {
+		page_head = compound_head(page);
+		atomic_inc(&page_head->_count);
+		got = true;
Why not just return here and don't increase indentantion level for rest of the function?

Good point.

Andrew, I can rework the patch if you would like.

Thanks,
Khalid