slub: Avoid direct compaction if possible

PATCH In review Roman Gushchin 5 participants 13 messages Jun 14, 2013 – Jul 18, 2013

Subsystems (4)

Memory management MM core Page allocator Slab allocator

lore

Share & tools

lore

Roman Gushchin

author

2013-06-14 13:18 UTC #0 lore raw

PATCH 1/1 slub: Avoid direct compaction if possible

Slub tries to use contiguous pages to fasten allocations and to minimize management overhead. If necessary, it can easily fall back to the minimum order allocations.

Slub tries to allocate contiguous pages even if memory is fragmented and there are no free contiguous pages. In this case it calls direct compaction to allocate contiguous page. Compaction requires the taking of some heavily contended locks (e.g. zone locks). So, running compaction (direct and using kswapd) simultaneously on several processors can cause serious performance issues.

It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.

Signed-off-by Roman Gushchin <[email protected]>

---
 include/linux/gfp.h | 4 +++-
 mm/page_alloc.c     | 3 +++
 mm/slub.c           | 3 ++-
 3 files changed, 8 insertions(+), 2 deletions(-)

include/linux/gfp.h +3 -1

index 0f615eb..073a90a 100644
@@ -35,6 +35,7 @@ struct vm_area_struct;
 #define ___GFP_NO_KSWAPD	0x400000u
 #define ___GFP_OTHER_NODE	0x800000u
 #define ___GFP_WRITE		0x1000000u
+#define ___GFP_NOCOMPACT	0x2000000u
 /* If the above are modified, __GFP_BITS_SHIFT may need updating */
 
 /*

@@ -92,6 +93,7 @@ struct vm_area_struct;
 #define __GFP_OTHER_NODE ((__force gfp_t)___GFP_OTHER_NODE) /* On behalf of other node */
 #define __GFP_KMEMCG	((__force gfp_t)___GFP_KMEMCG) /* Allocation comes from a memcg-accounted resource */
 #define __GFP_WRITE	((__force gfp_t)___GFP_WRITE)	/* Allocator intends to dirty page */
+#define __GFP_NOCOMPACT ((__force gfp_t)___GFP_NOCOMPACT) /* Avoid direct compaction */
 
 /*
  * This may seem redundant, but it's a way of annotating false positives vs.

@@ -99,7 +101,7 @@ struct vm_area_struct;
  */
 #define __GFP_NOTRACK_FALSE_POSITIVE (__GFP_NOTRACK)
 
-#define __GFP_BITS_SHIFT 25	/* Room for N __GFP_FOO bits */
+#define __GFP_BITS_SHIFT 26	/* Room for N __GFP_FOO bits */
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
 
 /* This equals 0, but use constants in case they ever change */

mm/page_alloc.c +3 -0

index c3edb62..292562f 100644
@@ -2482,6 +2482,9 @@ rebalance:
 	if (test_thread_flag(TIF_MEMDIE) && !(gfp_mask & __GFP_NOFAIL))
 		goto nopage;
 
+	if (order && (gfp_mask & __GFP_NOCOMPACT))
+		goto nopage;
+
 	/*
 	 * Try direct compaction. The first pass is asynchronous. Subsequent
 	 * attempts after direct reclaim are synchronous

mm/slub.c +2 -1

index 57707f0..a38733b 100644
@@ -1287,7 +1287,8 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	 * Let the initial higher-order allocation fail under memory pressure
 	 * so we fall-back to the minimum order allocation.
 	 */
-	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
+	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOCOMPACT) &
+		~__GFP_NOFAIL;
 
 	page = alloc_slab_page(alloc_gfp, node, oo);
 	if (unlikely(!page)) {

Signature · 7 lines

1.8.1.2

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Christoph Lameter

reviewer

2013-06-14 14:32 UTC #1 lore raw

Quoted 6 lines · Roman Gushchin

Slub tries to allocate contiguous pages even if memory is fragmented and there are no free contiguous pages. In this case it calls direct compaction to allocate contiguous page. Compaction requires the taking of some heavily contended locks (e.g. zone locks). So, running compaction (direct and using kswapd) simultaneously on several processors can cause serious performance issues.

The main thing that this patch does is to add a nocompact flag to the page allocator. That needs to be a separate patch. Also fix the description. Slub does not invoke compaction. The page allocator initiates compaction under certain conditions.

Quoted 5 lines

It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.

Sounds like a good idea. Do you have some numbers to show the effect of this patch?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Roman Gushchin

author

2013-06-14 15:17 UTC #2 lore raw

Quoted 9 lines · Christoph Lameter

Quoted 6 lines · Roman Gushchin
Slub tries to allocate contiguous pages even if memory is fragmented and there are no free contiguous pages. In this case it calls direct compaction to allocate contiguous page. Compaction requires the taking of some heavily contended locks (e.g. zone locks). So, running compaction (direct and using kswapd) simultaneously on several processors can cause serious performance issues.

Quoted 4 lines

The main thing that this patch does is to add a nocompact flag to the page allocator. That needs to be a separate patch. Also fix the description. Slub does not invoke compaction. The page allocator initiates compaction under certain conditions.

Ok, I'll do.

Quoted 9 lines

Quoted 6 lines
It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.
Sounds like a good idea. Do you have some numbers to show the effect of this patch?

No.
It seems that any numbers here depend on memory fragmentation, so it's not easy to make a reproducible measurement. If you have any ideas here, you are welcome.

But there is an actual problem, that this patch solves. Sometimes I saw the following issue on some machines:
all CPUs are performing compaction, system time is about 80%, system is completely unreliable. It occurs only on machines with specific workload (distributed data storage system, so, intensive disk i/o is performed). A system can fall into this state fast and unexpectedly or by progressive degradation.

This patch solves this problem.

Thank you for your comments and suggestions!

Regards,
Roman

Christoph Lameter

reviewer

2013-06-14 16:08 UTC #3 lore raw

Quoted 7 lines · Roman Gushchin

But there is an actual problem, that this patch solves. Sometimes I saw the following issue on some machines:
all CPUs are performing compaction, system time is about 80%, system is completely unreliable. It occurs only on machines with specific workload (distributed data storage system, so, intensive disk i/o is performed). A system can fall into this state fast and unexpectedly or by progressive degradation.

Well that is not a slab allocator specific issue but related to compaction concurrency. Likely cache line contention is causing a severe slowday. But that issue could be triggered by any subsystem that does lots of memory allocations. I would suggest that we try to address the problem in the compaction logic rather than modifying allocators.

Mel?

Roman Gushchin

author

2013-06-14 16:52 UTC #4 lore raw

Quoted 10 lines · Christoph Lameter

Quoted 7 lines · Roman Gushchin
But there is an actual problem, that this patch solves. Sometimes I saw the following issue on some machines:
all CPUs are performing compaction, system time is about 80%, system is completely unreliable. It occurs only on machines with specific workload (distributed data storage system, so, intensive disk i/o is performed). A system can fall into this state fast and unexpectedly or by progressive degradation.

Quoted 5 lines

Well that is not a slab allocator specific issue but related to compaction concurrency. Likely cache line contention is causing a severe slowday. But that issue could be triggered by any subsystem that does lots of memory allocations. I would suggest that we try to address the problem in the compaction logic rather than modifying allocators.

I agree, that it's good to address the original issue. But I'm not sure, that it's a compaction issue. If someone wants to participate here, I can provide more information. The main problem here is that it's __very__ hard to reproduce the issue.

But, I think, all that shouldn't stop us from modifying the allocator. Falling back to minimal order is in any case better than running direct compaction. Just because it's faster. Am I wrong?

Regards,
Roman

David Rientjes

reviewer

2013-06-14 20:26 UTC #5 lore raw

In reply to Christoph Lameter

Quoted 9 lines · Christoph Lameter

Quoted 5 lines
It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.
Sounds like a good idea. Do you have some numbers to show the effect of this patch?

I'm surprised you like this patch, it basically makes slub allocations to be atomic and doesn't try memory compaction nor reclaim. Asynchronous compaction certainly isn't aggressive enough to mimick the effects of the old lumpy reclaim that would have resulted in less fragmented memory. If slub is the only thing that is doing high-order allocations, it will start falling back to the smallest page order much much more often.

I agree that this doesn't seem like a slub issue at all but rather a page allocator issue; if we have many simultaneous thp faults at the same time and /sys/kernel/mm/transparent_hugepage/defrag is "always" then you'll get the same problem if deferred compaction isn't helping.

So I don't think we should be patching slub in any special way here.

Roman, are you using the latest kernel? If so, what does grep compact_ /proc/vmstat show after one or more of these events?

Roman Gushchin

author

2013-06-17 12:34 UTC #6 lore raw

Quoted 24 lines · David Rientjes

Quoted 9 lines · Christoph Lameter
Quoted 5 lines
It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.
Sounds like a good idea. Do you have some numbers to show the effect of this patch?
I'm surprised you like this patch, it basically makes slub allocations to be atomic and doesn't try memory compaction nor reclaim. Asynchronous compaction certainly isn't aggressive enough to mimick the effects of the old lumpy reclaim that would have resulted in less fragmented memory. If slub is the only thing that is doing high-order allocations, it will start falling back to the smallest page order much much more often.
I agree that this doesn't seem like a slub issue at all but rather a page allocator issue; if we have many simultaneous thp faults at the same time and /sys/kernel/mm/transparent_hugepage/defrag is "always" then you'll get the same problem if deferred compaction isn't helping.

Quoted 5 lines

So I don't think we should be patching slub in any special way here.
Roman, are you using the latest kernel? If so, what does grep compact_ /proc/vmstat show after one or more of these events?

We're using 3.4. And the problem reveals when we moved from 3.2 to 3.4. It can be also reproduced on 3.5.

I'll send the exact numbers as soon I'll reproduce it again. It can take up to 1 week.

Thanks!

Regards,
Roman

Michal Hocko

2013-06-17 14:27 UTC #7 lore raw

Quoted 25 lines · Roman Gushchin

Quoted 24 lines · David Rientjes
Quoted 9 lines · Christoph Lameter
Quoted 5 lines
It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.
Sounds like a good idea. Do you have some numbers to show the effect of this patch?
I'm surprised you like this patch, it basically makes slub allocations to be atomic and doesn't try memory compaction nor reclaim. Asynchronous compaction certainly isn't aggressive enough to mimick the effects of the old lumpy reclaim that would have resulted in less fragmented memory. If slub is the only thing that is doing high-order allocations, it will start falling back to the smallest page order much much more often.
I agree that this doesn't seem like a slub issue at all but rather a page allocator issue; if we have many simultaneous thp faults at the same time and /sys/kernel/mm/transparent_hugepage/defrag is "always" then you'll get the same problem if deferred compaction isn't helping.

Quoted 8 lines

Quoted 5 lines
So I don't think we should be patching slub in any special way here.
Roman, are you using the latest kernel? If so, what does grep compact_ /proc/vmstat show after one or more of these events?
We're using 3.4. And the problem reveals when we moved from 3.2 to 3.4. It can be also reproduced on 3.5.

FWIW, there were some compaction locking related patches merged around 3.7. See 2a1402aa044b55c2d30ab0ed9405693ef06fb07c and follow ups.

Quoted 13 lines

I'll send the exact numbers as soon I'll reproduce it again. It can take up to 1 week.
Thanks!
Regards,
Roman
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Roman Gushchin

author

2013-06-17 14:54 UTC #8 lore raw

Quoted 32 lines · Michal Hocko

Quoted 31 lines · Roman Gushchin
Quoted 29 lines · David Rientjes
Quoted 9 lines · Christoph Lameter
Quoted 5 lines
It's possible to avoid such problems (or at least to make them less probable) by avoiding direct compaction. If it's not possible to allocate a contiguous page without compaction, slub will fall back to order 0 page(s). In this case kswapd will be woken to perform asynchronous compaction. So, slub can return to default order allocations as soon as memory will be de-fragmented.
Sounds like a good idea. Do you have some numbers to show the effect of this patch?
I'm surprised you like this patch, it basically makes slub allocations to be atomic and doesn't try memory compaction nor reclaim. Asynchronous compaction certainly isn't aggressive enough to mimick the effects of the old lumpy reclaim that would have resulted in less fragmented memory. If slub is the only thing that is doing high-order allocations, it will start falling back to the smallest page order much much more often.
I agree that this doesn't seem like a slub issue at all but rather a page allocator issue; if we have many simultaneous thp faults at the same time and /sys/kernel/mm/transparent_hugepage/defrag is "always" then you'll get the same problem if deferred compaction isn't helping.
So I don't think we should be patching slub in any special way here.
Roman, are you using the latest kernel? If so, what does grep compact_ /proc/vmstat show after one or more of these events?

Quoted 5 lines

Quoted 2 lines
We're using 3.4. And the problem reveals when we moved from 3.2 to 3.4. It can be also reproduced on 3.5.
FWIW, there were some compaction locking related patches merged around 3.7. See 2a1402aa044b55c2d30ab0ed9405693ef06fb07c and follow ups.

Thanks, Michal.
I've already tried to backport some of those patches, but it didn't help (may be it wasn't enough).
I'll try to reproduce the issue on raw 3.9.

Regards,
Roman

David Rientjes

reviewer

2013-06-17 21:44 UTC #9 lore raw

Quoted 6 lines · Roman Gushchin

Quoted 2 lines
FWIW, there were some compaction locking related patches merged around 3.7. See 2a1402aa044b55c2d30ab0ed9405693ef06fb07c and follow ups.
Thanks, Michal.
I've already tried to backport some of those patches, but it didn't help (may be it wasn't enough).

They certainly aren't enough, the kernel you're running suffers from a couple different memory compaction issues that were fixed in 3.7. I couldn't sympathize with your situation more, I faced the same issue because of thp and not slub (we use slab).

Quoted 2 lines

I'll try to reproduce the issue on raw 3.9.

Thanks. If you need to go back to 3.4, try using these, they significantly helped our issues:

bb13ffeb9f6bfeb301443994dfbf29f91117dfb3
627260595ca6abcb16d68a3732bac6b547e112d6
c89511ab2f8fe2b47585e60da8af7fd213ec877e
62997027ca5b3d4618198ed8b1aba40b61b1137b
a9aacbccf3145355190d87f0df1731fb84fdd8c8

Roman Gushchin

author

2013-06-27 08:49 UTC #10 lore raw

Quoted 9 lines · David Rientjes

On Mon, 17 Jun 2013, Roman Gushchin wrote:
They certainly aren't enough, the kernel you're running suffers from a couple different memory compaction issues that were fixed in 3.7. I couldn't sympathize with your situation more, I faced the same issue because of thp and not slub (we use slab).
Quoted 2 lines
I'll try to reproduce the issue on raw 3.9.

I can't reproduce the issue on 3.9.
It seems that compaction fixes in 3.7 solve the problem.

Quoted 9 lines

Thanks. If you need to go back to 3.4, try using these, they significantly helped our issues:
bb13ffeb9f6bfeb301443994dfbf29f91117dfb3
627260595ca6abcb16d68a3732bac6b547e112d6
c89511ab2f8fe2b47585e60da8af7fd213ec877e
62997027ca5b3d4618198ed8b1aba40b61b1137b
a9aacbccf3145355190d87f0df1731fb84fdd8c8

Thank you!

Regards,
Roman

David Rientjes

reviewer

2013-06-27 20:41 UTC #11 lore raw

Quoted 11 lines · Roman Gushchin

Quoted 7 lines
They certainly aren't enough, the kernel you're running suffers from a couple different memory compaction issues that were fixed in 3.7. I couldn't sympathize with your situation more, I faced the same issue because of thp and not slub (we use slab).
Quoted 2 lines
I'll try to reproduce the issue on raw 3.9.
I can't reproduce the issue on 3.9.
It seems that compaction fixes in 3.7 solve the problem.

Yeah, we had significant problems with memory compaction in 3.3 and 3.4 kernels, so if you need to run with such a kernel you'll want to backport the listed commits. I'm not sure we could get such invasive changes into a stable release, unfortunately.

Vinson Lee

2013-07-18 20:29 UTC #12 lore raw

Quoted 10 lines · David Rientjes

Quoted 8 lines · Roman Gushchin
Quoted 7 lines
They certainly aren't enough, the kernel you're running suffers from a couple different memory compaction issues that were fixed in 3.7. I couldn't sympathize with your situation more, I faced the same issue because of thp and not slub (we use slab).
Quoted 2 lines
I'll try to reproduce the issue on raw 3.9.

Quoted 8 lines

Quoted 3 lines
I can't reproduce the issue on 3.9.
It seems that compaction fixes in 3.7 solve the problem.
Yeah, we had significant problems with memory compaction in 3.3 and 3.4 kernels, so if you need to run with such a kernel you'll want to backport the listed commits. I'm not sure we could get such invasive changes into a stable release, unfortunately.

Hi.

Can the diff with the backport of the listed patches be posted? The patches don't easily apply to 3.4.

I think I'm seeing memory compaction issues as well with Linux kernel 3.4 and would welcome any help. I'm planning to try the backporting approach to see if it improves things on my side.

Cheers,
Vinson