Aggressively allocate the pages on cma reserved memory

We already have retry logic in migrate_pages(). It does retry 10 times. So if we keep this retrying code in __alloc_contig_migrate_range(), we would try to migrate some unmigratable page in 50 times. There is just one small difference in -ENOMEM case. migrate_pages() don't do retry in this case, however, current __alloc_contig_migrate_range() does. But, I think that this isn't problem, because in this case, we may fail again with same reason.

Signed-off-by Joonsoo Kim <[email protected]>

mm/page_alloc.c +4 -5

index 5dba293..674ade7 100644
@@ -6185,7 +6185,6 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
 	/* This function is based on compact_zone() from compaction.c. */
 	unsigned long nr_reclaimed;
 	unsigned long pfn = start;
-	unsigned int tries = 0;
 	int ret = 0;
 
 	migrate_prep();

@@ -6204,10 +6203,6 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
 				ret = -EINTR;
 				break;
 			}
-			tries = 0;
-		} else if (++tries == 5) {
-			ret = ret < 0 ? ret : -EBUSY;
-			break;
 		}
 
 		nr_reclaimed = reclaim_clean_pages_from_list(cc->zone,

@@ -6216,6 +6211,10 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
 
 		ret = migrate_pages(&cc->migratepages, alloc_migrate_target,
 				    0, MIGRATE_SYNC, MR_CMA);
+		if (ret) {
+			ret = ret < 0 ? ret : -EBUSY;
+			break;
+		}
 	}
 	if (ret < 0) {
 		putback_movable_pages(&cc->migratepages);

Signed-off-by Joonsoo Kim <[email protected]>

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess.

So, I implement #2 approach.
One thing I should note is that we should not change allocation target (movable list or cma) on each allocation attempt, since this prevent allocated pages to be in physically succession, so some I/O devices can be hurt their performance. To solve this, I keep allocation target in at least pageblock_nr_pages attempts and make this number reflect ratio, free pages without free cma pages to free cma pages. With this approach, system works very smoothly and fully utilize the pages on cma reserved memory.

Following is the experimental result of this patch.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

<Before>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.8           361.8
Average-MemFree:        283880 KB       530851 KB
pswpin:                 7               110064
pswpout:                452             767502

<After>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.2           235.6
Average-MemFree:        281651 KB       290227 KB
pswpin:                 8               8
pswpout:                430             510

There is no difference if we don't have cma reserved memory (0 MB case). But, with cma reserved memory (512 MB case), we fully utilize these reserved memory through this patch and the system behaves like as it doesn't reserve any memory.

With this patch, we aggressively allocate the pages on cma reserved memory so latency of CMA can arise. Below is the experimental result about latency.

4 CPUs, 1024 MB, VIRTUAL MACHINE
CMA reserve: 512 MB
Backgound Workload: make -jN
Real Workload: 8 MB CMA allocation/free 20 times with 5 sec interval

N:                    1        4       8        16
Elapsed-time(Before): 4309.75  9511.09 12276.1  77103.5
Elapsed-time(After):  5391.69 16114.1  19380.3  34879.2

So generally we can see latency increase. Ratio of this increase is rather big - up to 70%. But, under the heavy workload, it shows latency decrease - up to 55%. This may be worst-case scenario, but reducing it would be important for some system, so, I can say that this patch have advantages and disadvantages in terms of latency.

Signed-off-by Joonsoo Kim <[email protected]>

include/linux/mmzone.h +6 -0

index fac5509..3ff24d4 100644
@@ -389,6 +389,12 @@ struct zone {
 	int			compact_order_failed;
 #endif
 
+#ifdef CONFIG_CMA
+	int has_cma;
+	int nr_try_cma;
+	int nr_try_movable;
+#endif
+
 	ZONE_PADDING(_pad1_)
 
 	/* Fields commonly accessed by the page reclaim scanner */

mm/page_alloc.c +80 -2

index 674ade7..6f2b27b 100644
@@ -788,6 +788,16 @@ void __init __free_pages_bootmem(struct page *page, unsigned int order)
 }
 
 #ifdef CONFIG_CMA
+void __init init_alloc_ratio_counter(struct zone *zone)
+{
+	if (zone->has_cma)
+		return;
+
+	zone->has_cma = 1;
+	zone->nr_try_movable = 0;
+	zone->nr_try_cma = 0;
+}
+
 /* Free whole pageblock and set its migration type to MIGRATE_CMA. */
 void __init init_cma_reserved_pageblock(struct page *page)
 {

@@ -803,6 +813,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 	set_pageblock_migratetype(page, MIGRATE_CMA);
 	__free_pages(page, pageblock_order);
 	adjust_managed_page_count(page, pageblock_nr_pages);
+	init_alloc_ratio_counter(page_zone(page));
 }
 #endif
 

@@ -1136,6 +1147,69 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 	return NULL;
 }
 
+#ifdef CONFIG_CMA
+static struct page *__rmqueue_cma(struct zone *zone, unsigned int order,
+						int migratetype)
+{
+	long free, free_cma, free_wmark;
+	struct page *page;
+
+	if (migratetype != MIGRATE_MOVABLE || !zone->has_cma)
+		return NULL;
+
+	if (zone->nr_try_movable)
+		goto alloc_movable;
+
+alloc_cma:
+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}
+
+	/* Reset ratio counter */
+	free_cma = zone_page_state(zone, NR_FREE_CMA_PAGES);
+
+	/* No cma free pages, so recharge only movable allocation */
+	if (free_cma <= 0) {
+		zone->nr_try_movable = pageblock_nr_pages;
+		goto alloc_movable;
+	}
+
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	free_wmark = free - free_cma - high_wmark_pages(zone);
+
+	/*
+	 * free_wmark is below than 0, and it means that normal pages
+	 * are under the pressure, so we recharge only cma allocation.
+	 */
+	if (free_wmark <= 0) {
+		zone->nr_try_cma = pageblock_nr_pages;
+		goto alloc_cma;
+	}
+
+	if (free_wmark > free_cma) {
+		zone->nr_try_movable =
+			(free_wmark * pageblock_nr_pages) / free_cma;
+		zone->nr_try_cma = pageblock_nr_pages;
+	} else {
+		zone->nr_try_movable = pageblock_nr_pages;
+		zone->nr_try_cma = free_cma * pageblock_nr_pages / free_wmark;
+	}
+
+	/* Reset complete, start on movable first */
+alloc_movable:
+	zone->nr_try_movable--;
+	return NULL;
+}
+#endif
+
 /*
  * Do the hard work of removing an element from the buddy allocator.
  * Call me with the zone->lock already held.

@@ -1143,10 +1217,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 static struct page *__rmqueue(struct zone *zone, unsigned int order,
 						int migratetype)
 {
-	struct page *page;
+	struct page *page = NULL;
+
+	if (IS_ENABLED(CONFIG_CMA))
+		page = __rmqueue_cma(zone, order, migratetype);
 
 retry_reserve:
-	page = __rmqueue_smallest(zone, order, migratetype);
+	if (!page)
+		page = __rmqueue_smallest(zone, order, migratetype);
 
 	if (unlikely(!page) && migratetype != MIGRATE_RESERVE) {
 		page = __rmqueue_fallback(zone, order, migratetype);

Hi,

Quoted 48 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 8 lines

I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess.

We have an out of tree implementation of #1 and so far it's worked for us although we weren't looking at the same metrics. I don't completely understand the issue you pointed out with #1. It sounds like the issue is that CMA pages are already in use by other processes and on LRU lists and because the pages are on LRU lists these aren't counted towards the watermark by kswapd. Is my understanding correct?

Quoted 30 lines

So, I implement #2 approach.
One thing I should note is that we should not change allocation target (movable list or cma) on each allocation attempt, since this prevent allocated pages to be in physically succession, so some I/O devices can be hurt their performance. To solve this, I keep allocation target in at least pageblock_nr_pages attempts and make this number reflect ratio, free pages without free cma pages to free cma pages. With this approach, system works very smoothly and fully utilize the pages on cma reserved memory.

Following is the experimental result of this patch.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

<Before>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.8           361.8
Average-MemFree:        283880 KB       530851 KB
pswpin:                 7               110064
pswpout:                452             767502

<After>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.2           235.6
Average-MemFree:        281651 KB       290227 KB
pswpin:                 8               8
pswpout:                430             510

Quoted 4 lines

There is no difference if we don't have cma reserved memory (0 MB case). But, with cma reserved memory (512 MB case), we fully utilize these reserved memory through this patch and the system behaves like as it doesn't reserve any memory.

What metric are you using to determine all CMA memory was fully used? We've been checking /proc/pagetypeinfo

Quoted 14 lines

With this patch, we aggressively allocate the pages on cma reserved memory so latency of CMA can arise. Below is the experimental result about latency.

4 CPUs, 1024 MB, VIRTUAL MACHINE
CMA reserve: 512 MB
Backgound Workload: make -jN
Real Workload: 8 MB CMA allocation/free 20 times with 5 sec interval

N:                    1        4       8        16
Elapsed-time(Before): 4309.75  9511.09 12276.1  77103.5
Elapsed-time(After):  5391.69 16114.1  19380.3  34879.2

Quoted 6 lines

So generally we can see latency increase. Ratio of this increase is rather big - up to 70%. But, under the heavy workload, it shows latency decrease - up to 55%. This may be worst-case scenario, but reducing it would be important for some system, so, I can say that this patch have advantages and disadvantages in terms of latency.

Do you have any statistics related to failed migration from this? Latency and utilization are issues but so is migration success. In the past we've found that an increase in CMA utilization was related to increase in CMA migration failures because pages were unmigratable. The current workaround for this is limiting CMA pages to be used for user processes only and not the file cache. Both of these have their own problems.

Quoted 142 lines

Signed-off-by Joonsoo Kim <[email protected]>

include/linux/mmzone.h +6 -0

index fac5509..3ff24d4 100644
@@ -389,6 +389,12 @@ struct zone {
 	int			compact_order_failed;
 #endif
 
+#ifdef CONFIG_CMA
+	int has_cma;
+	int nr_try_cma;
+	int nr_try_movable;
+#endif
+
 	ZONE_PADDING(_pad1_)
 
 	/* Fields commonly accessed by the page reclaim scanner */

mm/page_alloc.c +82 -2

index 674ade7..6f2b27b 100644
@@ -788,6 +788,16 @@ void __init __free_pages_bootmem(struct page *page, unsigned int order)
 }
 
 #ifdef CONFIG_CMA
+void __init init_alloc_ratio_counter(struct zone *zone)
+{
+	if (zone->has_cma)
+		return;
+
+	zone->has_cma = 1;
+	zone->nr_try_movable = 0;
+	zone->nr_try_cma = 0;
+}
+
 /* Free whole pageblock and set its migration type to MIGRATE_CMA. */
 void __init init_cma_reserved_pageblock(struct page *page)
 {

@@ -803,6 +813,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 	set_pageblock_migratetype(page, MIGRATE_CMA);
 	__free_pages(page, pageblock_order);
 	adjust_managed_page_count(page, pageblock_nr_pages);
+	init_alloc_ratio_counter(page_zone(page));
 }
 #endif
 

@@ -1136,6 +1147,69 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 	return NULL;
 }
 
+#ifdef CONFIG_CMA
+static struct page *__rmqueue_cma(struct zone *zone, unsigned int order,
+						int migratetype)
+{
+	long free, free_cma, free_wmark;
+	struct page *page;
+
+	if (migratetype != MIGRATE_MOVABLE || !zone->has_cma)
+		return NULL;
+
+	if (zone->nr_try_movable)
+		goto alloc_movable;
+
+alloc_cma:
+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}
+
+	/* Reset ratio counter */
+	free_cma = zone_page_state(zone, NR_FREE_CMA_PAGES);
+
+	/* No cma free pages, so recharge only movable allocation */
+	if (free_cma <= 0) {
+		zone->nr_try_movable = pageblock_nr_pages;
+		goto alloc_movable;
+	}
+
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	free_wmark = free - free_cma - high_wmark_pages(zone);
+
+	/*
+	 * free_wmark is below than 0, and it means that normal pages
+	 * are under the pressure, so we recharge only cma allocation.
+	 */
+	if (free_wmark <= 0) {
+		zone->nr_try_cma = pageblock_nr_pages;
+		goto alloc_cma;
+	}
+
+	if (free_wmark > free_cma) {
+		zone->nr_try_movable =
+			(free_wmark * pageblock_nr_pages) / free_cma;
+		zone->nr_try_cma = pageblock_nr_pages;
+	} else {
+		zone->nr_try_movable = pageblock_nr_pages;
+		zone->nr_try_cma = free_cma * pageblock_nr_pages / free_wmark;
+	}
+
+	/* Reset complete, start on movable first */
+alloc_movable:
+	zone->nr_try_movable--;
+	return NULL;
+}
+#endif
+
 /*
  * Do the hard work of removing an element from the buddy allocator.
  * Call me with the zone->lock already held.

@@ -1143,10 +1217,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 static struct page *__rmqueue(struct zone *zone, unsigned int order,
 						int migratetype)
 {
-	struct page *page;
+	struct page *page = NULL;
+
+	if (IS_ENABLED(CONFIG_CMA))
+		page = __rmqueue_cma(zone, order, migratetype);
 
 retry_reserve:
-	page = __rmqueue_smallest(zone, order, migratetype);
+	if (!page)
+		page = __rmqueue_smallest(zone, order, migratetype);
 
 	if (unlikely(!page) && migratetype != MIGRATE_RESERVE) {
 		page = __rmqueue_fallback(zone, order, migratetype);

@@ -4849,6 +4927,8 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 		zone_seqlock_init(zone);
 		zone->zone_pgdat = pgdat;
 		zone_pcp_init(zone);
+		if (IS_ENABLED(CONFIG_CMA))
+			zone->has_cma = 0;
 

mm/page_alloc.c static void __paginginit free_area_init_core(struct pglist_data *pgdat, · line 4934

Quoted 3 lines

mm/page_alloc.c +0 -0

 		/* For bootup, initialized properly in watermark setup */
 		mod_zone_page_state(zone, NR_ALLOC_BATCH, zone->managed_pages);

I'm going to see about running this through tests internally for comparison. Hopefully I'll get useful results in a day or so.

Hi,

Quoted 48 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 8 lines

I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess.

We have an out of tree implementation of #1 and so far it's worked for us although we weren't looking at the same metrics. I don't completely understand the issue you pointed out with #1. It sounds like the issue is that CMA pages are already in use by other processes and on LRU lists and because the pages are on LRU lists these aren't counted towards the watermark by kswapd. Is my understanding correct?

Quoted 30 lines

So, I implement #2 approach.
One thing I should note is that we should not change allocation target (movable list or cma) on each allocation attempt, since this prevent allocated pages to be in physically succession, so some I/O devices can be hurt their performance. To solve this, I keep allocation target in at least pageblock_nr_pages attempts and make this number reflect ratio, free pages without free cma pages to free cma pages. With this approach, system works very smoothly and fully utilize the pages on cma reserved memory.

Following is the experimental result of this patch.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

<Before>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.8           361.8
Average-MemFree:        283880 KB       530851 KB
pswpin:                 7               110064
pswpout:                452             767502

<After>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.2           235.6
Average-MemFree:        281651 KB       290227 KB
pswpin:                 8               8
pswpout:                430             510

Quoted 4 lines

There is no difference if we don't have cma reserved memory (0 MB case). But, with cma reserved memory (512 MB case), we fully utilize these reserved memory through this patch and the system behaves like as it doesn't reserve any memory.

What metric are you using to determine all CMA memory was fully used? We've been checking /proc/pagetypeinfo

Quoted 14 lines

With this patch, we aggressively allocate the pages on cma reserved memory so latency of CMA can arise. Below is the experimental result about latency.

4 CPUs, 1024 MB, VIRTUAL MACHINE
CMA reserve: 512 MB
Backgound Workload: make -jN
Real Workload: 8 MB CMA allocation/free 20 times with 5 sec interval

N:                    1        4       8        16
Elapsed-time(Before): 4309.75  9511.09 12276.1  77103.5
Elapsed-time(After):  5391.69 16114.1  19380.3  34879.2

Quoted 6 lines

So generally we can see latency increase. Ratio of this increase is rather big - up to 70%. But, under the heavy workload, it shows latency decrease - up to 55%. This may be worst-case scenario, but reducing it would be important for some system, so, I can say that this patch have advantages and disadvantages in terms of latency.

Do you have any statistics related to failed migration from this? Latency and utilization are issues but so is migration success. In the past we've found that an increase in CMA utilization was related to increase in CMA migration failures because pages were unmigratable. The current workaround for this is limiting CMA pages to be used for user processes only and not the file cache. Both of these have their own problems.

Quoted 138 lines

Signed-off-by Joonsoo Kim <[email protected]>

include/linux/mmzone.h +6 -0

index fac5509..3ff24d4 100644
@@ -389,6 +389,12 @@ struct zone {
 	int			compact_order_failed;
 #endif
 
+#ifdef CONFIG_CMA
+	int has_cma;
+	int nr_try_cma;
+	int nr_try_movable;
+#endif
+
 	ZONE_PADDING(_pad1_)
 
 	/* Fields commonly accessed by the page reclaim scanner */

mm/page_alloc.c +80 -2

index 674ade7..6f2b27b 100644
@@ -788,6 +788,16 @@ void __init __free_pages_bootmem(struct page *page, unsigned int order)
 }
 
 #ifdef CONFIG_CMA
+void __init init_alloc_ratio_counter(struct zone *zone)
+{
+	if (zone->has_cma)
+		return;
+
+	zone->has_cma = 1;
+	zone->nr_try_movable = 0;
+	zone->nr_try_cma = 0;
+}
+
 /* Free whole pageblock and set its migration type to MIGRATE_CMA. */
 void __init init_cma_reserved_pageblock(struct page *page)
 {

@@ -803,6 +813,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 	set_pageblock_migratetype(page, MIGRATE_CMA);
 	__free_pages(page, pageblock_order);
 	adjust_managed_page_count(page, pageblock_nr_pages);
+	init_alloc_ratio_counter(page_zone(page));
 }
 #endif
 

@@ -1136,6 +1147,69 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 	return NULL;
 }
 
+#ifdef CONFIG_CMA
+static struct page *__rmqueue_cma(struct zone *zone, unsigned int order,
+						int migratetype)
+{
+	long free, free_cma, free_wmark;
+	struct page *page;
+
+	if (migratetype != MIGRATE_MOVABLE || !zone->has_cma)
+		return NULL;
+
+	if (zone->nr_try_movable)
+		goto alloc_movable;
+
+alloc_cma:
+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}
+
+	/* Reset ratio counter */
+	free_cma = zone_page_state(zone, NR_FREE_CMA_PAGES);
+
+	/* No cma free pages, so recharge only movable allocation */
+	if (free_cma <= 0) {
+		zone->nr_try_movable = pageblock_nr_pages;
+		goto alloc_movable;
+	}
+
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	free_wmark = free - free_cma - high_wmark_pages(zone);
+
+	/*
+	 * free_wmark is below than 0, and it means that normal pages
+	 * are under the pressure, so we recharge only cma allocation.
+	 */
+	if (free_wmark <= 0) {
+		zone->nr_try_cma = pageblock_nr_pages;
+		goto alloc_cma;
+	}
+
+	if (free_wmark > free_cma) {
+		zone->nr_try_movable =
+			(free_wmark * pageblock_nr_pages) / free_cma;
+		zone->nr_try_cma = pageblock_nr_pages;
+	} else {
+		zone->nr_try_movable = pageblock_nr_pages;
+		zone->nr_try_cma = free_cma * pageblock_nr_pages / free_wmark;
+	}
+
+	/* Reset complete, start on movable first */
+alloc_movable:
+	zone->nr_try_movable--;
+	return NULL;
+}
+#endif
+
 /*
  * Do the hard work of removing an element from the buddy allocator.
  * Call me with the zone->lock already held.

@@ -1143,10 +1217,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 static struct page *__rmqueue(struct zone *zone, unsigned int order,
 						int migratetype)
 {
-	struct page *page;
+	struct page *page = NULL;
+
+	if (IS_ENABLED(CONFIG_CMA))
+		page = __rmqueue_cma(zone, order, migratetype);
 
 retry_reserve:
-	page = __rmqueue_smallest(zone, order, migratetype);
+	if (!page)
+		page = __rmqueue_smallest(zone, order, migratetype);
 
 	if (unlikely(!page) && migratetype != MIGRATE_RESERVE) {
 		page = __rmqueue_fallback(zone, order, migratetype);

mm/page_alloc.c static void __paginginit free_area_init_core(struct pglist_data *pgdat, · line 4934

Quoted 8 lines

mm/page_alloc.c +2 -0

@@ -4849,6 +4927,8 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 		zone_seqlock_init(zone);
 		zone->zone_pgdat = pgdat;
 		zone_pcp_init(zone);
+		if (IS_ENABLED(CONFIG_CMA))
+			zone->has_cma = 0;
 
 		/* For bootup, initialized properly in watermark setup */
 		mod_zone_page_state(zone, NR_ALLOC_BATCH, zone->managed_pages);

I'm going to see about running this through tests internally for comparison. Hopefully I'll get useful results in a day or so.

Thanks,
Laura

Signature · 8 lines

Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm going to see about running this through tests internally for comparison. Hopefully I'll get useful results in a day or so.

Thanks,
Laura

Quoted 7 lines · Laura Abbott

I'm going to see about running this through tests internally for comparison. Hopefully I'll get useful results in a day or so.

Thanks,
Laura

We ran some tests internally and found that for our purposes these patches made the benchmarks worse vs. the existing implementation of using CMA first for some pages. These are mostly androidisms but androidisms that we care about for having a device be useful.

The foreground memory headroom on the device was on average about 40 MB smaller
when using these patches vs our existing implementation of something like
solution #1. By foreground memory headroom we simply mean the amount of memory that the foreground application can allocate before it is killed by the Android
Low Memory killer.

We also found that when running a sequence of app launches these patches had more high priority app kills by the LMK and more alloc stalls. The test did a total of 500 hundred app launches (using 9 separate applications) The CMA memory in our system is rarely used by its client and is therefore available to the system most of the time.

Test device
- 4 CPUs
- Android 4.4.2
- 512MB of RAM
- 68 MB of CMA

Results:

Existing solution:
Foreground headroom: 200MB
Number of higher priority LMK kills (oom_score_adj < 529): 332 Number of alloc stalls: 607

Test patches:
Foreground headroom: 160MB
Number of higher priority LMK kills (oom_score_adj < 529):
459 Number of alloc stalls: 29538

We believe that the issues seen with these patches are the result of the LMK being more aggressive. The LMK will be more aggressive because it will ignore free CMA pages for unmovable allocations, and since most calls to the LMK are made by kswapd (which uses GFP_KERNEL) the LMK will mostly ignore free CMA pages. Because the LMK thresholds are higher than the zone watermarks, there will often be a lot of free CMA pages in the system when the LMK is called, which the LMK will usually ignore.

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess.

So, I implement #2 approach.
One thing I should note is that we should not change allocation target (movable list or cma) on each allocation attempt, since this prevent allocated pages to be in physically succession, so some I/O devices can be hurt their performance. To solve this, I keep allocation target in at least pageblock_nr_pages attempts and make this number reflect ratio, free pages without free cma pages to free cma pages. With this approach, system works very smoothly and fully utilize the pages on cma reserved memory.

Following is the experimental result of this patch.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

<Before>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.8           361.8
Average-MemFree:        283880 KB       530851 KB
pswpin:                 7               110064
pswpout:                452             767502

<After>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.2           235.6
Average-MemFree:        281651 KB       290227 KB
pswpin:                 8               8
pswpout:                430             510

There is no difference if we don't have cma reserved memory (0 MB case). But, with cma reserved memory (512 MB case), we fully utilize these reserved memory through this patch and the system behaves like as it doesn't reserve any memory.

With this patch, we aggressively allocate the pages on cma reserved memory so latency of CMA can arise. Below is the experimental result about latency.

4 CPUs, 1024 MB, VIRTUAL MACHINE
CMA reserve: 512 MB
Backgound Workload: make -jN
Real Workload: 8 MB CMA allocation/free 20 times with 5 sec interval

N:                    1        4       8        16
Elapsed-time(Before): 4309.75  9511.09 12276.1  77103.5
Elapsed-time(After):  5391.69 16114.1  19380.3  34879.2

So generally we can see latency increase. Ratio of this increase is rather big - up to 70%. But, under the heavy workload, it shows latency decrease - up to 55%. This may be worst-case scenario, but reducing it would be important for some system, so, I can say that this patch have advantages and disadvantages in terms of latency.

Signed-off-by Joonsoo Kim <[email protected]>

include/linux/mmzone.h +6 -0

index fac5509..3ff24d4 100644
@@ -389,6 +389,12 @@ struct zone {
 	int			compact_order_failed;
 #endif
 
+#ifdef CONFIG_CMA
+	int has_cma;
+	int nr_try_cma;
+	int nr_try_movable;
+#endif
+
 	ZONE_PADDING(_pad1_)
 
 	/* Fields commonly accessed by the page reclaim scanner */

mm/page_alloc.c +82 -2

index 674ade7..6f2b27b 100644
@@ -788,6 +788,16 @@ void __init __free_pages_bootmem(struct page *page, unsigned int order)
 }
 
 #ifdef CONFIG_CMA
+void __init init_alloc_ratio_counter(struct zone *zone)
+{
+	if (zone->has_cma)
+		return;
+
+	zone->has_cma = 1;
+	zone->nr_try_movable = 0;
+	zone->nr_try_cma = 0;
+}
+
 /* Free whole pageblock and set its migration type to MIGRATE_CMA. */
 void __init init_cma_reserved_pageblock(struct page *page)
 {

@@ -803,6 +813,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 	set_pageblock_migratetype(page, MIGRATE_CMA);
 	__free_pages(page, pageblock_order);
 	adjust_managed_page_count(page, pageblock_nr_pages);
+	init_alloc_ratio_counter(page_zone(page));
 }
 #endif
 

@@ -1136,6 +1147,69 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 	return NULL;
 }
 
+#ifdef CONFIG_CMA
+static struct page *__rmqueue_cma(struct zone *zone, unsigned int order,
+						int migratetype)
+{
+	long free, free_cma, free_wmark;
+	struct page *page;
+
+	if (migratetype != MIGRATE_MOVABLE || !zone->has_cma)
+		return NULL;
+
+	if (zone->nr_try_movable)
+		goto alloc_movable;
+
+alloc_cma:
+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}
+
+	/* Reset ratio counter */
+	free_cma = zone_page_state(zone, NR_FREE_CMA_PAGES);
+
+	/* No cma free pages, so recharge only movable allocation */
+	if (free_cma <= 0) {
+		zone->nr_try_movable = pageblock_nr_pages;
+		goto alloc_movable;
+	}
+
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	free_wmark = free - free_cma - high_wmark_pages(zone);
+
+	/*
+	 * free_wmark is below than 0, and it means that normal pages
+	 * are under the pressure, so we recharge only cma allocation.
+	 */
+	if (free_wmark <= 0) {
+		zone->nr_try_cma = pageblock_nr_pages;
+		goto alloc_cma;
+	}
+
+	if (free_wmark > free_cma) {
+		zone->nr_try_movable =
+			(free_wmark * pageblock_nr_pages) / free_cma;
+		zone->nr_try_cma = pageblock_nr_pages;
+	} else {
+		zone->nr_try_movable = pageblock_nr_pages;
+		zone->nr_try_cma = free_cma * pageblock_nr_pages / free_wmark;
+	}
+
+	/* Reset complete, start on movable first */
+alloc_movable:
+	zone->nr_try_movable--;
+	return NULL;
+}
+#endif
+
 /*
  * Do the hard work of removing an element from the buddy allocator.
  * Call me with the zone->lock already held.

@@ -1143,10 +1217,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 static struct page *__rmqueue(struct zone *zone, unsigned int order,
 						int migratetype)
 {
-	struct page *page;
+	struct page *page = NULL;
+
+	if (IS_ENABLED(CONFIG_CMA))
+		page = __rmqueue_cma(zone, order, migratetype);
 
 retry_reserve:
-	page = __rmqueue_smallest(zone, order, migratetype);
+	if (!page)
+		page = __rmqueue_smallest(zone, order, migratetype);
 
 	if (unlikely(!page) && migratetype != MIGRATE_RESERVE) {
 		page = __rmqueue_fallback(zone, order, migratetype);

@@ -4849,6 +4927,8 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 		zone_seqlock_init(zone);
 		zone->zone_pgdat = pgdat;
 		zone_pcp_init(zone);
+		if (IS_ENABLED(CONFIG_CMA))
+			zone->has_cma = 0;
 
 		/* For bootup, initialized properly in watermark setup */
 		mod_zone_page_state(zone, NR_ALLOC_BATCH, zone->managed_pages);

Signature · 6 lines

1.7.9.5

_______________________________________________
OTC mailing list
[email protected]
http://blackduck.lge.com/mailman/listinfo/otc

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

And we couldn't do it in zone_watermark_ok with set/reset ALLOC_CMA? If possible, it would be better becauser it's generic function to check free pages and cause trigger reclaim/compaction logic.

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

This is just my quick thought, so please let me correct if I am wrong.

Quoted 5 lines

And we couldn't do it in zone_watermark_ok with set/reset ALLOC_CMA? If possible, it would be better becauser it's generic function to check free pages and cause trigger reclaim/compaction logic.

I guess, your *it* means ratio computation. Right?

I don't like putting it on zone_watermark_ok(). Although it need to refer to free cma pages value which are also referred in zone_watermark_ok(), this computation is for determining ratio, not for triggering reclaim/compaction. And this zone_watermark_ok() is on more hot-path, so putting this logic into zone_watermark_ok() looks not better to me.

I will think better place to do it.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 60 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Quoted 6 lines

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

Sounds good.

Quoted 12 lines

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

Why should reclaimer reclaim unnecessary pages?
So, your answer is that it would be better because upcoming newly allocated pages would be allocated easily without interrupt. But it could reclaim too much pages until watermark for unmovable allocation is okay. Even, sometime, you might see OOM.

Moreover, how could you handle current trobule?
For example, there is atomic allocation and the only thing to save the world is kswapd because it's one of kswapd role but kswapd is spending many time to reclaim CMA pages, which is pointless so the allocation would be easily failed.

Quoted 8 lines

Quoted 6 lines

And we couldn't do it in zone_watermark_ok with set/reset ALLOC_CMA? If possible, it would be better becauser it's generic function to check free pages and cause trigger reclaim/compaction logic.

I guess, your *it* means ratio computation. Right?

I meant just get_page_from_freelist like fair zone allocation for consistency but as we discussed offline, i'm not against with you if it's not right place.

Quoted 90 lines · Minchan Kim

Quoted 60 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Quoted 6 lines

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

Sounds good.

Quoted 12 lines

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

Why should reclaimer reclaim unnecessary pages?
So, your answer is that it would be better because upcoming newly allocated pages would be allocated easily without interrupt. But it could reclaim too much pages until watermark for unmovable allocation is okay. Even, sometime, you might see OOM.

Quoted 4 lines

Moreover, how could you handle current trobule?
For example, there is atomic allocation and the only thing to save the world is kswapd because it's one of kswapd role but kswapd is spending many time to reclaim CMA pages, which is pointless so the allocation would be easily failed.

Hello,

I guess that it isn't the problem. In lru, movable pages and cma pages would be interleaved. So it doesn't takes too long time to get the page for non-movable allocation.

IMHO, in generally, memory shortage is made by movable allocation, so to distinguish allocation type and to handle them differently has marginal effect.

Anyway, I will think more deeply.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

And we couldn't do it in zone_watermark_ok with set/reset ALLOC_CMA? If possible, it would be better becauser it's generic function to check free pages and cause trigger reclaim/compaction logic.

I guess, your *it* means ratio computation. Right?

I meant just get_page_from_freelist like fair zone allocation for consistency but as we discussed offline, i'm not against with you if it's not right place.

Okay :)

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 96 lines · Joonsoo Kim

Quoted 90 lines · Minchan Kim

Quoted 60 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Quoted 6 lines

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

Sounds good.

Quoted 12 lines

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

Why should reclaimer reclaim unnecessary pages?
So, your answer is that it would be better because upcoming newly allocated pages would be allocated easily without interrupt. But it could reclaim too much pages until watermark for unmovable allocation is okay. Even, sometime, you might see OOM.

Quoted 4 lines

Moreover, how could you handle current trobule?
For example, there is atomic allocation and the only thing to save the world is kswapd because it's one of kswapd role but kswapd is spending many time to reclaim CMA pages, which is pointless so the allocation would be easily failed.

Quoted 5 lines

Hello,

I guess that it isn't the problem. In lru, movable pages and cma pages would be interleaved. So it doesn't takes too long time to get the page for non-movable allocation.

Please, don't assume there are ideal LRU ordering.
Newly allocated page by fairness allocation is located by head of LRU while old pages are approaching the tail so there is huge time gab. During the time, old pages could be dropped/promoting so one of side could be filled with one type rather than interleaving both types pages you expected.

Additionally, if you uses syncable backed device like ramdisk/zram or something, pageout can be synchronized with page I/O. In this case, reclaim time wouldn't be trivial than async I/O. For exmaple, zram-swap case, it needs page copy + comperssion and the speed depends on your CPU speed.

Quoted 103 lines · Minchan Kim

Quoted 96 lines · Joonsoo Kim

Quoted 90 lines · Minchan Kim

Quoted 60 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Quoted 6 lines

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

Sounds good.

Quoted 12 lines

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

Why should reclaimer reclaim unnecessary pages?
So, your answer is that it would be better because upcoming newly allocated pages would be allocated easily without interrupt. But it could reclaim too much pages until watermark for unmovable allocation is okay. Even, sometime, you might see OOM.

Quoted 4 lines

Moreover, how could you handle current trobule?
For example, there is atomic allocation and the only thing to save the world is kswapd because it's one of kswapd role but kswapd is spending many time to reclaim CMA pages, which is pointless so the allocation would be easily failed.

Quoted 5 lines

Hello,

I guess that it isn't the problem. In lru, movable pages and cma pages would be interleaved. So it doesn't takes too long time to get the page for non-movable allocation.

Quoted 6 lines

Please, don't assume there are ideal LRU ordering.
Newly allocated page by fairness allocation is located by head of LRU while old pages are approaching the tail so there is huge time gab. During the time, old pages could be dropped/promoting so one of side could be filled with one type rather than interleaving both types pages you expected.

I assumed general case, not ideal case.
Your example can be possible, but would be corner case.

Quoted 7 lines

Additionally, if you uses syncable backed device like ramdisk/zram or something, pageout can be synchronized with page I/O. In this case, reclaim time wouldn't be trivial than async I/O. For exmaple, zram-swap case, it needs page copy + comperssion and the speed depends on your CPU speed.

This is a general problem what zram-swap have, although reclaiming cma pages worse the situation.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 111 lines · Joonsoo Kim

Quoted 103 lines · Minchan Kim

Quoted 96 lines · Joonsoo Kim

Quoted 90 lines · Minchan Kim

Quoted 60 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Quoted 6 lines

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

Sounds good.

Quoted 12 lines

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

Why should reclaimer reclaim unnecessary pages?
So, your answer is that it would be better because upcoming newly allocated pages would be allocated easily without interrupt. But it could reclaim too much pages until watermark for unmovable allocation is okay. Even, sometime, you might see OOM.

Quoted 4 lines

Moreover, how could you handle current trobule?
For example, there is atomic allocation and the only thing to save the world is kswapd because it's one of kswapd role but kswapd is spending many time to reclaim CMA pages, which is pointless so the allocation would be easily failed.

Quoted 5 lines

Hello,

I guess that it isn't the problem. In lru, movable pages and cma pages would be interleaved. So it doesn't takes too long time to get the page for non-movable allocation.

Quoted 6 lines

Please, don't assume there are ideal LRU ordering.
Newly allocated page by fairness allocation is located by head of LRU while old pages are approaching the tail so there is huge time gab. During the time, old pages could be dropped/promoting so one of side could be filled with one type rather than interleaving both types pages you expected.

Quoted 2 lines

I assumed general case, not ideal case.
Your example can be possible, but would be corner case.

I talked with Joonsoo yesterday and should post our conclusion for other reviewers/maintainers.

It's not a corner case and it could happen depending on zone and CMA configuration. For example, there is 330M high zone and CMA consumes 300M in the space while normal movable area consumes just 30M. In the case, unmovable allocation could make too many unnecessary reclaiming of the zone so the conclusion we reached is to need target reclaiming(ex, isolate_mode_t).

But not sure it should be part of this patchset because this patchset is surely enhance(ie, before, it was hard to allocate page from CMA area but this patchset makes it works) but this patchset could make mentioned problem as side-effect so I think we could solve the issue(ie, too many reclaiming in unbalanced zone) in another patchset.

Joonsoo, please mention this problem in the description when you respin so other MM guys can notice that and give ideas, which would be helpful a lot.

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

This is just my quick thought, so please let me correct if I am wrong.

Quoted 5 lines

And we couldn't do it in zone_watermark_ok with set/reset ALLOC_CMA? If possible, it would be better becauser it's generic function to check free pages and cause trigger reclaim/compaction logic.

I guess, your *it* means ratio computation. Right?
I don't like putting it on zone_watermark_ok(). Although it need to refer to free cma pages value which are also referred in zone_watermark_ok(), this computation is for determining ratio, not for triggering reclaim/compaction. And this zone_watermark_ok() is on more hot-path, so putting this logic into zone_watermark_ok() looks not better to me.

I will think better place to do it.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hello,

Quoted 78 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

We have an out of tree implementation that is completely the same with the approach Minchan said and it works, but it has definitely some side-effects as you pointed, distorting the LRU and evicting hot pages. I do not attach code fragments in this thread for some

reasons, but it must be easy for yourself. I am wondering if it could help also in your case.

Thanks,
Heesub

Quoted 15 lines

Quoted 5 lines

And we couldn't do it in zone_watermark_ok with set/reset ALLOC_CMA? If possible, it would be better becauser it's generic function to check free pages and cause trigger reclaim/compaction logic.

I guess, your *it* means ratio computation. Right?
I don't like putting it on zone_watermark_ok(). Although it need to refer to free cma pages value which are also referred in zone_watermark_ok(), this computation is for determining ratio, not for triggering reclaim/compaction. And this zone_watermark_ok() is on more hot-path, so putting this logic into zone_watermark_ok() looks not better to me.

I will think better place to do it.

Quoted 8 lines

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hello,

Quoted 78 lines · Joonsoo Kim

Quoted 51 lines · Minchan Kim

Hey Joonsoo,

Quoted 42 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

Quoted 5 lines

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

Quoted 7 lines

I love this idea but when I see the code, I don't like that.
In allocation path, just try to allocate pages by round-robin so it's role of allocator. If one of migratetype is full, just pass mission to reclaimer with hint(ie, Hey reclaimer, it's non-movable allocation fail so there is pointless if you reclaim MIGRATE_CMA pages) so that reclaimer can filter it out during page scanning.
We already have an tool to achieve it(ie, isolate_mode_t).

Hello,

I agree with leaving fast allocation path as simple as possible. I will remove runtime computation for determining ratio in __rmqueue_cma() and, instead, will use pre-computed value calculated on the other path.

I am not sure that whether your second suggestion(Hey relaimer part) is good or not. In my quick thought, that could be helpful in the situation that many free cma pages remained. But, it would be not helpful when there are neither free movable and cma pages. In generally, most workloads mainly uses movable pages for page cache or anonymous mapping. Although reclaim is triggered by non-movable allocation failure, reclaimed pages are used mostly by movable allocation. We can handle these allocation request even if we reclaim the pages just in lru order. If we rotate the lru list for finding movable pages, it could cause more useful pages to be evicted.

Quoted 1 line

This is just my quick thought, so please let me correct if I am wrong.

We have an out of tree implementation that is completely the same with the approach Minchan said and it works, but it has definitely some side-effects as you pointed, distorting the LRU and evicting hot pages. I do not attach code fragments in this thread for some reasons, but it must be easy for yourself. I am wondering if it could help also in your case.

Thanks,
Heesub

If CMA_SIZE_MBYTES is allowed to be zero, there should be defense code to check CMA is initlaized correctly. And atomic_pool initialization should be done by __alloc_remap_buffer instead of __alloc_from_contiguous if __alloc_from_contiguous is failed.

IMPO, it is more simple and powerful to restrict CMA_SIZE_MBYTES_MAX configuration to be larger than zero.

Quoted 4 lines

If CMA_SIZE_MBYTES is allowed to be zero, there should be defense code to check CMA is initlaized correctly. And atomic_pool initialization should be done by __alloc_remap_buffer instead of __alloc_from_contiguous if __alloc_from_contiguous is failed.

Agreed, and this is the correct fix.

Quoted 2 lines

IMPO, it is more simple and powerful to restrict CMA_SIZE_MBYTES_MAX configuration to be larger than zero.

No, because it makes it impossible to have CMA disabled by default and only enabled if command line argument is given.

Furthermore, your patch does *not* guarantee CMA region to always be allocated. If CMA_SIZE_SEL_PERCENTAGE is selected for instance. Or if user explicitly passes 0 on command line.

Thank you for your advice. I didn't notice it.

I'm adding followings according to your advice:

- range restrict for CMA_SIZE_MBYTES and *CMA_SIZE_PERCENTAGE* I think this can prevent the wrong kernel option.

- change size_cmdline into default value SZ_16M
I am not sure this can prevent if cma=0 cmdline option is also with base and limit options.

Quoted 9 lines

Thank you for your advice. I didn't notice it.

I'm adding followings according to your advice:

- range restrict for CMA_SIZE_MBYTES and *CMA_SIZE_PERCENTAGE* I think this can prevent the wrong kernel option.

- change size_cmdline into default value SZ_16M
I am not sure this can prevent if cma=0 cmdline option is also with base and limit options.

Hello,

I think that this problem is originated from atomic_pool_init(). If configured coherent_pool size is larger than default cma size, it can be failed even if this patch is applied.

How about below patch?
It uses fallback allocation if CMA is failed.

Thanks.

-----------------8<---------------------

arch/arm/mm/dma-mapping.c +1 -1

index 6b00be1..2909ab9 100644
@@ -379,7 +379,7 @@ static int __init atomic_pool_init(void)
         unsigned long *bitmap;
         struct page *page;
         struct page **pages;
-       void *ptr;
+       void *ptr = NULL;
         int bitmap_size = BITS_TO_LONGS(nr_pages) * sizeof(long);
 
         bitmap = kzalloc(bitmap_size, GFP_KERNEL);

I think that this problem is originated from atomic_pool_init(). If configured coherent_pool size is larger than default cma size, it can be failed even if this patch is applied.

How about below patch?
It uses fallback allocation if CMA is failed.

-----------------8<---------------------

arch/arm/mm/dma-mapping.c +2 -2

index 6b00be1..2909ab9 100644
@@ -379,7 +379,7 @@ static int __init atomic_pool_init(void)
        unsigned long *bitmap;
        struct page *page;
        struct page **pages;
-       void *ptr;
+       void *ptr = NULL;
        int bitmap_size = BITS_TO_LONGS(nr_pages) * sizeof(long);
 
        bitmap = kzalloc(bitmap_size, GFP_KERNEL);

@@ -393,7 +393,7 @@ static int __init atomic_pool_init(void)
        if (IS_ENABLED(CONFIG_DMA_CMA))
                ptr = __alloc_from_contiguous(NULL, pool->size, prot, &page,
                                              atomic_pool_init);
-       else
+       if (!ptr)
                ptr = __alloc_remap_buffer(NULL, pool->size, gfp, prot, &page,
                                           atomic_pool_init);
        if (ptr) {

Quoted 3 lines

I think that this problem is originated from atomic_pool_init(). If configured coherent_pool size is larger than default cma size, it can be failed even if this patch is applied.

Quoted 3 lines

How about below patch?
It uses fallback allocation if CMA is failed.

Yes, I thought about it, but __dma_alloc uses similar code:

	else if (!IS_ENABLED(CONFIG_DMA_CMA))
		addr = __alloc_remap_buffer(dev, size, gfp, prot, &page, caller);
	else
		addr = __alloc_from_contiguous(dev, size, prot, &page, caller);

so it probably needs to be changed as well.

Quoted 14 lines

-----------------8<---------------------

arch/arm/mm/dma-mapping.c +1 -1

index 6b00be1..2909ab9 100644
@@ -379,7 +379,7 @@ static int __init atomic_pool_init(void)
         unsigned long *bitmap;
         struct page *page;
         struct page **pages;
-       void *ptr;
+       void *ptr = NULL;
         int bitmap_size = BITS_TO_LONGS(nr_pages) * sizeof(long);
 

arch/arm/mm/dma-mapping.c static int __init atomic_pool_init(void) · line 399

Quoted 11 lines

arch/arm/mm/dma-mapping.c +1 -1

         bitmap = kzalloc(bitmap_size, GFP_KERNEL);

@@ -393,7 +393,7 @@ static int __init atomic_pool_init(void)
         if (IS_ENABLED(CONFIG_DMA_CMA))
                 ptr = __alloc_from_contiguous(NULL, pool->size, prot, &page,
                                               atomic_pool_init);
-       else
+       if (!ptr)
                 ptr = __alloc_remap_buffer(NULL, pool->size, gfp, prot, &page,
                                            atomic_pool_init);
         if (ptr) {

If CMA option is not selected, __alloc_from_contiguous would not be called. We don't need to the fallback allocation.

And if CMA option is selected and initialized correctly, the cma allocation can fail in case of no-CMA-memory situation. I thinks in that case we don't need to the fallback allocation also, because it is normal case.

Therefore I think the restriction of CMA size option and make CMA work can cover every cases.

Quoted 8 lines

If CMA option is not selected, __alloc_from_contiguous would not be called. We don't need to the fallback allocation.

And if CMA option is selected and initialized correctly, the cma allocation can fail in case of no-CMA-memory situation. I thinks in that case we don't need to the fallback allocation also, because it is normal case.

Quoted 2 lines

Therefore I think the restriction of CMA size option and make CMA work can cover every cases.

Wait, you just wrote that if CMA is not initialised correctly, it's fine for atomic pool initialisation to fail, but if CMA size is initialised correctly but too small, this is somehow worse situation? I'm a bit confused to be honest.

IMO, cma=0 command line argument should be supported, as should having the default CMA size zero. If CMA size is set to zero, kernel should behave as if CMA was not enabled at compile time.

My point is atomic_pool should be able to work with/without CMA.

Quoted 3 lines

IMO, cma=0 command line argument should be supported, as should having the default CMA size zero. If CMA size is set to zero, kernel should behave as if CMA was not enabled at compile time.

It's also good if atomic_pool can work well with zero CMA size.

2014-05-20 i??i ? 4:59, Michal Nazarewicz i?' e,?:

Quoted 1 line · Joonsoo Kim

Quoted 3 lines

Quoted 3 lines

I think that this problem is originated from atomic_pool_init(). If configured coherent_pool size is larger than default cma size, it can be failed even if this patch is applied.

The coherent_pool size (atomic_pool.size) should be restricted smaller than cma size.

This is another issue, however I think the default atomic pool size is too small. Only one port of USB host needs at most 256Kbytes coherent memory (according to the USB host spec).

If a platform has several ports, it needs more than 1MB. Therefore the default atomic pool size should be at least 1MB.

Quoted 6 lines

Quoted 3 lines

How about below patch?
It uses fallback allocation if CMA is failed.

Yes, I thought about it, but __dma_alloc uses similar code:

Quoted 7 lines

else if (!IS_ENABLED(CONFIG_DMA_CMA))
addr = __alloc_remap_buffer(dev, size, gfp, prot, &page,

caller);
    else
        addr = __alloc_from_contiguous(dev, size, prot, &page, caller);

so it probably needs to be changed as well.

If CMA option is not selected, __alloc_from_contiguous would not be called. We don't need to the fallback allocation.

And if CMA option is selected and initialized correctly, the cma allocation can fail in case of no-CMA-memory situation. I thinks in that case we don't need to the fallback allocation also, because it is normal case.

Therefore I think the restriction of CMA size option and make CMA work can cover every cases.

I think below patch is also good choice.
If both of you, Michal and Joonsoo, do not agree with me, please inform me. I will make a patch including option restriction and fallback allocation.

Hello,

Quoted 11 lines · Gioh Kim

2014-05-20 오전 4:59, Michal Nazarewicz 쓴 글:

Quoted 7 lines · Joonsoo Kim

Quoted 5 lines

I think that this problem is originated from atomic_pool_init(). If configured coherent_pool size is larger than default cma size, it can be failed even if this patch is applied.

Quoted 8 lines

The coherent_pool size (atomic_pool.size) should be restricted smaller than cma size.

This is another issue, however I think the default atomic pool size is too small.
Only one port of USB host needs at most 256Kbytes coherent memory (according to the USB host spec).

This pool is used only for allocation done in atomic context (allocations done with GFP_ATOMIC flag), otherwise the standard allocation path is used. Are you sure that each usb host port really needs so much memory allocated in atomic context?

If a platform has several ports, it needs more than 1MB.

Quoted 26 lines

Therefore the default atomic pool size should be at least 1MB.

Quoted 4 lines

Quoted 3 lines

How about below patch?
It uses fallback allocation if CMA is failed.

Quoted 9 lines

Yes, I thought about it, but __dma_alloc uses similar code:

    else if (!IS_ENABLED(CONFIG_DMA_CMA))
        addr = __alloc_remap_buffer(dev, size, gfp, prot, &page, caller);
    else
        addr = __alloc_from_contiguous(dev, size, prot, &page, caller);

so it probably needs to be changed as well.

If CMA option is not selected, __alloc_from_contiguous would not be called.
We don't need to the fallback allocation.

And if CMA option is selected and initialized correctly, the cma allocation can fail in case of no-CMA-memory situation. I thinks in that case we don't need to the fallback allocation also, because it is normal case.

Quoted 8 lines

Therefore I think the restriction of CMA size option and make CMA work can cover every cases.

I think below patch is also good choice.
If both of you, Michal and Joonsoo, do not agree with me, please inform me.
I will make a patch including option restriction and fallback allocation.

I'm not sure if we need a fallback for failed CMA allocation. The only issue that
have been mentioned here and needs to be resolved is support for disabling cma by
kernel command line. Right now it will fails completely.

Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Hello,

Quoted 10 lines · Gioh Kim

2014-05-20 i??i ? 4:59, Michal Nazarewicz i?' e,?:

Quoted 4 lines · Joonsoo Kim

Quoted 3 lines

I think that this problem is originated from atomic_pool_init(). If configured coherent_pool size is larger than default cma size, it can be failed even if this patch is applied.

The coherent_pool size (atomic_pool.size) should be restricted smaller than cma size.

Quoted 2 lines

This is another issue, however I think the default atomic pool size is too small. Only one port of USB host needs at most 256Kbytes coherent memory (according to the USB host spec).

This pool is used only for allocation done in atomic context (allocations done with GFP_ATOMIC flag), otherwise the standard allocation path is used. Are you sure that each usb host port really needs so much memory allocated in atomic context?

Quoted 27 lines

If a platform has several ports, it needs more than 1MB. Therefore the default atomic pool size should be at least 1MB.

Quoted 4 lines

Quoted 3 lines

How about below patch?
It uses fallback allocation if CMA is failed.

Quoted 8 lines

Yes, I thought about it, but __dma_alloc uses similar code:

    else if (!IS_ENABLED(CONFIG_DMA_CMA))
        addr = __alloc_remap_buffer(dev, size, gfp, prot, &page, caller);
    else
        addr = __alloc_from_contiguous(dev, size, prot, &page, caller);

so it probably needs to be changed as well.

If CMA option is not selected, __alloc_from_contiguous would not be called. We don't need to the fallback allocation.

And if CMA option is selected and initialized correctly, the cma allocation can fail in case of no-CMA-memory situation. I thinks in that case we don't need to the fallback allocation also, because it is normal case.

Therefore I think the restriction of CMA size option and make CMA work can cover every cases.

Quoted 3 lines

I think below patch is also good choice.
If both of you, Michal and Joonsoo, do not agree with me, please inform me. I will make a patch including option restriction and fallback allocation.

I'm not sure if we need a fallback for failed CMA allocation. The only issue that have been mentioned here and needs to be resolved is support for disabling cma by kernel command line. Right now it will fails completely.

Best regards

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free

memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess.

So, I implement #2 approach.
One thing I should note is that we should not change allocation target (movable list or cma) on each allocation attempt, since this prevent allocated pages to be in physically succession, so some I/O devices can be hurt their performance. To solve this, I keep allocation target in at least pageblock_nr_pages attempts and make this number reflect ratio, free pages without free cma pages to free cma pages. With this approach, system works very smoothly and fully utilize the pages on cma reserved memory.

Following is the experimental result of this patch.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

<Before>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.8           361.8
Average-MemFree:        283880 KB       530851 KB
pswpin:                 7               110064
pswpout:                452             767502

<After>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.2           235.6
Average-MemFree:        281651 KB       290227 KB
pswpin:                 8               8
pswpout:                430             510

There is no difference if we don't have cma reserved memory (0 MB case). But, with cma reserved memory (512 MB case), we fully utilize these reserved memory through this patch and the system behaves like as it doesn't reserve any memory.

With this patch, we aggressively allocate the pages on cma reserved memory so latency of CMA can arise. Below is the experimental result about latency.

4 CPUs, 1024 MB, VIRTUAL MACHINE
CMA reserve: 512 MB
Backgound Workload: make -jN
Real Workload: 8 MB CMA allocation/free 20 times with 5 sec interval

N:                    1        4       8        16
Elapsed-time(Before): 4309.75  9511.09 12276.1  77103.5
Elapsed-time(After):  5391.69 16114.1  19380.3  34879.2

So generally we can see latency increase. Ratio of this increase is rather big - up to 70%. But, under the heavy workload, it shows latency decrease - up to 55%. This may be worst-case scenario, but reducing it would be important for some system, so, I can say that this patch have advantages and disadvantages in terms of latency.

Signed-off-by Joonsoo Kim <[email protected]>

include/linux/mmzone.h +0 -0

index fac5509..3ff24d4 100644

include/linux/mmzone.h +1 -0

+
 	ZONE_PADDING(_pad1_)
 
 	/* Fields commonly accessed by the page reclaim scanner */

mm/page_alloc.c +56 -0

index 674ade7..6f2b27b 100644
@@ -788,6 +788,16 @@ void __init __free_pages_bootmem(struct page *page, unsigned int order)
 }
 
 #ifdef CONFIG_CMA
+void __init init_alloc_ratio_counter(struct zone *zone)
+{
+	if (zone->has_cma)
+		return;
+
+	zone->has_cma = 1;
+	zone->nr_try_movable = 0;
+	zone->nr_try_cma = 0;
+}
+
 /* Free whole pageblock and set its migration type to MIGRATE_CMA. */
 void __init init_cma_reserved_pageblock(struct page *page)
 {

@@ -803,6 +813,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 	set_pageblock_migratetype(page, MIGRATE_CMA);
 	__free_pages(page, pageblock_order);
 	adjust_managed_page_count(page, pageblock_nr_pages);
+	init_alloc_ratio_counter(page_zone(page));
 }
 #endif
 

@@ -1136,6 +1147,69 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 	return NULL;
 }
 
+#ifdef CONFIG_CMA
+static struct page *__rmqueue_cma(struct zone *zone, unsigned int order,
+						int migratetype)
+{
+	long free, free_cma, free_wmark;
+	struct page *page;
+
+	if (migratetype != MIGRATE_MOVABLE || !zone->has_cma)
+		return NULL;
+
+	if (zone->nr_try_movable)
+		goto alloc_movable;
+
+alloc_cma:
+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}
+
+	/* Reset ratio counter */
+	free_cma = zone_page_state(zone, NR_FREE_CMA_PAGES);
+
+	/* No cma free pages, so recharge only movable allocation */
+	if (free_cma <= 0) {
+		zone->nr_try_movable = pageblock_nr_pages;
+		goto alloc_movable;
+	}
+
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	free_wmark = free - free_cma - high_wmark_pages(zone);
+
+	/*
+	 * free_wmark is below than 0, and it means that normal pages
+	 * are under the pressure, so we recharge only cma allocation.
+	 */
+	if (free_wmark <= 0) {
+		zone->nr_try_cma = pageblock_nr_pages;
+		goto alloc_cma;

mm/page_alloc.c +8 -0

+
+	/* Reset complete, start on movable first */
+alloc_movable:
+	zone->nr_try_movable--;
+	return NULL;
+}
+#endif
+
 /*
  * Do the hard work of removing an element from the buddy allocator.
  * Call me with the zone->lock already held.

mm/page_alloc.c +4 -1

 retry_reserve:
-	page = __rmqueue_smallest(zone, order, migratetype);
+	if (!page)
+		page = __rmqueue_smallest(zone, order, migratetype);
 
 	if (unlikely(!page) && migratetype != MIGRATE_RESERVE) {
 		page = __rmqueue_fallback(zone, order, migratetype);

@@ -4849,6 +4927,8 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 		zone_seqlock_init(zone);
 		zone->zone_pgdat = pgdat;
 		zone_pcp_init(zone);
+		if (IS_ENABLED(CONFIG_CMA))
+			zone->has_cma = 0;
 
 		/* For bootup, initialized properly in watermark setup */
 		mod_zone_page_state(zone, NR_ALLOC_BATCH, zone->managed_pages);

Signature · 7 lines

1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 24 lines · Joonsoo Kim

CMA is introduced to provide physically contiguous pages at runtime. For this purpose, it reserves memory at boot time. Although it reserve memory, this reserved memory can be used for movable memory allocation request. This usecase is beneficial to the system that needs this CMA reserved memory infrequently and it is one of main purpose of introducing CMA.

But, there is a problem in current implementation. The problem is that it works like as just reserved memory approach. The pages on cma reserved memory are hardly used for movable memory allocation. This is caused by combination of allocation and reclaim policy.

The pages on cma reserved memory are allocated if there is no movable memory, that is, as fallback allocation. So the time this fallback allocation is started is under heavy memory pressure. Although it is under memory pressure, movable allocation easily succeed, since there would be many pages on cma reserved memory. But this is not the case for unmovable and reclaimable allocation, because they can't use the pages on cma reserved memory. These allocations regard system's free memory as (free pages - free cma pages) on watermark checking, that is, free unmovable pages + free reclaimable pages + free movable pages. Because we already exhausted movable pages, only free pages we have are unmovable and reclaimable types and this would be really small amount. So watermark checking would be failed. It will wake up kswapd to make enough free

Quoted 8 lines

memory for unmovable and reclaimable allocation and kswapd will do. So before we fully utilize pages on cma reserved memory, kswapd start to reclaim memory and try to make free memory over the high watermark. This watermark checking by kswapd doesn't take care free cma pages so many movable pages would be reclaimed. After then, we have a lot of movable pages again, so fallback allocation doesn't happen again. To conclude, amount of free memory on meminfo which includes free CMA pages is moving around 512 MB if I reserve 512 MB memory for CMA.

Another issue i am facing with the current code is the atomic allocation failing even with large number of CMA pages around. In my case we never reclaimed because large part of the memory is consumed by the page cache and for that, free memory check doesn't include at free_cma. I will test with this patchset and update here once i have the results.

Quoted 87 lines

I found this problem on following experiment.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

CMA reserve: 0 MB 512 MB
Elapsed-time: 234.8 361.8
Average-MemFree: 283880 KB 530851 KB

To solve this problem, I can think following 2 possible solutions.
1. allocate the pages on cma reserved memory first, and if they are exhausted, allocate movable pages.
2. interleaved allocation: try to allocate specific amounts of memory from cma reserved memory and then allocate from free movable memory.

I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess.

So, I implement #2 approach.
One thing I should note is that we should not change allocation target (movable list or cma) on each allocation attempt, since this prevent allocated pages to be in physically succession, so some I/O devices can be hurt their performance. To solve this, I keep allocation target in at least pageblock_nr_pages attempts and make this number reflect ratio, free pages without free cma pages to free cma pages. With this approach, system works very smoothly and fully utilize the pages on cma reserved memory.

Following is the experimental result of this patch.

4 CPUs, 1024 MB, VIRTUAL MACHINE
make -j24

<Before>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.8           361.8
Average-MemFree:        283880 KB       530851 KB
pswpin:                 7               110064
pswpout:                452             767502

<After>

CMA reserve:            0 MB            512 MB
Elapsed-time:           234.2           235.6
Average-MemFree:        281651 KB       290227 KB
pswpin:                 8               8
pswpout:                430             510

There is no difference if we don't have cma reserved memory (0 MB case). But, with cma reserved memory (512 MB case), we fully utilize these reserved memory through this patch and the system behaves like as it doesn't reserve any memory.

With this patch, we aggressively allocate the pages on cma reserved memory so latency of CMA can arise. Below is the experimental result about latency.

4 CPUs, 1024 MB, VIRTUAL MACHINE
CMA reserve: 512 MB
Backgound Workload: make -jN
Real Workload: 8 MB CMA allocation/free 20 times with 5 sec interval

N:                    1        4       8        16
Elapsed-time(Before): 4309.75  9511.09 12276.1  77103.5
Elapsed-time(After):  5391.69 16114.1  19380.3  34879.2

So generally we can see latency increase. Ratio of this increase is rather big - up to 70%. But, under the heavy workload, it shows latency decrease - up to 55%. This may be worst-case scenario, but reducing it would be important for some system, so, I can say that this patch have advantages and disadvantages in terms of latency.

Signed-off-by Joonsoo Kim <[email protected]>

include/linux/mmzone.h +0 -0

index fac5509..3ff24d4 100644
@@ -389,6 +389,12 @@ struct zone {
 	int			compact_order_failed;
 #endif
 

include/linux/mmzone.h struct zone { · line 396

Quoted 5 lines

include/linux/mmzone.h +5 -0

+#ifdef CONFIG_CMA
+	int has_cma;
+	int nr_try_cma;
+	int nr_try_movable;
+#endif

Can you write documentation around this ?

Quoted 86 lines

include/linux/mmzone.h +1 -0

+
 	ZONE_PADDING(_pad1_)
 
 	/* Fields commonly accessed by the page reclaim scanner */

mm/page_alloc.c +60 -0

index 674ade7..6f2b27b 100644
@@ -788,6 +788,16 @@ void __init __free_pages_bootmem(struct page *page, unsigned int order)
 }
 
 #ifdef CONFIG_CMA
+void __init init_alloc_ratio_counter(struct zone *zone)
+{
+	if (zone->has_cma)
+		return;
+
+	zone->has_cma = 1;
+	zone->nr_try_movable = 0;
+	zone->nr_try_cma = 0;
+}
+
 /* Free whole pageblock and set its migration type to MIGRATE_CMA. */
 void __init init_cma_reserved_pageblock(struct page *page)
 {

@@ -803,6 +813,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 	set_pageblock_migratetype(page, MIGRATE_CMA);
 	__free_pages(page, pageblock_order);
 	adjust_managed_page_count(page, pageblock_nr_pages);
+	init_alloc_ratio_counter(page_zone(page));
 }
 #endif
 

@@ -1136,6 +1147,69 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 	return NULL;
 }
 
+#ifdef CONFIG_CMA
+static struct page *__rmqueue_cma(struct zone *zone, unsigned int order,
+						int migratetype)
+{
+	long free, free_cma, free_wmark;
+	struct page *page;
+
+	if (migratetype != MIGRATE_MOVABLE || !zone->has_cma)
+		return NULL;
+
+	if (zone->nr_try_movable)
+		goto alloc_movable;
+
+alloc_cma:
+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}
+
+	/* Reset ratio counter */
+	free_cma = zone_page_state(zone, NR_FREE_CMA_PAGES);
+
+	/* No cma free pages, so recharge only movable allocation */
+	if (free_cma <= 0) {
+		zone->nr_try_movable = pageblock_nr_pages;
+		goto alloc_movable;
+	}
+
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	free_wmark = free - free_cma - high_wmark_pages(zone);
+
+	/*
+	 * free_wmark is below than 0, and it means that normal pages
+	 * are under the pressure, so we recharge only cma allocation.
+	 */
+	if (free_wmark <= 0) {
+		zone->nr_try_cma = pageblock_nr_pages;
+		goto alloc_cma;
+	}
+
+	if (free_wmark > free_cma) {
+		zone->nr_try_movable =

mm/page_alloc.c __rmqueue_fallback(struct zone *zone, int order, int start_migratetype) · line 1204

Quoted 6 lines

mm/page_alloc.c +6 -0

+			(free_wmark * pageblock_nr_pages) / free_cma;
+		zone->nr_try_cma = pageblock_nr_pages;
+	} else {
+		zone->nr_try_movable = pageblock_nr_pages;
+		zone->nr_try_cma = free_cma * pageblock_nr_pages / free_wmark;
+	}

Can you add the commit message documentation here.

Quoted 12 lines

mm/page_alloc.c +8 -0

+
+	/* Reset complete, start on movable first */
+alloc_movable:
+	zone->nr_try_movable--;
+	return NULL;
+}
+#endif
+
 /*
  * Do the hard work of removing an element from the buddy allocator.
  * Call me with the zone->lock already held.

mm/page_alloc.c __rmqueue_fallback(struct zone *zone, int order, int start_migratetype) · line 1223

Quoted 8 lines

mm/page_alloc.c +4 -1

@@ -1143,10 +1217,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)
 static struct page *__rmqueue(struct zone *zone, unsigned int order,
 						int migratetype)
 {
-	struct page *page;
+	struct page *page = NULL;
+
+	if (IS_ENABLED(CONFIG_CMA))
+		page = __rmqueue_cma(zone, order, migratetype);

It would be better to move the migrate check here, So that it becomes

         /* For migrate movable allocation try cma area first */
	if (IS_ENABLED(CONFIG_CMA) && (migratetype == MIGRATE_MOVABLE))

Quoted 10 lines · Aneesh Kumar K.V

Joonsoo Kim <[email protected]> writes:

Another issue i am facing with the current code is the atomic allocation failing even with large number of CMA pages around. In my case we never reclaimed because large part of the memory is consumed by the page cache and for that, free memory check doesn't include at free_cma. I will test with this patchset and update here once i have the results.

Hello,

Could you elaborate more on your issue?
I can't completely understand your problem.
So your atomic allocation is movable? And although there are many free cma pages, that request is fail?

Quoted 12 lines · Joonsoo Kim

Quoted 10 lines · Aneesh Kumar K.V

Joonsoo Kim <[email protected]> writes:

Another issue i am facing with the current code is the atomic allocation failing even with large number of CMA pages around. In my case we never reclaimed because large part of the memory is consumed by the page cache and for that, free memory check doesn't include at free_cma. I will test with this patchset and update here once i have the results.

Quoted 7 lines

Hello,

Could you elaborate more on your issue?
I can't completely understand your problem.
So your atomic allocation is movable? And although there are many free cma pages, that request is fail?

non movable atomic allocations are failing because we don't have anything other than CMA pages left and kswapd is yet to catchup ?

swapper/0: page allocation failure: order:0, mode:0x20

  CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.23-1500.pkvm2_1.5.ppc64 #1
  Call Trace:
 [c000000ffffcb610] [c000000000017330] .show_stack+0x130/0x200 (unreliable)
 [c000000ffffcb6e0] [c00000000087a8c8] .dump_stack+0x28/0x3c
 [c000000ffffcb750] [c0000000001e06f0] .warn_alloc_failed+0x110/0x160
 [c000000ffffcb800] [c0000000001e5984] .__alloc_pages_nodemask+0x9d4/0xbf0
 [c000000ffffcb9e0] [c00000000023775c] .alloc_pages_current+0xcc/0x1b0
 [c000000ffffcba80] [c0000000007098d4] .__netdev_alloc_frag+0x1a4/0x1d0
 [c000000ffffcbb20] [c00000000070d750] .__netdev_alloc_skb+0xc0/0x130
 [c000000ffffcbbb0] [d000000009639b40] .tg3_poll_work+0x900/0x1110 [tg3]
 [c000000ffffcbd10] [d00000000963a3a4] .tg3_poll_msix+0x54/0x200 [tg3]
 [c000000ffffcbdb0] [c00000000071fcec] .net_rx_action+0x1dc/0x310
 [c000000ffffcbe90] [c0000000000c1b08] .__do_softirq+0x158/0x330
 [c000000ffffcbf90] [c000000000025744] .call_do_softirq+0x14/0x24
 [c000000ffffc7e00] [c000000000011684] .do_softirq+0xf4/0x130
 [c000000ffffc7e90] [c0000000000c1f18] .irq_exit+0xc8/0x110
 [c000000ffffc7f10] [c000000000011258] .__do_irq+0xc8/0x1f0
 [c000000ffffc7f90] [c000000000025768] .call_do_irq+0x14/0x24
 [c00000000137b750] [c00000000001142c] .do_IRQ+0xac/0x130
 [c00000000137b800] [c000000000002a64]
 hardware_interrupt_common+0x164/0x180

....

Node 0 DMA: 408*64kB (C) 408*128kB (C) 408*256kB (C) 408*512kB (C) 408*1024kB (C) 406*2048kB (C) 199*4096kB (C) 97*8192kB (C) 6*16384kB (C) = 3348992kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16384kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16777216kB

meminfo details:

MemTotal: 65875584 kB
MemFree: 8001856 kB
Buffers: 49330368 kB
Cached: 178752 kB
SwapCached: 0 kB
Active: 28550464 kB
Inactive: 25476416 kB
Active(anon): 3771008 kB
Inactive(anon): 767360 kB
Active(file): 24779456 kB
Inactive(file): 24709056 kB
Unevictable: 15104 kB
Mlocked: 15104 kB
SwapTotal: 8384448 kB
SwapFree: 8384448 kB
Dirty: 0 kB

-aneesh

commit d95ea5d1('cma: fix watermark checking') introduces ALLOC_CMA flag for alloc flag and treats free cma pages as free pages if this flag is passed to watermark checking. Intention of that patch is that movable page allocation can be be handled from cma reserved region without starting kswapd. Now, previous patch changes the behaviour of allocator that movable allocation uses the page on cma reserved region aggressively, so this watermark hack isn't needed anymore. Therefore remove it.

Signed-off-by Joonsoo Kim <[email protected]>

This series tries to improve CMA.

CMA is introduced to provide physically contiguous pages at runtime without reserving memory area. But, current implementation works like as reserving memory approach, because allocation on cma reserved region only occurs as fallback of migrate_movable allocation. We can allocate from it when there is no movable page. In that situation, kswapd would be invoked easily since unmovable and reclaimable allocation consider (free pages - free CMA pages) as free memory on the system and free memory may be lower than high watermark in that case. If kswapd start to reclaim memory, then fallback allocation doesn't occur much.

In my experiment, I found that if system memory has 1024 MB memory and has 512 MB reserved memory for CMA, kswapd is mostly invoked around the 512MB free memory boundary. And invoked kswapd tries to make free memory until (free pages - free CMA pages) is higher than high watermark, so free memory on meminfo is moving around 512MB boundary consistently.

To fix this problem, we should allocate the pages on cma reserved memory more aggressively and intelligenetly. Patch 2 implements the solution. Patch 1 is the simple optimization which remove useless re-trial and patch 3 is for removing useless alloc flag, so these are not important. See patch 2 for more detailed description.

This patchset is based on v3.15-rc4.

Thanks.
Joonsoo Kim (3):
CMA: remove redundant retrying code in __alloc_contig_migrate_range
CMA: aggressively allocate the pages on cma reserved memory when not
used
CMA: always treat free cma pages as non-free on watermark checking

  include/linux/mmzone.h |    6 +++
  mm/compaction.c        |    4 --
  mm/internal.h          |    3 +-
  mm/page_alloc.c        |  117 +++++++++++++++++++++++++++++++++++++++---------
  4 files changed, 102 insertions(+), 28 deletions(-)

Hello,

Quoted 18 lines · Joonsoo Kim

This series tries to improve CMA.

CMA is introduced to provide physically contiguous pages at runtime without reserving memory area. But, current implementation works like as reserving memory approach, because allocation on cma reserved region only occurs as fallback of migrate_movable allocation. We can allocate from it when there is no movable page. In that situation, kswapd would be invoked easily since unmovable and reclaimable allocation consider (free pages - free CMA pages) as free memory on the system and free memory may be lower than high watermark in that case. If kswapd start to reclaim memory, then fallback allocation doesn't occur much.

In my experiment, I found that if system memory has 1024 MB memory and has 512 MB reserved memory for CMA, kswapd is mostly invoked around the 512MB free memory boundary. And invoked kswapd tries to make free memory until (free pages - free CMA pages) is higher than high watermark, so free memory on meminfo is moving around 512MB boundary consistently.

Quoted 7 lines

To fix this problem, we should allocate the pages on cma reserved memory more aggressively and intelligenetly. Patch 2 implements the solution. Patch 1 is the simple optimization which remove useless re-trial and patch 3 is for removing useless alloc flag, so these are not important. See patch 2 for more detailed description.

This patchset is based on v3.15-rc4.

Thanks for posting those patches. It basically reminds me the following discussion:
http://thread.gmane.org/gmane.linux.kernel/1391989/focus=1399524

Your approach is basically the same. I hope that your patches can be improved
in such a way that they will be accepted by mm maintainers. I only wonder if the
third patch is really necessary. Without it kswapd wakeup might be still avoided
in some cases.

Quoted 33 lines · Marek Szyprowski

Hello,

Quoted 18 lines · Joonsoo Kim

This series tries to improve CMA.

CMA is introduced to provide physically contiguous pages at runtime without reserving memory area. But, current implementation works like as reserving memory approach, because allocation on cma reserved region only occurs as fallback of migrate_movable allocation. We can allocate from it when there is no movable page. In that situation, kswapd would be invoked easily since unmovable and reclaimable allocation consider (free pages - free CMA pages) as free memory on the system and free memory may be lower than high watermark in that case. If kswapd start to reclaim memory, then fallback allocation doesn't occur much.

In my experiment, I found that if system memory has 1024 MB memory and has 512 MB reserved memory for CMA, kswapd is mostly invoked around the 512MB free memory boundary. And invoked kswapd tries to make free memory until (free pages - free CMA pages) is higher than high watermark, so free memory on meminfo is moving around 512MB boundary consistently.

Quoted 7 lines

To fix this problem, we should allocate the pages on cma reserved memory more aggressively and intelligenetly. Patch 2 implements the solution. Patch 1 is the simple optimization which remove useless re-trial and patch 3 is for removing useless alloc flag, so these are not important. See patch 2 for more detailed description.

This patchset is based on v3.15-rc4.

Thanks for posting those patches. It basically reminds me the following discussion:
http://thread.gmane.org/gmane.linux.kernel/1391989/focus=1399524

Quoted 7 lines

Your approach is basically the same. I hope that your patches can be improved
in such a way that they will be accepted by mm maintainers. I only wonder if the
third patch is really necessary. Without it kswapd wakeup might be still avoided
in some cases.

Hello,

Oh... I didn't know that patch and discussion, because I have no interest on CMA at that time. Your approach looks similar to #1
approach of mine and could have same problem of #1 approach which I mentioned in patch 2/3. Please refer that patch description. :)

And, there is different purpose between this and yours. This patch is intended to better use of CMA pages and so get maximum performance. Just to not trigger oom, it can be possible to put this logic on reclaim path. But that is sub-optimal to get higher performance, because it needs migration in some cases.

If second patch works as intended, there are just a few of cma free pages when we are toward on the watermark. So benefit of third patch would be marginal and we can remove ALLOC_CMA.

Thanks.

Quoted 42 lines · Joonsoo Kim

Quoted 33 lines · Marek Szyprowski

Hello,

Quoted 18 lines · Joonsoo Kim

This series tries to improve CMA.

CMA is introduced to provide physically contiguous pages at runtime without reserving memory area. But, current implementation works like as reserving memory approach, because allocation on cma reserved region only occurs as fallback of migrate_movable allocation. We can allocate from it when there is no movable page. In that situation, kswapd would be invoked easily since unmovable and reclaimable allocation consider (free pages - free CMA pages) as free memory on the system and free memory may be lower than high watermark in that case. If kswapd start to reclaim memory, then fallback allocation doesn't occur much.

In my experiment, I found that if system memory has 1024 MB memory and has 512 MB reserved memory for CMA, kswapd is mostly invoked around the 512MB free memory boundary. And invoked kswapd tries to make free memory until (free pages - free CMA pages) is higher than high watermark, so free memory on meminfo is moving around 512MB boundary consistently.

Quoted 7 lines

To fix this problem, we should allocate the pages on cma reserved memory more aggressively and intelligenetly. Patch 2 implements the solution. Patch 1 is the simple optimization which remove useless re-trial and patch 3 is for removing useless alloc flag, so these are not important. See patch 2 for more detailed description.

This patchset is based on v3.15-rc4.

Thanks for posting those patches. It basically reminds me the following discussion:
http://thread.gmane.org/gmane.linux.kernel/1391989/focus=1399524

Quoted 7 lines

Your approach is basically the same. I hope that your patches can be improved
in such a way that they will be accepted by mm maintainers. I only wonder if the
third patch is really necessary. Without it kswapd wakeup might be still avoided
in some cases.

Quoted 6 lines

Hello,

Oh... I didn't know that patch and discussion, because I have no interest on CMA at that time. Your approach looks similar to #1
approach of mine and could have same problem of #1 approach which I mentioned in patch 2/3. Please refer that patch description. :)

IIUC that patch also interleave right ?

That doesn't always prefer CMA region. It would be nice to understand why grouping in pageblock_nr_pages is beneficial. Also in your patch you decrement nr_try_cma for every 'order' allocation. Why ?

(quoted diff) +11 -0

+	if (zone->nr_try_cma) {
+		/* Okay. Now, we can try to allocate the page from cma region */
+		zone->nr_try_cma--;
+		page = __rmqueue_smallest(zone, order, MIGRATE_CMA);
+
+		/* CMA pages can vanish through CMA allocation */
+		if (unlikely(!page && order == 0))
+			zone->nr_try_cma = 0;
+
+		return page;
+	}

If we fail above MIGRATE_CMA alloc should we return failure ? Why not try MOVABLE allocation on failure (ie fallthrough the code path) ?

Quoted 3 lines

That doesn't always prefer CMA region. It would be nice to understand why grouping in pageblock_nr_pages is beneficial. Also in your patch you decrement nr_try_cma for every 'order' allocation. Why ?

pageblock_nr_pages is just magic value with no rationale. :)

Quoted 5 lines · Joonsoo Kim

Quoted 3 lines

That doesn't always prefer CMA region. It would be nice to understand why grouping in pageblock_nr_pages is beneficial. Also in your patch you decrement nr_try_cma for every 'order' allocation. Why ?

pageblock_nr_pages is just magic value with no rationale. :)

I'm not following this discussions closely but there is rational to that value -- it's the size of a huge page for that architecture. At the time the fragmentation avoidance was implemented this was the largest allocation size of interest.