Support high-order page bulk allocation

unsigned long isolate_freepages_range(struct compact_control *cc,

mm/internal.h +2 -1

-			unsigned long start_pfn, unsigned long end_pfn);
+			unsigned long start_pfn, unsigned long end_pfn,
+			struct list_head *freepage_list);
 unsigned long
 isolate_migratepages_range(struct compact_control *cc,
 			   unsigned long low_pfn, unsigned long end_pfn);

mm/page_alloc.c +0 -0

index caf393d8b413..cdf956feae80 100644

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

int alloc_pages_bulk(unsigned long start, unsigned long end,
unsigned int migratetype, gfp_t gfp_mask, unsigned int order, unsigned int nr_elem, struct page **pages);

It will investigate the [start, end) and migrate movable pages out there by best effort(by upcoming patches) to make requested order's free pages.

The allocated pages will be returned using pages parameter. Return value represents how many of requested order pages we got. It could be less than user requested by nr_elem.

Quoted 2 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

... but you haven't shown that user.

Quoted 10 lines

int alloc_pages_bulk(unsigned long start, unsigned long end,
unsigned int migratetype, gfp_t gfp_mask, unsigned int order, unsigned int nr_elem, struct page **pages);

It will investigate the [start, end) and migrate movable pages out there by best effort(by upcoming patches) to make requested order's free pages.

Quoted 3 lines

The allocated pages will be returned using pages parameter. Return value represents how many of requested order pages we got. It could be less than user requested by nr_elem.

I don't understand why a user would need to know the PFNs to allocate between. This seems like something that's usually specified by GFP_DMA32 or similar.

Is it useful to return fewer pages than requested?

Quoted 5 lines · Matthew Wilcox

Quoted 2 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

... but you haven't shown that user.

Kyoungho is working on it.
I am not sure how much he could share but hopefully, he could show some. Kyoungho?

Quoted 10 lines

Quoted 10 lines

int alloc_pages_bulk(unsigned long start, unsigned long end,
unsigned int migratetype, gfp_t gfp_mask, unsigned int order, unsigned int nr_elem, struct page **pages);

It will investigate the [start, end) and migrate movable pages out there by best effort(by upcoming patches) to make requested order's free pages.

Quoted 7 lines

Quoted 3 lines

The allocated pages will be returned using pages parameter. Return value represents how many of requested order pages we got. It could be less than user requested by nr_elem.

I don't understand why a user would need to know the PFNs to allocate between. This seems like something that's usually specified by GFP_DMA32 or similar.

I wanted to let the API wok from CMA area and/or movable zone where are always fulled with migrable pages.
If we carry on only zone flags without pfn range, it couldn't fulfil cma area cases.
Other reason is if user see fewer pages returned, he could try subsequent ranges to get remained ones.

Quoted 1 line

Is it useful to return fewer pages than requested?

It's useful because user could ask further than what they need or retry.

Quoted 5 lines · Matthew Wilcox

Quoted 2 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

... but you haven't shown that user.

Kyoungho is working on it.
I am not sure how much he could share but hopefully, he could show some. Kyoungho?

Quoted 10 lines

Quoted 10 lines

int alloc_pages_bulk(unsigned long start, unsigned long end,
unsigned int migratetype, gfp_t gfp_mask, unsigned int order, unsigned int nr_elem, struct page **pages);

It will investigate the [start, end) and migrate movable pages out there by best effort(by upcoming patches) to make requested order's free pages.

Quoted 7 lines

Quoted 3 lines

The allocated pages will be returned using pages parameter. Return value represents how many of requested order pages we got. It could be less than user requested by nr_elem.

I don't understand why a user would need to know the PFNs to allocate between. This seems like something that's usually specified by GFP_DMA32 or similar.

I wanted to let the API wok from CMA area and/or movable zone where are always fulled with migrable pages.
If we carry on only zone flags without pfn range, it couldn't fulfil cma area cases.
Other reason is if user see fewer pages returned, he could try subsequent ranges to get remained ones.

Quoted 1 line

Is it useful to return fewer pages than requested?

It's useful because user could ask further than what they need or retry.

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

To solve issues, this patch introduces alloc_pages_bulk.

int alloc_pages_bulk(unsigned long start, unsigned long end,
unsigned int migratetype, gfp_t gfp_mask, unsigned int order, unsigned int nr_elem, struct page **pages);

It will investigate the [start, end) and migrate movable pages out there by best effort(by upcoming patches) to make requested order's free pages.

The allocated pages will be returned using pages parameter. Return value represents how many of requested order pages we got. It could be less than user requested by nr_elem.

/**
* alloc_pages_bulk() -- tries to allocate high order pages
* by batch from given range [start, end)
* @start: start PFN to allocate
* @end: one-past-the-last PFN to allocate

 * @migratetype:        migratetype of the underlaying pageblocks (either
 *                      #MIGRATE_MOVABLE or #MIGRATE_CMA).  All pageblocks
 *                      in range must have the same migratetype and it must
 *                      be either of the two.
 * @gfp_mask:   GFP mask to use during compaction
 * @order:      page order requested
 * @nr_elem:    the number of high-order pages to allocate
 * @pages:      page array pointer to store allocated pages (must
 *              have space for at least nr_elem elements)

*
* The PFN range does not have to be pageblock or MAX_ORDER_NR_PAGES
* aligned. The PFN range must belong to a single zone.
*
* Return: the number of pages allocated on success or negative error code.
* The allocated pages should be freed using __free_pages
*/

The test goes order-4 * 4800 allocation(i.e., total 300MB) under kernel build workload. System RAM size is 1.5GB and CMA is 500M.

With using CMA to allocate to 300M, ran 10 times trial, 10 time failed with big latency(up to several seconds).

With this alloc_pages_bulk API, ran 10 time trial, 7 times are successful to allocate 4800 times. Rest 3 times are allocated 4799, 4789 and 4799. They are all done with 300ms.

This patchset is against on next-20200813

Minchan Kim (7):
mm: page_owner: split page by order
mm: introduce split_page_by_order
mm: compaction: deal with upcoming high-order page splitting
mm: factor __alloc_contig_range out
mm: introduce alloc_pages_bulk API
mm: make alloc_pages_bulk best effort mm/page_isolation: avoid drain_all_pages for alloc_pages_bulk

 include/linux/gfp.h            |   5 +
 include/linux/mm.h             |   2 +
 include/linux/page-isolation.h |   1 +
 include/linux/page_owner.h     |  10 +-
 mm/compaction.c                |  64 +++++++----
 mm/huge_memory.c               |   2 +-
 mm/internal.h                  |   5 +-
 mm/page_alloc.c                | 198 ++++++++++++++++++++++++++-------
 mm/page_isolation.c            |  10 +-
 mm/page_owner.c                |   7 +-
 10 files changed, 230 insertions(+), 74 deletions(-)

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I don't see a real need for a completely new range allocator function for this special case yet.

Quoted 71 lines

To solve issues, this patch introduces alloc_pages_bulk.

int alloc_pages_bulk(unsigned long start, unsigned long end,
unsigned int migratetype, gfp_t gfp_mask, unsigned int order, unsigned int nr_elem, struct page **pages);

It will investigate the [start, end) and migrate movable pages out there by best effort(by upcoming patches) to make requested order's free pages.

The allocated pages will be returned using pages parameter. Return value represents how many of requested order pages we got. It could be less than user requested by nr_elem.

/**
* alloc_pages_bulk() -- tries to allocate high order pages
* by batch from given range [start, end)
* @start: start PFN to allocate
* @end: one-past-the-last PFN to allocate

 * @migratetype:        migratetype of the underlaying pageblocks (either
 *                      #MIGRATE_MOVABLE or #MIGRATE_CMA).  All pageblocks
 *                      in range must have the same migratetype and it must
 *                      be either of the two.
 * @gfp_mask:   GFP mask to use during compaction
 * @order:      page order requested
 * @nr_elem:    the number of high-order pages to allocate
 * @pages:      page array pointer to store allocated pages (must
 *              have space for at least nr_elem elements)

*
* The PFN range does not have to be pageblock or MAX_ORDER_NR_PAGES
* aligned. The PFN range must belong to a single zone.
*
* Return: the number of pages allocated on success or negative error code.
* The allocated pages should be freed using __free_pages
*/

The test goes order-4 * 4800 allocation(i.e., total 300MB) under kernel build workload. System RAM size is 1.5GB and CMA is 500M.

With using CMA to allocate to 300M, ran 10 times trial, 10 time failed with big latency(up to several seconds).

With this alloc_pages_bulk API, ran 10 time trial, 7 times are successful to allocate 4800 times. Rest 3 times are allocated 4799, 4789 and 4799. They are all done with 300ms.

This patchset is against on next-20200813

Minchan Kim (7):
mm: page_owner: split page by order
mm: introduce split_page_by_order
mm: compaction: deal with upcoming high-order page splitting
mm: factor __alloc_contig_range out
mm: introduce alloc_pages_bulk API
mm: make alloc_pages_bulk best effort mm/page_isolation: avoid drain_all_pages for alloc_pages_bulk

 include/linux/gfp.h            |   5 +
 include/linux/mm.h             |   2 +
 include/linux/page-isolation.h |   1 +
 include/linux/page_owner.h     |  10 +-
 mm/compaction.c                |  64 +++++++----
 mm/huge_memory.c               |   2 +-
 mm/internal.h                  |   5 +-
 mm/page_alloc.c                | 198 ++++++++++++++++++++++++++-------
 mm/page_isolation.c            |  10 +-
 mm/page_owner.c                |   7 +-
 10 files changed, 230 insertions(+), 74 deletions(-)

Signature · 3 lines

Thanks,

David / dhildenb

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Quoted 30 lines · Minchan Kim

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

Quoted 4 lines

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

I was talking about doing MAX_ORDER - 1 CMA allocations instead of one big 300M allocation. As you correctly note, memory placed into CMA should be movable, except for (short/long) term pinnings. In these cases, doing allocations smaller than 300M and splitting them up should be good enough to reduce the call frequency, no?

Quoted 6 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Just wondering

a) Why does it have to be fast?

b) Why does it need that many order-4 pages?

c) How dynamic is the device need at runtime?

d) Would it be reasonable in your setup to mark a CMA region in a way such that it will never be used for other (movable) allocations,

guaranteeing that you can immediately allocate it? Something like, reserving a region during boot you know you'll immediately need later completely for a device?

Signature · 3 lines

Thanks,

David / dhildenb

Quoted 36 lines · David Hildenbrand

Quoted 30 lines · Minchan Kim

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

Quoted 4 lines

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

Quoted 5 lines

I was talking about doing MAX_ORDER - 1 CMA allocations instead of one big 300M allocation. As you correctly note, memory placed into CMA should be movable, except for (short/long) term pinnings. In these cases, doing allocations smaller than 300M and splitting them up should be good enough to reduce the call frequency, no?

I should have written that. The 300M I mentioned is really minimum size. In some scenraio, we need way bigger than 300M, up to several GB. Furthermore, the demand would be increased in near future.

Quoted 7 lines

Quoted 6 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Quoted 3 lines

Just wondering

a) Why does it have to be fast?

That's because it's related to application latency, which ends up user feel bad.

Quoted 2 lines

b) Why does it need that many order-4 pages?

It's HW requirement. I couldn't say much about that.

Quoted 2 lines

c) How dynamic is the device need at runtime?

Whenever the application launched. It depends on user's usage pattern.

Quoted 2 lines

d) Would it be reasonable in your setup to mark a CMA region in a way such that it will never be used for other (movable) allocations,

I don't get your point. If we don't want the area to used up for other movable allocation, why should we use it as CMA first? It sounds like reserved memory and just wasted the memory.

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Just wondering

a) Why does it have to be fast?

Quoted 2 lines

That's because it's related to application latency, which ends up user feel bad.

Okay, but in theory, your device-needs are very similar to application-needs, besides you requiring order-4 pages, correct? Similar to an application that starts up and pins 300M (or more), just with ordr-4 pages.

Quoted 43 lines · Minchan Kim

Quoted 36 lines · David Hildenbrand

Quoted 30 lines · Minchan Kim

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

Quoted 4 lines

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

Quoted 5 lines

I was talking about doing MAX_ORDER - 1 CMA allocations instead of one big 300M allocation. As you correctly note, memory placed into CMA should be movable, except for (short/long) term pinnings. In these cases, doing allocations smaller than 300M and splitting them up should be good enough to reduce the call frequency, no?

Quoted 3 lines

I should have written that. The 300M I mentioned is really minimum size. In some scenraio, we need way bigger than 300M, up to several GB. Furthermore, the demand would be increased in near future.

And what will the driver do with that data besides providing it to the device? Can it be mapped to user space? I think we really need more information / the actual user.

Quoted 11 lines

Quoted 10 lines

Quoted 6 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Just wondering

a) Why does it have to be fast?

Quoted 2 lines

That's because it's related to application latency, which ends up user feel bad.

Okay, but in theory, your device-needs are very similar to application-needs, besides you requiring order-4 pages, correct? Similar to an application that starts up and pins 300M (or more), just with ordr-4 pages.

I don't get quite yet why you need a range allocator for that. Because you intend to use CMA?

Quoted 5 lines

Quoted 3 lines

b) Why does it need that many order-4 pages?

It's HW requirement. I couldn't say much about that.

Hm.

Quoted 11 lines

Quoted 2 lines

c) How dynamic is the device need at runtime?

Whenever the application launched. It depends on user's usage pattern.

Quoted 2 lines

d) Would it be reasonable in your setup to mark a CMA region in a way such that it will never be used for other (movable) allocations,

I don't get your point. If we don't want the area to used up for other movable allocation, why should we use it as CMA first? It sounds like reserved memory and just wasted the memory.

Right, it's just very hard to get what you are trying to achieve without the actual user at hand.

For example, will the pages you allocate be movable? Does the device allow for that? If not, then the MOVABLE zone is usually not valid (similar to gigantic pages not being allocated from the MOVABLE zone). So your stuck with the NORMAL zone or CMA. Especially for the NORMAL zone, alloc_contig_range() is currently not prepared to properly handle sub-MAX_ORDER - 1 ranges. If any involved pageblock contains an unmovable page, the allcoation will fail (see pageblock isolation / has_unmovable_pages()). So CMA would be your only option.

Quoted 66 lines · Minchan Kim

Quoted 43 lines · David Hildenbrand

Quoted 36 lines · Minchan Kim

Quoted 30 lines · David Hildenbrand

Quoted 11 lines · Minchan Kim

Quoted 10 lines

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

Quoted 4 lines

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

Quoted 5 lines

I was talking about doing MAX_ORDER - 1 CMA allocations instead of one big 300M allocation. As you correctly note, memory placed into CMA should be movable, except for (short/long) term pinnings. In these cases, doing allocations smaller than 300M and splitting them up should be good enough to reduce the call frequency, no?

Quoted 3 lines

I should have written that. The 300M I mentioned is really minimum size. In some scenraio, we need way bigger than 300M, up to several GB. Furthermore, the demand would be increased in near future.

And what will the driver do with that data besides providing it to the device? Can it be mapped to user space? I think we really need more information / the actual user.

Quoted 13 lines

Quoted 10 lines

Quoted 6 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Just wondering

a) Why does it have to be fast?

That's because it's related to application latency, which ends up user feel bad.

Quoted 4 lines

Okay, but in theory, your device-needs are very similar to application-needs, besides you requiring order-4 pages, correct? Similar to an application that starts up and pins 300M (or more), just with ordr-4 pages.

Yes.

Quoted 63 lines · David Hildenbrand

Quoted 43 lines · Minchan Kim

Quoted 36 lines · David Hildenbrand

Quoted 30 lines · Minchan Kim

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

Quoted 4 lines

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

Quoted 5 lines

I was talking about doing MAX_ORDER - 1 CMA allocations instead of one big 300M allocation. As you correctly note, memory placed into CMA should be movable, except for (short/long) term pinnings. In these cases, doing allocations smaller than 300M and splitting them up should be good enough to reduce the call frequency, no?

Quoted 3 lines

I should have written that. The 300M I mentioned is really minimum size. In some scenraio, we need way bigger than 300M, up to several GB. Furthermore, the demand would be increased in near future.

And what will the driver do with that data besides providing it to the device? Can it be mapped to user space? I think we really need more information / the actual user.

Quoted 11 lines

Quoted 10 lines

Quoted 6 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Just wondering

a) Why does it have to be fast?

Quoted 7 lines

Quoted 2 lines

That's because it's related to application latency, which ends up user feel bad.

Okay, but in theory, your device-needs are very similar to application-needs, besides you requiring order-4 pages, correct? Similar to an application that starts up and pins 300M (or more), just with ordr-4 pages.

Yes.

Quoted 3 lines

I don't get quite yet why you need a range allocator for that. Because you intend to use CMA?

Yes, with CMA, it could be more guaranteed and fast enough with little tweaking. Currently, CMA is too slow due to below IPI overheads.

1. set_migratetype_isolate does drain_all_pages for every pageblock.
2. __aloc_contig_migrate_range does migrate_prep
3. alloc_contig_range does lru_add_drain_all.

Thus, if we need to increase frequency of call as your suggestion, the set up overhead is also scaling up depending on the size. Such overhead makes sense if caller requests big contiguous memory but it's too much for normal high-order allocations.

Maybe, we might optimize those call sites to reduce or/remove frequency of those IPI call smarter way but that would need to deal with success ratio vs fastness dance in the end.

Another concern to use existing cma API is it's trying to make allocation successful at the cost of latency. For example, waiting a page writeback.

That's the this new sematic API comes from for compromise since I believe we need some way to separate original CMA alloc(biased to be guaranteed but slower) with this new API(biased to be fast but less guaranteed).

Is there any idea to work without existing cma API tweaking?

Quoted 98 lines · Minchan Kim

Quoted 63 lines · David Hildenbrand

Quoted 43 lines · Minchan Kim

Quoted 36 lines · David Hildenbrand

Quoted 30 lines · Minchan Kim

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Minchan Kim

There is a need for special HW to require bulk allocation of high-order pages. For example, 4800 * order-4 pages.

To meet the requirement, a option is using CMA area because page allocator with compaction under memory pressure is easily failed to meet the requirement and too slow for 4800 times. However, CMA has also the following drawbacks:

* 4800 of order-4 * cma_alloc is too slow

Quoted 8 lines

Quoted 5 lines

To avoid the slowness, we could try to allocate 300M contiguous memory once and then split them into order-4 chunks. The problem of this approach is CMA allocation fails one of the pages in those range couldn't migrate out, which happens easily with fs write under memory pressure.

Why not chose a value in between? Like try to allocate MAX_ORDER - 1 chunks and split them. That would already heavily reduce the call frequency.

I think you meant this:

alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)

It would work if system has lots of non-fragmented free memory. However, once they are fragmented, it doesn't work. That's why we have seen even order-4 allocation failure in the field easily and that's why CMA was there.

Quoted 4 lines

CMA has more logics to isolate the memory during allocation/freeing as well as fragmentation avoidance so that it has less chance to be stealed from others and increase high success ratio. That's why I want this API to be used with CMA or movable zone.

Quoted 5 lines

I was talking about doing MAX_ORDER - 1 CMA allocations instead of one big 300M allocation. As you correctly note, memory placed into CMA should be movable, except for (short/long) term pinnings. In these cases, doing allocations smaller than 300M and splitting them up should be good enough to reduce the call frequency, no?

Quoted 3 lines

I should have written that. The 300M I mentioned is really minimum size. In some scenraio, we need way bigger than 300M, up to several GB. Furthermore, the demand would be increased in near future.

And what will the driver do with that data besides providing it to the device? Can it be mapped to user space? I think we really need more information / the actual user.

Quoted 11 lines

Quoted 10 lines

Quoted 6 lines

A usecase is device can set a exclusive CMA area up when system boots. When device needs 4800 * order-4 pages, it could call this bulk against of the area so that it could effectively be guaranteed to allocate enough fast.

Just wondering

a) Why does it have to be fast?

Quoted 7 lines

Quoted 2 lines

That's because it's related to application latency, which ends up user feel bad.

Okay, but in theory, your device-needs are very similar to application-needs, besides you requiring order-4 pages, correct? Similar to an application that starts up and pins 300M (or more), just with ordr-4 pages.

Yes.

Quoted 3 lines

I don't get quite yet why you need a range allocator for that. Because you intend to use CMA?

Yes, with CMA, it could be more guaranteed and fast enough with little tweaking. Currently, CMA is too slow due to below IPI overheads.

1. set_migratetype_isolate does drain_all_pages for every pageblock.
2. __aloc_contig_migrate_range does migrate_prep
3. alloc_contig_range does lru_add_drain_all.

Thus, if we need to increase frequency of call as your suggestion, the set up overhead is also scaling up depending on the size. Such overhead makes sense if caller requests big contiguous memory but it's too much for normal high-order allocations.

Maybe, we might optimize those call sites to reduce or/remove frequency of those IPI call smarter way but that would need to deal with success ratio vs fastness dance in the end.

Another concern to use existing cma API is it's trying to make allocation successful at the cost of latency. For example, waiting a page writeback.

Quoted 5 lines

That's the this new sematic API comes from for compromise since I believe we need some way to separate original CMA alloc(biased to be guaranteed but slower) with this new API(biased to be fast but less guaranteed).

Is there any idea to work without existing cma API tweaking?

Let me try to summarize:

1. Your driver needs a lot of order-4 pages. And it's needs them fast, because of observerable lag/delay in an application. The pages will be unmovable by the driver.

2. Your idea is to use CMA, as that avoids unmovable allocations, theoretically allowing you to allocate all memory. But you don't actually want a large contiguous memory area.

3. Doing a whole bunch of order-4 cma allocations is slow.

4. Doing a single large cma allocation and splitting it manually in the caller can fail easily due to temporary page pinnings.

Regarding 4., [1] comes to mind, which has the same issues with temporary page pinnings and solves it by simply retrying. Yeah, there will be some lag, but maybe it's overall faster than doing separate order-4 cma allocations?

In general, proactive compaction [2] comes to mind, does that help?

[1]
https://lore.kernel.org/r/[email protected]/
[2] https://nitingupta.dev/post/proactive-compaction/

I understand pfn stuff in the API is not pretty but the concept of idea makes sense to me in that go though the *migratable area* and get possible order pages with hard effort. It looks like GFP_NORETRY version for kmem_cache_alloc_bulk.

How about this?

    int cma_alloc(struct cma *cma, int order, unsigned int nr_elem, struct page **pages);

Quoted 5 lines · Minchan Kim

I understand pfn stuff in the API is not pretty but the concept of idea makes sense to me in that go though the *migratable area* and get possible order pages with hard effort. It looks like GFP_NORETRY version for kmem_cache_alloc_bulk.

Quoted 3 lines

How about this?

    int cma_alloc(struct cma *cma, int order, unsigned int nr_elem, struct page **pages);

I think that makes a lot more sense as an API. Although I think you want

int cma_bulk_alloc(struct cma *cma, unsigned order, unsigned nr_elem,

		struct page **pages);

Quoted 6 lines · Matthew Wilcox

Quoted 5 lines · Minchan Kim

I understand pfn stuff in the API is not pretty but the concept of idea makes sense to me in that go though the *migratable area* and get possible order pages with hard effort. It looks like GFP_NORETRY version for kmem_cache_alloc_bulk.

Quoted 9 lines

Quoted 3 lines

How about this?

    int cma_alloc(struct cma *cma, int order, unsigned int nr_elem, struct page **pages);

I think that makes a lot more sense as an API. Although I think you want

int cma_bulk_alloc(struct cma *cma, unsigned order, unsigned nr_elem,

		struct page **pages);

Right, and I would start with a very simple implementation that does not mess with alloc_contig_range() (meaning: modify it).

I'd then much rather want to see simple tweaks to alloc_contig_range() to improve the situation. E.g., some kind of "fail fast" flag that let's the caller specify to skip some draining (or do it manually in cma before a bulk allocation) and rather fail fast than really trying to allocate the range whatever it costs.

There are multiple optimizations you can play with then (start with big granularity and split, move to smaller granularity on demand, etc., all nicely wrapped in cma_bulk_alloc()).

Quoted 6 lines · Matthew Wilcox

Quoted 5 lines · Minchan Kim

I understand pfn stuff in the API is not pretty but the concept of idea makes sense to me in that go though the *migratable area* and get possible order pages with hard effort. It looks like GFP_NORETRY version for kmem_cache_alloc_bulk.

Quoted 9 lines

Quoted 3 lines

How about this?

    int cma_alloc(struct cma *cma, int order, unsigned int nr_elem, struct page **pages);

I think that makes a lot more sense as an API. Although I think you want

int cma_bulk_alloc(struct cma *cma, unsigned order, unsigned nr_elem,

              struct page **pages);

Right, and I would start with a very simple implementation that does not mess with alloc_contig_range() (meaning: modify it).

I'd then much rather want to see simple tweaks to alloc_contig_range() to improve the situation. E.g., some kind of "fail fast" flag that let's the caller specify to skip some draining (or do it manually in cma before a bulk allocation) and rather fail fast than really trying to allocate the range whatever it costs.

There are multiple optimizations you can play with then (start with big granularity and split, move to smaller granularity on demand, etc., all nicely wrapped in cma_bulk_alloc()).

Yes, it might not end up as fast as this big hack (sorry) here, but as Nicholas correctly said, it's not our motivation to implement and maintain such complexity just to squeeze the last milliseconds out of an allocation path for "broken devices".

I absolutely dislike pushing this very specific allocation policy down to the core range allocator. It's already makes my head spin every time I look at it in detail.

--
Thanks,

David / dhildenb