Contiguous Memory Allocator

I am curious about one thing

Why do we need this allocator. Why not use allocate_resource and friends. The kernel generic resource handler already handles object alignment and subranges. It just seems to be a surplus allocator that could perhaps be mostly removed by using the kernel resource allocator we already have ?

I plan to replace it with lib/bitmap.c bitmap_* based allocator (similar like it it is used by dma_declare_coherent_memory() and friends in drivers/base/dma-coherent.c). We need something really simple for CMA area management.

IMHO allocate_resource and friends a bit too heavy here, but good to know that such allocator also exists.

Quoted 7 lines

I plan to replace it with lib/bitmap.c bitmap_* based allocator (similar like
it it is used by dma_declare_coherent_memory() and friends in drivers/base/dma-coherent.c). We need something really simple for CMA area
management.

Quoted 3 lines

IMHO allocate_resource and friends a bit too heavy here, but good to know
that such allocator also exists.

Not sure I'd class allocate_resource as heavyweight but providing it's using something that already exists rather than inventing yet another allocator.

Quoted 7 lines

I plan to replace it with lib/bitmap.c bitmap_* based allocator (similar like it it is used by dma_declare_coherent_memory() and friends in drivers/base/dma-coherent.c). We need something really simple for CMA area management.

IMHO allocate_resource and friends a bit too heavy here, but good to know that such allocator also exists.

Not sure I'd class allocate_resource as heavyweight but providing it's using something that already exists rather than inventing yet another allocator.

This wants dealing with before it goes upstream though so the chaneges in lib/*c etc never have to reach mainline and then get changed back.

The Contiguous Memory Allocator is a set of functions that lets one initialise a region of memory which then can be used to perform allocations of contiguous memory chunks from.

CMA allows for creation of separate contexts. Kernel is allowed to allocate movable pages within CMA's managed memory so that it can be used for page cache when CMA devices do not use it. On cm_alloc() request such pages are migrated out of CMA area to free required contiguous block.

+ * Driver usage
+ *

+ *   CMA should not be used directly by the device drivers. It should
+ *   be considered as helper framework for dma-mapping subsystm and
+ *   respective (platform)bus drivers.

+ *
+ * The CMA client needs to have a pointer to a CMA context
+ * represented by a struct cma (which is an opaque data type).
+ *
+ * Once such pointer is obtained, a caller may allocate contiguous
+ * memory chunk using the following function:
+ *
+ * cm_alloc()
+ *

+ *   This function returns a pointer to the first struct page which
+ *   represent a contiguous memory chunk.  This pointer
+ *   may be used with the following function:

+ *
+ * cm_free() -- frees allocated contiguous memory

+ * Platform/machine integration
+ *

+ *   CMA context must be created on platform or machine initialisation
+ *   and passed to respective subsystem that will be a client for CMA.
+ *   The latter may be done by a global variable or some filed in
+ *   struct device.  For the former CMA provides the following functions:
+ *
+ *     cma_init_migratetype()
+ *     cma_reserve()
+ *     cma_create()
+ *
+ *   The first one initialises a portion of reserved memory so that it
+ *   can be used with CMA.  The second first tries to reserve memory
+ *   (using memblock) and then initialise it.
+ *
+ *   The cma_reserve() function must be called when memblock is still
+ *   operational and reserving memory with it is still possible.  On
+ *   ARM platform the "reserve" machine callback is a perfect place to
+ *   call it.
+ *
+ *   The last function creates a CMA context on a range of previously
+ *   initialised memory addresses.  Because it uses kmalloc() it needs
+ *   to be called after SLAB is initialised.

+ */

Quoted 9 lines · Marek Szyprowski

The Contiguous Memory Allocator is a set of functions that lets one initialise a region of memory which then can be used to perform allocations of contiguous memory chunks from.

CMA allows for creation of separate contexts. Kernel is allowed to allocate movable pages within CMA's managed memory so that it can be used for page cache when CMA devices do not use it. On cm_alloc() request such pages are migrated out of CMA area to free required contiguous block.

Hi Marek,

I'm generally happy with the patches 1 through 7, i.e the heavy lifting to make contiguous allocations work. Thank you very much for keeping up the work and submitting these in a good shape.

I do think that we need to discuss the driver-visible API a bit more. My feeling is that this is rather un-Linux-like and it needs to be simplified some more. Of course, I don't mind being overruled by the memory management experts here, or if you can argue that it's really the right way to do it.

Quoted 14 lines

+ * Driver usage
+ *

+ *   CMA should not be used directly by the device drivers. It should
+ *   be considered as helper framework for dma-mapping subsystm and
+ *   respective (platform)bus drivers.

+ *
+ * The CMA client needs to have a pointer to a CMA context
+ * represented by a struct cma (which is an opaque data type).
+ *
+ * Once such pointer is obtained, a caller may allocate contiguous
+ * memory chunk using the following function:
+ *
+ * cm_alloc()
+ *

Quoted 5 lines

+ *   This function returns a pointer to the first struct page which
+ *   represent a contiguous memory chunk.  This pointer
+ *   may be used with the following function:

+ *
+ * cm_free() -- frees allocated contiguous memory

Please explain why you want a new top-level API here. I think it would be much nicer if a device driver could simply call alloc_pages(..., GFP_CMA) or similar, where all the complexity in here gets hidden behind a conditional deep inside of the page allocator.

Evidently, the two functions you describe here have an extra argument for the context. Can you elaborate why that is really needed? What is the specific requirement to have multiple such contexts in one system and what is the worst-case effect that you would get when the interface is simplified to have only one for all devices?

Alternatively, would it be possible to provide the cm_alloc/cm_free functions only as backing to the dma mapping API and not export them as a generic device driver interface?

Quoted 6 lines

+ * Platform/machine integration
+ *

+ *   CMA context must be created on platform or machine initialisation
+ *   and passed to respective subsystem that will be a client for CMA.
+ *   The latter may be done by a global variable or some filed in
+ *   struct device.  For the former CMA provides the following

functions:

Quoted 17 lines

+ *

+ *     cma_init_migratetype()
+ *     cma_reserve()
+ *     cma_create()
+ *
+ *   The first one initialises a portion of reserved memory so that it
+ *   can be used with CMA.  The second first tries to reserve memory
+ *   (using memblock) and then initialise it.
+ *
+ *   The cma_reserve() function must be called when memblock is still
+ *   operational and reserving memory with it is still possible.  On
+ *   ARM platform the "reserve" machine callback is a perfect place to
+ *   call it.
+ *
+ *   The last function creates a CMA context on a range of previously
+ *   initialised memory addresses.  Because it uses kmalloc() it needs
+ *   to be called after SLAB is initialised.

Quoted 1 line

+ */

This interface looks flawed to me for multiple reasons:

* It requires you to call three distinct functions in order to do one thing, and they all take the same arguments (more or less). Why not have one function call at the latest possible point where you can still change the memory attributes, and have everything else happen automatically?

* It requires you to pass the exact location of the area. I can see why you want that on certain machines that require DMA areas to be spread across multiple memory buses, but IMHO it's not appropriate for a generic API.

* It requires you to hardcode the size in a machine specific source file. This probably seems to be a natural thing to do when you have worked a lot on the ARM architecture, but it's really not. We really want to get rid of board files and replace them with generic probing based on the device tree, and the size of the area is more policy than a property of the hardware that can be accurately described in the device tree or a board file.

I'm sure that we can find a solution for all of these problems, it just takes a few good ideas. Especially for the last point, I don't have a better suggestion yet, but hopefully someone else can come up with one.

cm_alloc/free are definitely not meant to be called from device drivers. They should be only considered as a backend for dma-mapping.

'Raw' contiguous memory block doesn't really make sense for the device drivers. What the drivers require is a contiguous memory block that is somehow mapped into respective bus address space, so dma-mapping framework is the right interface.

alloc_pages(..., GFP_CMA) looks nice but in fact it is really impractical. The driver will need to map such buffer to dma context anyway, so imho dma_alloc_attributed() will give the drivers much more flexibility. In terms of dma-mapping the context argument isn't anything odd.

If possible I would like to make cma something similar to declare_dma_coherent()&friends, so the board/platform/bus startup code will just call declare_dma_contiguous() to enable support for cma for particular devices.

Quoted 7 lines

This interface looks flawed to me for multiple reasons:

* It requires you to call three distinct functions in order to do one thing, and they all take the same arguments (more or less). Why not have one function call at the latest possible point where you can still change the memory attributes, and have everything else happen automatically?

Initialization part will be definitely simplified, I must confess that I was in hurry to post the patches before the weekend and just forgot to cleanup this part...

Quoted 4 lines

* It requires you to pass the exact location of the area. I can see why you want that on certain machines that require DMA areas to be spread across multiple memory buses, but IMHO it's not appropriate for a generic API.

IMHO we can also use some NULL context to indicate some global, system wide CMA area and again -> in terms of dma-mapping api having a context isn't anything uncommon.

Quoted 7 lines

* It requires you to hardcode the size in a machine specific source file. This probably seems to be a natural thing to do when you have worked a lot on the ARM architecture, but it's really not. We really want to get rid of board files and replace them with generic probing based on the device tree, and the size of the area is more policy than a property of the hardware that can be accurately described in the device tree or a board file.

The problem is the fact that right now, we still have board files and we have to live with them for a while (with all advantages and disadvantages). I hope that you won't require me to rewrite the whole support for all ARM platforms to get rid of board files to get CMA merged ;)

I see no problem defining CMA areas in device tree, as this is something really specific to particular board configuration.

Please explain the exact requirements that lead you to defining multiple contexts.

Quoted 2 lines · Arnd Bergmann

Please explain the exact requirements that lead you to defining multiple contexts.

Some devices may have access only to some banks of memory. Some devices may use different banks of memory for different purposes.

Quoted 3 lines · Arnd Bergmann

Quoted 2 lines

Please explain the exact requirements that lead you to defining multiple contexts.

Quoted 2 lines · Michal Nazarewicz

Some devices may have access only to some banks of memory. Some devices may use different banks of memory for different purposes.

For all I know, that is something that is only true for a few very special Samsung devices,

I would suggest going forward without having multiple regions:

* Remove the registration of specific addresses from the initial patch set (but keep the patch).
* Add a heuristic plus command-line override to automatically come up with a reasonable location+size for *one* CMA area in the system.

Quoted 2 lines · Arnd Bergmann

For all I know, that is something that is only true for a few very special Samsung devices,

Maybe. I'm just answering your question. :)

Ah yes, I forgot that separate regions for different purposes could decrease fragmentation.

Quoted 1 line

I would suggest going forward without having multiple regions:

Is having support for multiple regions a bad thing? Frankly, removing this support will change code from reading context passed as argument to code reading context from global variable. Nothing is gained; functionality is lost.

Quoted 4 lines

Is having support for multiple regions a bad thing? Frankly, removing this support will change code from reading context passed as argument to code reading context from global variable. Nothing is gained; functionality is lost.

What is bad IMHO is making them the default, which forces the board code to care about memory management details. I would much prefer to have contiguous allocation parameters tuned automatically to just work on most boards before we add ways to do board-specific hacks.

Quoted 8 lines · On Tuesday 14 June 2011 18:58:35 Michal Nazarewicz wrote:

Quoted 2 lines · Arnd Bergmann

For all I know, that is something that is only true for a few very special Samsung devices,

Maybe. I'm just answering your question. :)

Ah yes, I forgot that separate regions for different purposes could decrease fragmentation.

That is indeed a good point, but having a good allocator algorithm could also solve this. I don't know too much about these allocation algorithms, but there are probably multiple working approaches to this.

Quoted 6 lines

Quoted 1 line

I would suggest going forward without having multiple regions:

Is having support for multiple regions a bad thing? Frankly, removing this support will change code from reading context passed as argument to code reading context from global variable. Nothing is gained; functionality is lost.

What is bad IMHO is making them the default, which forces the board code to care about memory management details. I would much prefer to have contiguous allocation parameters tuned automatically to just work on most boards before we add ways to do board-specific hacks.

I see your concerns, but I really wonder how to determine the properties of the global/default cma pool. You definitely don't want to give all available memory o CMA, because it will have negative impact on kernel operation (kernel really needs to allocate unmovable pages from time to time).

The only solution I see now is to provide Kconfig entry to determine the size of the global CMA pool, but this still have some issues, especially for multi-board kernels (each board probably will have different amount of RAM and different memory-consuming devices available). It looks that each board startup code still might need to tweak the size of CMA pool. I can add a kernel command line option for it, but such solution also will not solve all the cases (afair there was a discussion about kernel command line parameters for memory configuration and the conclusion was that it should be avoided).

Quoted 5 lines · On Wednesday 15 June 2011 09:11:39 Marek Szyprowski wrote:

I see your concerns, but I really wonder how to determine the properties of the global/default cma pool. You definitely don't want to give all available memory o CMA, because it will have negative impact on kernel operation (kernel really needs to allocate unmovable pages from time to time).

Exactly. This is a hard problem, so I would prefer to see a solution for coming up with reasonable defaults.

Quoted 9 lines

The only solution I see now is to provide Kconfig entry to determine the size of the global CMA pool, but this still have some issues, especially for multi-board kernels (each board probably will have different amount of RAM and different memory-consuming devices available). It looks that each board startup code still might need to tweak the size of CMA pool. I can add a kernel command line option for it, but such solution also will not solve all the cases (afair there was a discussion about kernel command line parameters for memory configuration and the conclusion was that it should be avoided).

The command line option can be a last resort if the heuristics fail, but it's not much better than a fixed Kconfig setting.

How about a Kconfig option that defines the percentage of memory to set aside for contiguous allocations?

Quoted 5 lines · On Wednesday 15 June 2011 09:11:39 Marek Szyprowski wrote:

I see your concerns, but I really wonder how to determine the properties of the global/default cma pool. You definitely don't want to give all available memory o CMA, because it will have negative impact on kernel operation (kernel really needs to allocate unmovable pages from time to time).

Exactly. This is a hard problem, so I would prefer to see a solution for coming up with reasonable defaults.

Quoted 10 lines

The only solution I see now is to provide Kconfig entry to determine the size of the global CMA pool, but this still have some issues, especially for multi-board kernels (each board probably will have different amount of RAM and different memory-consuming devices available). It looks that each board startup code still might need to tweak the size of CMA pool. I can add a kernel command line option for it, but such solution also will not solve all the cases (afair there was a discussion about kernel command line parameters for memory configuration and the conclusion was that it should be avoided).

The command line option can be a last resort if the heuristics fail, but it's not much better than a fixed Kconfig setting.

How about a Kconfig option that defines the percentage of memory to set aside for contiguous allocations?

Arnd

_______________________________________________
Linaro-mm-sig mailing list
Linaro-mm-sig at lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-mm-sig

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Wednesday 15 June 2011 09:11:39 Marek Szyprowski wrote:

I see your concerns, but I really wonder how to determine the properties of the global/default cma pool. You definitely don't want to give all available memory o CMA, because it will have negative impact on kernel operation (kernel really needs to allocate unmovable pages from time to time).

Quoted 2 lines

Exactly. This is a hard problem, so I would prefer to see a solution for coming up with reasonable defaults.

Is this a situation where passing the information from device tree might help? I know this does not help short term, but I am trying to understand the sorts of problems device tree can help solve.

Quoted 5 lines · On Wednesday 15 June 2011 09:11:39 Marek Szyprowski wrote:

I see your concerns, but I really wonder how to determine the properties of the global/default cma pool. You definitely don't want to give all available memory o CMA, because it will have negative impact on kernel operation (kernel really needs to allocate unmovable pages from time to time).

Exactly. This is a hard problem, so I would prefer to see a solution for coming up with reasonable defaults.

Quoted 9 lines

The only solution I see now is to provide Kconfig entry to determine the size of the global CMA pool, but this still have some issues, especially for multi-board kernels (each board probably will have different amount of RAM and different memory-consuming devices available). It looks that each board startup code still might need to tweak the size of CMA pool. I can add a kernel command line option for it, but such solution also will not solve all the cases (afair there was a discussion about kernel command line parameters for memory configuration and the conclusion was that it should be avoided).

The command line option can be a last resort if the heuristics fail, but it's not much better than a fixed Kconfig setting.

How about a Kconfig option that defines the percentage of memory to set aside for contiguous allocations?

What I was wondering about is how this patch series changes the allocation in case it can't allocate from the CMA pool. Will it attempt to fall back to a 'normal' allocation?

Quoted 2 lines

How about a Kconfig option that defines the percentage of memory to set aside for contiguous allocations?

I would actually like to see a cma_size kernel option of some sort. This would be for the global CMA pool only as I don't think we should try to do anything more complicated here.

Quoted 4 lines · Hans Verkuil

Quoted 2 lines

How about a Kconfig option that defines the percentage of memory to set aside for contiguous allocations?

I would actually like to see a cma_size kernel option of some sort. This

would

Quoted 1 line

be for the global CMA pool only as I don't think we should try to do

anything

Quoted 1 line

more complicated here.

A command line is probably good to override the compile-time default, yes.

We could also go further and add a runtime sysctl mechanism like the one for hugepages, where you can grow the pool at run time as long as there is enough free contiguous memory (e.g. from init scripts), or shrink it later if you want to allow larger nonmovable allocations.

My feeling is that we need to find a way to integrate the global settings for four kinds of allocations:

* nonmovable kernel pages
* hugetlb pages
* CMA
* memory hotplug

These essentially fight over the same memory (though things are slightly different with dynamic hugepages), and they all face the same basic problem of getting as much for themselves without starving the other three.

Sounds really good, but it might be really hard to implemnt, at least for CMA, because it needs to tweak parameters of memory management internal structures very early, when buddy allocator has not been activated yet.

Quoted 3 lines

These essentially fight over the same memory (though things are slightly different with dynamic hugepages), and they all face the same basic problem of getting as much for themselves without starving the other three.

I'm not sure we can solve all such issues in the first version. Maybe we should first have each of the above fully working in mainline separately and then start the integration works.

Why that? I would expect you can do the same that hugepages (used to) do and just attempt high-order allocations. If they succeed, you can add them as a CMA region and free them again, into the movable set of pages, otherwise you just fail the request from user space when the memory is already fragmented.

Quoted 4 lines

We could also go further and add a runtime sysctl mechanism like the one for hugepages, where you can grow the pool at run time as long as there is enough free contiguous memory (e.g. from init scripts), or shrink it later if you want to allow larger nonmovable allocations.

Sounds really good, but it might be really hard to implement, at least for CMA, because it needs to tweak parameters of memory management internal structures very early, when buddy allocator has not been activated yet.

Quoted 2 lines · On Tuesday 14 June 2011 18:58:35 Michal Nazarewicz wrote:

Ah yes, I forgot that separate regions for different purposes could decrease fragmentation.

That is indeed a good point, but having a good allocator algorithm could also solve this. I don't know too much about these allocation algorithms, but there are probably multiple working approaches to this.

Quoted 9 lines · Arnd Bergmann

Quoted 3 lines · On Tuesday 14 June 2011 18:58:35 Michal Nazarewicz wrote:

Ah yes, I forgot that separate regions for different purposes could decrease fragmentation.

That is indeed a good point, but having a good allocator algorithm could also solve this. I don't know too much about these allocation algorithms, but there are probably multiple working approaches to this.

imo no allocator algorithm is gonna help if you have comparably large, variable-sized contiguous allocations out of a restricted address range. It might work well enough if there are only a few sizes and/or there's decent headroom. But for really generic workloads this would require sync objects and eviction callbacks (i.e. what Thomas Hellstrom pushed with ttm).

Quoted 7 lines · Arnd Bergmann

Quoted 2 lines · On Tuesday 14 June 2011 18:58:35 Michal Nazarewicz wrote:

Ah yes, I forgot that separate regions for different purposes could decrease fragmentation.

That is indeed a good point, but having a good allocator algorithm could also solve this. I don't know too much about these allocation algorithms, but there are probably multiple working approaches to this.

imo no allocator algorithm is gonna help if you have comparably large, variable-sized contiguous allocations out of a restricted address range. It might work well enough if there are only a few sizes and/or there's decent headroom. But for really generic workloads this would require sync objects and eviction callbacks (i.e. what Thomas Hellstrom pushed with ttm).

So if this is only a requirement on very few platforms and can be cheaply fixed with multiple cma allocation areas (heck, we have slabs for the same reasons in the kernel), it might be a sensible compromise.

Quoted 6 lines · Michal Nazarewicz

Quoted 2 lines · Arnd Bergmann

Please explain the exact requirements that lead you to defining multiple contexts.

Some devices may have access only to some banks of memory. Some devices may use different banks of memory for different purposes.

For all I know, that is something that is only true for a few very special Samsung devices, and is completely unrelated of the need for contiguous allocations, so this approach becomes pointless as soon as the next generation of that chip grows an IOMMU, where we don't handle the special bank attributes. Also, the way I understood the situation for the Samsung SoC during the Budapest discussion, it's only a performance hack, not a functional requirement, unless you count '1080p playback' as a functional requirement.

Hi,

Quoted 16 lines · Arnd Bergmann

Quoted 6 lines · Michal Nazarewicz

Quoted 2 lines · Arnd Bergmann

Please explain the exact requirements that lead you to defining multiple contexts.

Some devices may have access only to some banks of memory. ?Some devices may use different banks of memory for different purposes.

For all I know, that is something that is only true for a few very special Samsung devices, and is completely unrelated of the need for contiguous allocations, so this approach becomes pointless as soon as the next generation of that chip grows an IOMMU, where we don't handle the special bank attributes. Also, the way I understood the situation for the Samsung SoC during the Budapest discussion, it's only a performance hack, not a functional requirement, unless you count '1080p playback' as a functional requirement.

Hm, I think that was something similar but not quite the same: talking about having allocations split to lie between two banks of RAM to maximise the read/write speed for performance reasons. ?That's something that can be handled in the allocator, rather than an API constraint, as this is.

Not that I know of any hardware which is limited as such, but eh.

Cheers,
Daniel

_______________________________________________
Linaro-mm-sig mailing list
Linaro-mm-sig at lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-mm-sig

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Thanks for the insight, that's a much better argument than 'something may need it'. Are those all chips without an IOMMU or do we also need to solve the IOMMU case with split bank allocation?

Quoted 9 lines · Arnd Bergmann

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Quoted 3 lines

Thanks for the insight, that's a much better argument than 'something may need it'. Are those all chips without an IOMMU or do we also need to solve the IOMMU case with split bank allocation?

Yes. The IOMMU case with split bank allocation is key, especially for shared buffers. Consider the case where video is using a certain bank for performance purposes and that frame is shared with the GPU.

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Thanks for the insight, that's a much better argument than 'something may need it'. Are those all chips without an IOMMU or do we also need to solve the IOMMU case with split bank allocation?

I think I'd still prefer to see the support for multiple regions split out into one of the later patches, especially since that would defer the question of how to do the initialization for this case and make sure we first get a generic way.

You've convinced me that we need to solve the problem of allocating memory from a specific bank eventually, but separating it from the one at hand (contiguous allocation) should help getting the important groundwork in at first.

The possible conflict that I still see with per-bank CMA regions are:

* It completely destroys memory power management in cases where that is based on powering down entire memory banks.

* We still need to solve the same problem in case of IOMMU mappings at some point, even if today's hardware doesn't have this combination. It would be good to use the same solution for both.

Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Thanks for the insight, that's a much better argument than 'something may need it'. Are those all chips without an IOMMU or do we also need to solve the IOMMU case with split bank allocation?

I think I'd still prefer to see the support for multiple regions split out into one of the later patches, especially since that would defer the question of how to do the initialization for this case and make sure we first get a generic way.

You've convinced me that we need to solve the problem of allocating memory from a specific bank eventually, but separating it from the one at hand (contiguous allocation) should help getting the important groundwork in at first.

The possible conflict that I still see with per-bank CMA regions are:

* It completely destroys memory power management in cases where that is based on powering down entire memory banks.

Hello,

Quoted 8 lines · Arnd Bergmann

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Quoted 17 lines

Thanks for the insight, that's a much better argument than 'something may need it'. Are those all chips without an IOMMU or do we also need to solve the IOMMU case with split bank allocation?

I think I'd still prefer to see the support for multiple regions split out into one of the later patches, especially since that would defer the question of how to do the initialization for this case and make sure we first get a generic way.

You've convinced me that we need to solve the problem of allocating memory from a specific bank eventually, but separating it from the one at hand (contiguous allocation) should help getting the important groundwork in at first.

The possible conflict that I still see with per-bank CMA regions are:

Quoted 2 lines

* It completely destroys memory power management in cases where that is based on powering down entire memory banks.

I don't think that per-bank CMA regions destroys memory power management more than the global CMA pool. Please note that the contiguous buffers (or in general dma-buffers) right now are unmovable so they don't fit well into memory power management.

Best regards

Signature · 9 lines

Marek Szyprowski
Samsung Poland R&D Center



_______________________________________________
Linaro-mm-sig mailing list
Linaro-mm-sig at lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-mm-sig

Quoted 10 lines · Marek Szyprowski

Quoted 8 lines · Arnd Bergmann

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

We at Qualcomm have some platforms that have memory of different performance characteristics, some drivers will need a way of specifying that they need fast memory for an allocation (and would prefer an error if it is not available rather than a fallback to slower memory). It would also be bad if allocators who don't need fast memory got it "accidentally", depriving those who really need it.

Quoted 9 lines

Quoted 4 lines

The possible conflict that I still see with per-bank CMA regions are:

* It completely destroys memory power management in cases where that is based on powering down entire memory banks.

I don't think that per-bank CMA regions destroys memory power management more than the global CMA pool. Please note that the contiguous buffers (or in general dma-buffers) right now are unmovable so they don't fit well into memory power management.

We also have platforms where a well-defined part of the memory can be powered off, and other parts can't (or won't). We need a way to steer the place allocations come from to the memory that won't be turned off (so that CMA allocations are not an obstacle to memory hotremove).

Quoted 12 lines · On Wednesday 15 June 2011 23:39:58 Larry Bassel wrote:

Quoted 10 lines · Marek Szyprowski

Quoted 8 lines · Arnd Bergmann

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Quoted 6 lines

We at Qualcomm have some platforms that have memory of different performance characteristics, some drivers will need a way of specifying that they need fast memory for an allocation (and would prefer an error if it is not available rather than a fallback to slower memory). It would also be bad if allocators who don't need fast memory got it "accidentally", depriving those who really need it.

Can you describe how the memory areas differ specifically? Is there one that is always faster but very small, or are there just specific circumstances under which some memory is faster than another?

Quoted 4 lines

Can you describe how the memory areas differ specifically? Is there one that is always faster but very small, or are there just specific circumstances under which some memory is faster than another?

One is always faster, but very small (generally 2-10% the size of "normal" memory).

Quoted 12 lines · On Wednesday 15 June 2011 23:39:58 Larry Bassel wrote:

Quoted 10 lines · Marek Szyprowski

Quoted 8 lines · Arnd Bergmann

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

Quoted 6 lines

We at Qualcomm have some platforms that have memory of different performance characteristics, some drivers will need a way of specifying that they need fast memory for an allocation (and would prefer an error if it is not available rather than a fallback to slower memory). It would also be bad if allocators who don't need fast memory got it "accidentally", depriving those who really need it.

Can you describe how the memory areas differ specifically? Is there one that is always faster but very small, or are there just specific circumstances under which some memory is faster than another?

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

The possible conflict that I still see with per-bank CMA regions are:

* It completely destroys memory power management in cases where that is based on powering down entire memory banks.

We already established that we have to know something about the banks, and your additional input makes it even clearer that we need to consider the bigger picture here: We need to describe parts of memory separately regarding general performance, device specific allocations and hotplug characteristics.

It still sounds to me that this can be done using the NUMA properties that Linux already understands, and teaching more subsystems about it, but maybe the memory hotplug developers have already come up with another scheme. The way that memory hotplug and CMA choose their memory regions certainly needs to take both into account. As far as I can see there are both conflicting and synergistic effects when you combine the two.

Quoted 8 lines

It still sounds to me that this can be done using the NUMA properties that Linux already understands, and teaching more subsystems about it, but maybe the memory hotplug developers have already come up with another scheme. The way that memory hotplug and CMA choose their memory regions certainly needs to take both into account. As far as I can see there are both conflicting and synergistic effects when you combine the two.

Recently, we proposed a generic 'memory regions' framework to exploit the memory power management capabilities on the embedded boards. Think of some of the above CMA requirements could be met by this fraemwork. One of the main goals of regions is to make the VM aware of the hardware memory boundaries, like bank. For managing memory power consumption, memory regions are created aligned to the hardware granularity at which the power can be managed (ie, the memory power consumption operations like on/off can be performed). If attributed are associated with each of these regions, some of these regions could be marked as CMA-only, ensuring that only movable and per-bank memory is allocated. More details on the design can be found here:

http://lkml.org/lkml/2011/5/27/177
http://lkml.org/lkml/2011/6/29/202
http://lwn.net/Articles/446493/

Quoted 12 lines · Marek Szyprowski

Hello,

Quoted 8 lines · Arnd Bergmann

Quoted 7 lines · On Tuesday 14 June 2011 20:58:25 Zach Pfeffer wrote:

I've seen this split bank allocation in Qualcomm and TI SoCs, with Samsung, that makes 3 major SoC vendors (I would be surprised if Nvidia didn't also need to do this) - so I think some configurable method to control allocations is necessarily. The chips can't do decode without it (and by can't do I mean 1080P and higher decode is not functionally useful). Far from special, this would appear to be the default.

We at Qualcomm have some platforms that have memory of different performance characteristics, some drivers will need a way of specifying that they need fast memory for an allocation (and would prefer an error if it is not available rather than a fallback to slower memory). It would also be bad if allocators who don't need fast memory got it "accidentally", depriving those who really need it.

Quoted 17 lines

Quoted 17 lines

Thanks for the insight, that's a much better argument than 'something may need it'. Are those all chips without an IOMMU or do we also need to solve the IOMMU case with split bank allocation?

I think I'd still prefer to see the support for multiple regions split out into one of the later patches, especially since that would defer the question of how to do the initialization for this case and make sure we first get a generic way.

You've convinced me that we need to solve the problem of allocating memory from a specific bank eventually, but separating it from the one at hand (contiguous allocation) should help getting the important groundwork in at first.

The possible conflict that I still see with per-bank CMA regions are:

Quoted 7 lines

Quoted 2 lines

* It completely destroys memory power management in cases where that ? is based on powering down entire memory banks.

I don't think that per-bank CMA regions destroys memory power management more than the global CMA pool. Please note that the contiguous buffers (or in general dma-buffers) right now are unmovable so they don't fit well into memory power management.

We also have platforms where a well-defined part of the memory can be powered off, and other parts can't (or won't). We need a way to steer the place allocations come from to the memory that won't be turned off (so that CMA allocations are not an obstacle to memory hotremove).

Quoted 8 lines

Best regards
--
Marek Szyprowski
Samsung Poland R&D Center

Quoted 4 lines

_______________________________________________
Linaro-mm-sig mailing list
Linaro-mm-sig at lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-mm-sig

Larry Bassel

--
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

* We still need to solve the same problem in case of IOMMU mappings at some point, even if today's hardware doesn't have this combination. It would be good to use the same solution for both.

Quoted 3 lines · Arnd Bergmann

* We still need to solve the same problem in case of IOMMU mappings at some point, even if today's hardware doesn't have this combination. It would be good to use the same solution for both.

I don't think I follow. What does IOMMU has to do with CMA?

The point is that on the higher level device drivers, we want to hide the presence of CMA and/or IOMMU behind the dma mapping API, but the device drivers do need to know about the bank properties.

Quoted 6 lines · Michal Nazarewicz

Quoted 2 lines · Arnd Bergmann

Please explain the exact requirements that lead you to defining multiple contexts.

Some devices may have access only to some banks of memory. Some devices may use different banks of memory for different purposes.

For all I know, that is something that is only true for a few very special Samsung devices, and is completely unrelated of the need for contiguous allocations, so this approach becomes pointless as soon as the next generation of that chip grows an IOMMU, where we don't handle the special bank attributes. Also, the way I understood the situation for the Samsung SoC during the Budapest discussion, it's only a performance hack, not a functional requirement, unless you count '1080p playback' as a functional requirement.

Supporting contiguous allocation is a very useful goal and many people want this, but supporting a crazy one-off hardware design with lots of generic infrastructure is going a bit too far. If you can't be more specific than 'some devices may need this', I would suggest going forward without having multiple regions:

* Remove the registration of specific addresses from the initial patch set (but keep the patch).
* Add a heuristic plus command-line override to automatically come up with a reasonable location+size for *one* CMA area in the system.
* Ship the patch to add support for multiple CMA areas with the BSP for the boards that need it (if any).
* Wait for someone on a non-Samsung SoC to run into the same problem, then have /them/ get the final patch in.

Even if you think you can convince enough people that having support for distinct predefined regions is a good idea, I would recommend splitting that out of the initial merge so we can have that discussion separately from the other issues.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 8 lines · Marek Szyprowski

cm_alloc/free are definitely not meant to be called from device drivers. They should be only considered as a backend for dma-mapping.

'Raw' contiguous memory block doesn't really make sense for the device drivers. What the drivers require is a contiguous memory block that is somehow mapped into respective bus address space, so dma-mapping framework is the right interface.

Quoted 4 lines

alloc_pages(..., GFP_CMA) looks nice but in fact it is really impractical. The driver will need to map such buffer to dma context anyway, so imho dma_alloc_attributed() will give the drivers much more flexibility. In terms of dma-mapping the context argument isn't anything odd.

Ok.

Quoted 4 lines

If possible I would like to make cma something similar to declare_dma_coherent()&friends, so the board/platform/bus startup code will just call declare_dma_contiguous() to enable support for cma for particular devices.

Sounds good, I like that.

Quoted 8 lines

Quoted 4 lines

* It requires you to pass the exact location of the area. I can see why you want that on certain machines that require DMA areas to be spread across multiple memory buses, but IMHO it's not appropriate for a generic API.

IMHO we can also use some NULL context to indicate some global, system wide CMA area and again -> in terms of dma-mapping api having a context isn't anything uncommon.

Please explain the exact requirements that lead you to defining multiple contexts. My feeling is that in most cases we don't need them and can simply live with a single area. Depending on how obscure the cases are where we do need something beyond that, we can then come up with special-case solutions for them.

Quoted 1 line

Quoted 1 line

* It requires you to hardcode the size in a machine specific source

file.

Quoted 1 line

Quoted 1 line

This probably seems to be a natural thing to do when you have worked

a

Quoted 3 lines

Quoted 3 lines

lot on the ARM architecture, but it's really not. We really want to get rid of board files and replace them with generic probing based on the device tree, and the size of the area is more policy than a

property

Quoted 1 line

Quoted 1 line

of the hardware that can be accurately described in the device tree

or

Quoted 4 lines

Quoted 1 line

a board file.

The problem is the fact that right now, we still have board files and we have to live with them for a while (with all advantages and

disadvantages).

Quoted 2 lines

I hope that you won't require me to rewrite the whole support for all ARM platforms to get rid of board files to get CMA merged ;)

Of course not. But we need to know what we want a platform with device tree support to look like when it's using CMA, so we don't make it harder to change the platforms over than it already is.

Quoted 2 lines

I see no problem defining CMA areas in device tree, as this is something really specific to particular board configuration.

The problem with defining CMA areas in the device tree is that it's not a property of the hardware, but really policy. The device tree file should not contain anything related to a specific operating system because you might want to boot something on the board that doesn't know about CMA, and even when you only care about using Linux, the implementation might change to the point where hardcoded CMA areas no longer make sense.

IMHO we should first aim for a solution that works almost everywhere without the kernel knowing what board it's running on, and then we can add quirks for devices that have special requirements. I think the situation is similar to the vmalloc virtual address space, which normally has a hardcoded size that works almost everywhere, but there are certain drivers etc that require much more, or there are situations where you want to make it smaller in order to avoid highmem.

Quoted 6 lines

Quoted 4 lines

If possible I would like to make cma something similar to declare_dma_coherent()&friends, so the board/platform/bus startup code will just call declare_dma_contiguous() to enable support for cma for particular devices.

Sounds good, I like that.

Thanks. I thought a bit more on this and decided that I want to make this declare_dma_contiguous() optional for the drivers. It should be used only for some sophisticated cases like for example our video codec with two memory interfaces for 2 separate banks. By default the dma-mapping will use system-wide cma pool.

(snipped)

Quoted 5 lines

Please explain the exact requirements that lead you to defining multiple contexts. My feeling is that in most cases we don't need them and can simply live with a single area. Depending on how obscure the cases are where we do need something beyond that, we can then come up with special-case solutions for them.

Like it was already stated we need such feature for our multimedia codec to allocate buffers from different memory banks. I really don't see any problems with the possibility to have additional cma areas for special purposes.

Quoted 7 lines

The problem with defining CMA areas in the device tree is that it's not a property of the hardware, but really policy. The device tree file should not contain anything related to a specific operating system because you might want to boot something on the board that doesn't know about CMA, and even when you only care about using Linux, the implementation might change to the point where hardcoded CMA areas no longer make sense.

I really doubt that the device tree will carry only system-independent information.

Anyway, the preferred or required memory areas/banks for buffer allocation is something that is a property of the hardware not the OS policy.

Quoted 7 lines

IMHO we should first aim for a solution that works almost everywhere without the kernel knowing what board it's running on, and then we can add quirks for devices that have special requirements. I think the situation is similar to the vmalloc virtual address space, which normally has a hardcoded size that works almost everywhere, but there are certain drivers etc that require much more, or there are situations where you want to make it smaller in order to avoid highmem.

I'm trying to create something that will fulfill the requirements of my hardware, that's why I cannot focus on a generic case only.