Skip to contentSkip to conversation

The Contiguous Memory Allocator

PATCH Superseded Michal Nazarewicz 6 participants 19 messages Jul 26, 2010 – Aug 6, 2010

Subsystems (9)

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v2 cover The Contiguous Memory Allocator

Hello everyone,

The following patchset implements a Contiguous Memory Allocator. For those who have not yet stumbled across CMA an excerpt from documentation:

The Contiguous Memory Allocator (CMA) is a framework, which allows setting up a machine-specific configuration for physically-contiguous memory management. Memory for devices is then allocated according to that configuration.

The main role of the framework is not to allocate memory, but to parse and manage memory configurations, as well as to act as an in-between between device drivers and pluggable allocators. It is thus not tied to any memory allocation method or strategy.

For more information please refer to the second patch from the patchset which contains the documentation.

This is the second version of the patchset. All of the changes are concentrated in the second patch -- the other patches are almost identical.

Major observable changes are:

1. The "cma_map" command line have been removed. In exchange, a SysFS entry has been created under kernel/mm/contiguous.

The configuration strings passed to CMA are now called attributes in the documentation.

The intended way of specifying the attributes is
a cma_set_defaults() function called by platform initialisation code. "regions" attribute (the string specified by "cma" command line parameter) can be overwritten with command line parameter; the other attributes can be changed during run-time using the SysFS entries.

(I still believe that the other attributes should have their own command line arguments as well but since they posed a lot of controversy (and many stopped reading after encountering them) "cma_map" have been removed.)

2. The behaviour of the "map" attribute has been modified slightly. Currently, if no rule matches given device it is assigned regions specified by the "asterisk" attribute. It is by default built from the region names given in "regions" attribute.

This also means that if no "map" is specified all devices use all the regions specified in the "regions" attribute. This should be a handy default.

3. Devices can register private regions as well as regions that can be shared but are not reserved using standard CMA mechanisms. A private region has no name and can be accessed only by devices that have the pointer to it.

Moreover, if device manages to run its code early enough it can register an "early region". An early region is one memory has not been reserved for. At one point, platform initialisation code reserves memory for all registered early regions and if this succeeds those regions are registered as normal regions that can be used with the standard API. This may be handy for devices that need some private region but don't want to worry about reserving it.

4. The way allocators are registered has changed. Currently, a cma_allocator_register() function is used for that purpose. Moreover, allocators are attached to regions the first time memory is registered from the region or when allocator is registered which means that allocators can be dynamic modules that are loaded after the kernel booted (of course, it won't be possible to allocate a chunk of memory from a region if allocator is not loaded).

Index of new functions:

+static inline dma_addr_t __must_check
+cma_alloc_from(const char *regions, size_t size, dma_addr_t alignment)

+static inline int
+cma_info_about(struct cma_info *info, const const char *regions)

+int __must_check cma_region_register(struct cma_region *reg);

+dma_addr_t __must_check
+cma_alloc_from_region(struct cma_region *reg,
+ size_t size, dma_addr_t alignment);

+static inline dma_addr_t __must_check
+cma_alloc_from(const char *regions,
+ size_t size, dma_addr_t alignment);

+int cma_allocator_register(struct cma_allocator *alloc);

The patches in the patchset include:

Michal Nazarewicz (4):
lib: rbtree: rb_root_init() function added

The rb_root_init() function initialises an RB tree with a single node placed in the root. This is more convenient then initialising an empty tree and then adding an element.

mm: cma: Contiguous Memory Allocator added

This patch is the main patchset that implements the CMA framework including the best-fit allocator. It also adds a documentation.

mm: cma: Test device and application added

This patch adds a misc device that works as a proxy to the CMA framework and a simple testing application. This lets one test the whole framework from user space as well as reply an recorded allocate/free sequence.

arm: Added CMA to Aquila and Goni

This patch adds the CMA platform initialisation code to two ARM platforms. It serves as an example of how this is achieved.

 Documentation/00-INDEX                             |    2 +
 .../ABI/testing/sysfs-kernel-mm-contiguous         |    9 +
 Documentation/contiguous-memory.txt                |  646 +++++++++++
 Documentation/kernel-parameters.txt                |    4 +
 arch/arm/mach-s5pv210/mach-aquila.c                |   13 +
 arch/arm/mach-s5pv210/mach-goni.c                  |   13 +
 drivers/misc/Kconfig                               |    8 +
 drivers/misc/Makefile                              |    1 +
 drivers/misc/cma-dev.c                             |  184 +++
 include/linux/cma.h                                |  475 ++++++++
 include/linux/rbtree.h                             |   11 +
 mm/Kconfig                                         |   34 +
 mm/Makefile                                        |    3 +
 mm/cma-best-fit.c                                  |  407 +++++++
 mm/cma.c                                           | 1170 ++++++++++++++++++++
 tools/cma/cma-test.c                               |  373 +++++++
 16 files changed, 3353 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-contiguous
 create mode 100644 Documentation/contiguous-memory.txt
 create mode 100644 drivers/misc/cma-dev.c
 create mode 100644 include/linux/cma.h
 create mode 100644 mm/cma-best-fit.c
 create mode 100644 mm/cma.c
 create mode 100644 tools/cma/cma-test.c
#1 lore raw

PATCH v2 1/4 lib: rbtree: rb_root_init() function added

Added a rb_root_init() function which initialises a rb_root structure as a red-black tree with at most one element. The rationale is that using rb_root_init(root, node) is more straightforward and cleaner then first initialising and empty tree followed by an insert operation.

---
 include/linux/rbtree.h |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
include/linux/rbtree.h +11 -0
index 7066acb..5b6dc66 100644
@@ -130,6 +130,17 @@ static inline void rb_set_color(struct rb_node *rb, int color)
 }
 
 #define RB_ROOT	(struct rb_root) { NULL, }
+
+static inline void rb_root_init(struct rb_root *root, struct rb_node *node)
+{
+	root->rb_node = node;
+	if (node) {
+		node->rb_parent_color = RB_BLACK; /* black, no parent */
+		node->rb_left  = NULL;
+		node->rb_right = NULL;
+	}
+}
+
 #define	rb_entry(ptr, type, member) container_of(ptr, type, member)
 
 #define RB_EMPTY_ROOT(root)	((root)->rb_node == NULL)
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

PATCH v2 2/4 mm: cma: Contiguous Memory Allocator added

The Contiguous Memory Allocator framework is a set of APIs for allocating physically contiguous chunks of memory.

Various chips require contiguous blocks of memory to operate. Those chips include devices such as cameras, hardware video decoders and encoders, etc.

The code is highly modular and customisable to suit the needs of various users. Set of regions reserved for CMA can be configured on run-time and it is easy to add custom allocator algorithms if one has such need.

For more details see Documentation/contiguous-memory.txt.

---
 Documentation/00-INDEX                             |    2 +
 .../ABI/testing/sysfs-kernel-mm-contiguous         |    9 +
 Documentation/contiguous-memory.txt                |  646 +++++++++++
 Documentation/kernel-parameters.txt                |    4 +
 include/linux/cma.h                                |  445 ++++++++
 mm/Kconfig                                         |   34 +
 mm/Makefile                                        |    3 +
 mm/cma-best-fit.c                                  |  407 +++++++
 mm/cma.c                                           | 1170 ++++++++++++++++++++
 9 files changed, 2720 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-contiguous
 create mode 100644 Documentation/contiguous-memory.txt
 create mode 100644 include/linux/cma.h
 create mode 100644 mm/cma-best-fit.c
 create mode 100644 mm/cma.c
Documentation/00-INDEX +2 -0
index 5405f7a..bb50209 100644
@@ -94,6 +94,8 @@ connector/
 	- docs on the netlink based userspace<->kernel space communication mod.
 console/
 	- documentation on Linux console drivers.
+contiguous-memory.txt
+	- documentation on physically-contiguous memory allocation framework.
 cpu-freq/
 	- info on CPU frequency and voltage scaling.
 cpu-hotplug.txt
Documentation/ABI/testing/sysfs-kernel-mm-contiguous new file +9 -0
new file mode 100644
index 0000000..05e2f6a
@@ -0,0 +1,9 @@
+What:		/sys/kernel/mm/contiguous/
+Date:		July 2008
+Contact:	Michal Nazarewicz <[email protected]>
+Description:
+		/sys/kernel/mm/contiguous/ contains two files: asterisk and
+		map.  They are used to configure the Contiguous Memory
+		Allocator framework.
+
+		For details see Documentation/contiguous-memory.txt.
Documentation/contiguous-memory.txt new file +646 -0

646 lines not on this page · raw message

Documentation/kernel-parameters.txt +4 -0
index a698255..5c7af89 100644
@@ -43,6 +43,7 @@ parameter is applicable:
 	AVR32	AVR32 architecture is enabled.
 	AX25	Appropriate AX.25 support is enabled.
 	BLACKFIN Blackfin architecture is enabled.
+	CMA	Contiguous Memory Allocator is enabled.
 	DRM	Direct Rendering Management support is enabled.
 	EDD	BIOS Enhanced Disk Drive Services (EDD) is enabled
 	EFI	EFI Partitioning (GPT) is enabled
@@ -476,6 +477,9 @@ and is between 256 and 4096 characters. It is defined in the file
 			Also note the kernel might malfunction if you disable
 			some critical bits.
 
+	cma=		[CMA] List of CMA regions.
+			See Documentation/contiguous-memory.txt for details.
+
 	cmo_free_hint=	[PPC] Format: { yes | no }
 			Specify whether pages are marked as being inactive
 			when they are freed.  This is used in CMO environments
include/linux/cma.h new file +445 -0

445 lines not on this page · raw message

mm/Kconfig +34 -0
index f4e516e..468b874 100644
@@ -301,3 +301,37 @@ config NOMMU_INITIAL_TRIM_EXCESS
 	  of 1 says that all excess pages should be trimmed.
 
 	  See Documentation/nommu-mmap.txt for more information.
+
+
+config CMA
+	bool "Contiguous Memory Allocator framework"
+	# Currently there is only one allocator so force it on
+	select CMA_BEST_FIT
+	help
+	  This enables the Contiguous Memory Allocator framework which
+	  allows drivers to allocate big physically-contiguous blocks of
+	  memory for use with hardware components that do not support I/O
+	  map nor scatter-gather.
+
+	  If you select this option you will also have to select at least
+	  one allocator algorithm below.
+
+	  To make use of CMA you need to specify the regions and
+	  driver->region mapping on command line when booting the kernel.
+
+config CMA_DEBUG
+	bool "CMA debug messages"
+	depends on CMA
+	help
+	  Enable debug messages in CMA code.
+
+config CMA_BEST_FIT
+	bool "CMA best-fit allocator"
+	depends on CMA
+	default y
+	help
+	  This is a best-fit algorithm running in O(n log n) time where
+	  n is the number of existing holes (which is never greater then
+	  the number of allocated regions and usually much smaller).  It
+	  allocates area from the smallest hole that is big enough for
+	  allocation in question.
mm/Makefile +3 -0
index 34b2546..54b0e99 100644
@@ -47,3 +47,6 @@ obj-$(CONFIG_MEMORY_FAILURE) += memory-failure.o
 obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o
 obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
 obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
+
+obj-$(CONFIG_CMA) += cma.o
+obj-$(CONFIG_CMA_BEST_FIT) += cma-best-fit.o
mm/cma-best-fit.c new file +407 -0

407 lines not on this page · raw message

mm/cma.c new file +1170 -0

1,170 lines not on this page · raw message

Signature · 1 line
1.7.1
#3 lore raw

PATCH v2 3/4 mm: cma: Test device and application added

This patch adds a "cma" misc device which lets user space use the CMA API. This device is meant for testing. A testing application is also provided.

---
 drivers/misc/Kconfig   |    8 +
 drivers/misc/Makefile  |    1 +
 drivers/misc/cma-dev.c |  184 ++++++++++++++++++++++++
 include/linux/cma.h    |   30 ++++
 tools/cma/cma-test.c   |  373 ++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 596 insertions(+), 0 deletions(-)
 create mode 100644 drivers/misc/cma-dev.c
 create mode 100644 tools/cma/cma-test.c
drivers/misc/Kconfig +8 -0
index 9b089df..6ae3d9f 100644
@@ -368,4 +368,12 @@ source "drivers/misc/eeprom/Kconfig"
 source "drivers/misc/cb710/Kconfig"
 source "drivers/misc/iwmc3200top/Kconfig"
 
+config CMA_DEVICE
+	tristate "CMA misc device (DEVELOPEMENT)"
+	depends on CMA
+	help
+	  The CMA misc device allows allocating contiguous memory areas
+	  from user space.  This is mostly for testing of the CMA
+	  framework.
+
 endif # MISC_DEVICES
drivers/misc/Makefile +1 -0
index 67552d6..9921370 100644
@@ -32,3 +32,4 @@ obj-y				+= eeprom/
 obj-y				+= cb710/
 obj-$(CONFIG_VMWARE_BALLOON)	+= vmware_balloon.o
 obj-$(CONFIG_ARM_CHARLCD)	+= arm-charlcd.o
+obj-$(CONFIG_CMA_DEVICE)	+= cma-dev.o
drivers/misc/cma-dev.c new file +184 -0

184 lines not on this page · raw message

include/linux/cma.h +30 -0
index 96a8652..36e73fb 100644
@@ -11,6 +11,36 @@
  * See Documentation/contiguous-memory.txt for details.
  */
 
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+
+#define CMA_MAGIC (('c' << 24) | ('M' << 16) | ('a' << 8) | 0x42)
+
+/**
+ * An information about area exportable to user space.
+ * @magic: must always be CMA_MAGIC.
+ * @name:  name of the device to allocate as.
+ * @kind:  kind of the memory.
+ * @_pad:  reserved.
+ * @size:  size of the chunk to allocate.
+ * @alignment: desired alignment of the chunk (must be power of two or zero).
+ * @start: when ioctl() finishes this stores physical address of the chunk.
+ */
+struct cma_alloc_request {
+	__u32 magic;
+	char  name[17];
+	char  kind[17];
+	__u16 pad;
+	/* __u64 to be compatible accross 32 and 64 bit systems. */
+	__u64 size;
+	__u64 alignment;
+	__u64 start;
+};
+
+#define IOCTL_CMA_ALLOC    _IOWR('p', 0, struct cma_alloc_request)
+
+
 /***************************** Kernel lever API *****************************/
 
 #ifdef __KERNEL__
tools/cma/cma-test.c new file +373 -0

373 lines not on this page · raw message

Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw

PATCH v2 4/4 arm: Added CMA to Aquila and Goni

Added the CMA initialisation code to two Samsung platforms.

---
 arch/arm/mach-s5pv210/mach-aquila.c |   13 +++++++++++++
 arch/arm/mach-s5pv210/mach-goni.c   |   13 +++++++++++++
 2 files changed, 26 insertions(+), 0 deletions(-)
arch/arm/mach-s5pv210/mach-aquila.c +13 -0
index 0992618..ab156f9 100644
@@ -19,6 +19,7 @@
 #include <linux/gpio_keys.h>
 #include <linux/input.h>
 #include <linux/gpio.h>
+#include <linux/cma.h>
 
 #include <asm/mach/arch.h>
 #include <asm/mach/map.h>
@@ -454,6 +455,17 @@ static void __init aquila_map_io(void)
 	s3c24xx_init_uarts(aquila_uartcfgs, ARRAY_SIZE(aquila_uartcfgs));
 }
 
+static void __init aquila_reserve(void)
+{
+	static char regions[] __initdata =
+		"-mfc_fw=1M/128K;mfc_b1=32M;mfc_b2=16M@0x40000000";
+	static char map[] __initdata =
+		"s3c-mfc5/f=mfc_fw;s3c-mfc5/a=mfc_b1;s3c-mfc5/b=mfc_b2";
+
+	cma_set_defaults(regions, map, NULL);
+	cma_early_regions_reserve(NULL);
+}
+
 static void __init aquila_machine_init(void)
 {
 	/* PMIC */
@@ -478,4 +490,5 @@ MACHINE_START(AQUILA, "Aquila")
 	.map_io		= aquila_map_io,
 	.init_machine	= aquila_machine_init,
 	.timer		= &s3c24xx_timer,
+	.reserve	= aquila_reserve,
 MACHINE_END
arch/arm/mach-s5pv210/mach-goni.c +13 -0
index 7b18505..2b0a349 100644
@@ -19,6 +19,7 @@
 #include <linux/gpio_keys.h>
 #include <linux/input.h>
 #include <linux/gpio.h>
+#include <linux/cma.h>
 
 #include <asm/mach/arch.h>
 #include <asm/mach/map.h>
@@ -435,6 +436,17 @@ static void __init goni_map_io(void)
 	s3c24xx_init_uarts(goni_uartcfgs, ARRAY_SIZE(goni_uartcfgs));
 }
 
+static void __init goni_reserve(void)
+{
+	static char regions[] __initdata =
+		"-mfc_fw=1M/128K;mfc_b1=32M;mfc_b2=16M@0x40000000";
+	static char map[] __initdata =
+		"s3c-mfc5/f=mfc_fw;s3c-mfc5/a=mfc_b1;s3c-mfc5/b=mfc_b2";
+
+	cma_set_defaults(regions, map, NULL);
+	cma_early_regions_reserve(NULL);
+}
+
 static void __init goni_machine_init(void)
 {
 	/* PMIC */
@@ -456,4 +468,5 @@ MACHINE_START(GONI, "GONI")
 	.map_io		= goni_map_io,
 	.init_machine	= goni_machine_init,
 	.timer		= &s3c24xx_timer,
+	.reserve	= goni_reserve,
 MACHINE_END
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

In reply to Michal Nazarewicz

Hi Michal,

Thanks for working on this, we definitely need something along these lines.

Quoted 103 lines · On Monday 26 July 2010 16:40:30 Michal Nazarewicz wrote:

The Contiguous Memory Allocator framework is a set of APIs for allocating physically contiguous chunks of memory.

Various chips require contiguous blocks of memory to operate. Those chips include devices such as cameras, hardware video decoders and encoders, etc.

The code is highly modular and customisable to suit the needs of various users. Set of regions reserved for CMA can be configured on run-time and it is easy to add custom allocator algorithms if one has such need.

For more details see Documentation/contiguous-memory.txt.

---
 Documentation/00-INDEX                             |    2 +
 .../ABI/testing/sysfs-kernel-mm-contiguous         |    9 +
 Documentation/contiguous-memory.txt                |  646 +++++++++++
 Documentation/kernel-parameters.txt                |    4 +
 include/linux/cma.h                                |  445 ++++++++
 mm/Kconfig                                         |   34 +
 mm/Makefile                                        |    3 +
 mm/cma-best-fit.c                                  |  407 +++++++
 mm/cma.c                                           | 1170 ++++++++++++++++++++
 9 files changed, 2720 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-contiguous
 create mode 100644 Documentation/contiguous-memory.txt
 create mode 100644 include/linux/cma.h
 create mode 100644 mm/cma-best-fit.c
 create mode 100644 mm/cma.c
Documentation/00-INDEX +2 -0
index 5405f7a..bb50209 100644
@@ -94,6 +94,8 @@ connector/
 	- docs on the netlink based userspace<->kernel space communication mod.
 console/
 	- documentation on Linux console drivers.
+contiguous-memory.txt
+	- documentation on physically-contiguous memory allocation framework.
 cpu-freq/
 	- info on CPU frequency and voltage scaling.
 cpu-hotplug.txt
Documentation/ABI/testing/sysfs-kernel-mm-contiguous new file +9 -0
new file mode 100644
index 0000000..05e2f6a
@@ -0,0 +1,9 @@
+What:		/sys/kernel/mm/contiguous/
+Date:		July 2008
+Contact:	Michal Nazarewicz <[email protected]>
+Description:
+		/sys/kernel/mm/contiguous/ contains two files: asterisk and
+		map.  They are used to configure the Contiguous Memory
+		Allocator framework.
+
+		For details see Documentation/contiguous-memory.txt.
Documentation/contiguous-memory.txt new file +35 -0
new file mode 100644
index 0000000..6eb1295
@@ -0,0 +1,646 @@
+                                                             -*- org -*-
+
+* Contiguous Memory Allocator
+
+   The Contiguous Memory Allocator (CMA) is a framework, which allows
+   setting up a machine-specific configuration for physically-contiguous
+   memory management. Memory for devices is then allocated according
+   to that configuration.
+
+   The main role of the framework is not to allocate memory, but to
+   parse and manage memory configurations, as well as to act as an
+   in-between between device drivers and pluggable allocators. It is
+   thus not tied to any memory allocation method or strategy.
+
+** Why is it needed?
+
+    Various devices on embedded systems have no scatter-getter and/or
+    IO map support and as such require contiguous blocks of memory to
+    operate.  They include devices such as cameras, hardware video
+    decoders and encoders, etc.
+
+    Such devices often require big memory buffers (a full HD frame is,
+    for instance, more then 2 mega pixels large, i.e. more than 6 MB
+    of memory), which makes mechanisms such as kmalloc() ineffective.
+
+    Some embedded devices impose additional requirements on the
+    buffers, e.g. they can operate only on buffers allocated in
+    particular location/memory bank (if system has more than one
+    memory bank) or buffers aligned to a particular memory boundary.
+
+    Development of embedded devices have seen a big rise recently
+    (especially in the V4L area) and many such drivers include their
+    own memory allocation code. Most of them use bootmem-based methods.
+    CMA framework is an attempt to unify contiguous memory allocation
+    mechanisms and provide a simple API for device drivers, while

Documentation/contiguous-memory.txt line 45

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+    staying as customisable and modular as possible.
+
+** Design
+
+    The main design goal for the CMA was to provide a customisable and
+    modular framework, which could be configured to suit the needs of
+    individual systems.  Configuration specifies a list of memory
+    regions, which then are assigned to devices.  Memory regions can
+    be shared among many device drivers or assigned exclusively to
+    one.  This has been achieved in the following ways:

OK, I like the idea of regions, i.e. defining memory areas with specific properties or uses.

But why should it be possible to define regions through kernel parameters? Regions are typically fixed for a particular platform and can be setup in the platform specific code. Actually, one region could be setup by default:
DMA-able memory. That would be very handy in fact for many PCI-based TV capture drivers.

I think that the only thing that you want to set in the kernel params is the size of each region.

The same with assigning regions to drivers: why would you want to do that? The driver should know which regions it can use (with possible fallbacks). And it can know that provided regions are setup by the platform code and not created dynamically. This will simplify things enormously.

Documentation/contiguous-memory.txt line 56

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+    1. The core of the CMA does not handle allocation of memory and
+       management of free space.  Dedicated allocators are used for
+       that purpose.
+
+       This way, if the provided solution does not match demands
+       imposed on a given system, one can develop a new algorithm and
+       easily plug it into the CMA framework.
+
+       The presented solution includes an implementation of a best-fit
+       algorithm.

Again, do we really need user-settable per-region allocators? Just provide one with the option to later choose others through the kernel Kconfig files.

We can always add more complex scenarios later, but for an initial version I'd keep it simple.

Quoted 4 lines
Documentation/contiguous-memory.txt new file +4 -0
+
+    2. CMA allows a run-time configuration of the memory regions it
+       will use to allocate chunks of memory from.  The set of memory
+       regions is given on command line so it can be easily changed

Documentation/contiguous-memory.txt line 70

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+       without the need for recompiling the kernel.
+
+       Each region has it's own size, alignment demand, a start
+       address (physical address where it should be placed) and an
+       allocator algorithm assigned to the region.
+
+       This means that there can be different algorithms running at
+       the same time, if different devices on the platform have
+       distinct memory usage characteristics and different algorithm
+       match those the best way.

Seems overengineering to me. Just ensure that the code can be extended later to such hypothetical scenarios. They are hypothetical, right?

Documentation/contiguous-memory.txt line 84

Quoted 13 lines
Documentation/contiguous-memory.txt new file +13 -0
+    3. When requesting memory, devices have to introduce themselves.
+       This way CMA knows who the memory is allocated for.  This
+       allows for the system architect to specify which memory regions
+       each device should use.
+
+       3a. Devices can also specify a "kind" of memory they want.
+           This makes it possible to configure the system in such
+           a way, that a single device may get memory from different
+           memory regions, depending on the "kind" of memory it
+           requested.  For example, a video codec driver might want to
+           allocate some shared buffers from the first memory bank and
+           the other from the second to get the highest possible
+           memory throughput.

Not sure I understand this. Isn't this just two regions, one for each memory bank, and the driver requests some buffers from one region and some from the other? Not sure how a 'kind of memory' features in this.

Documentation/contiguous-memory.txt line 92

Quoted 7 lines
Documentation/contiguous-memory.txt new file +7 -0
+    4. For greater flexibility and extensibility, the framework allows
+       device drivers to register private regions of reserved memory
+       which then may be used only by them.
+
+       As an effect, if a driver would not use the rest of the CMA
+       interface, it can still use CMA allocators and other
+       mechanisms.

Why would you? Is there an actual driver that will need this?

Documentation/contiguous-memory.txt line 101

Quoted 9 lines
Documentation/contiguous-memory.txt new file +9 -0
+
+       4a. Early in boot process, device drivers can also request the
+           CMA framework to a reserve a region of memory for them
+           which then will be used as a private region.
+
+           This way, drivers do not need to directly call bootmem,
+           memblock or similar early allocator but merely register an
+           early region and the framework will handle the rest
+           including choosing the right early allocator.

The whole concept of private regions seems unnecessary to me.

<big snip>

It looks to me as if you tried to think of all possible hypothetical situations and write a framework for that. Of course, you may know more than I do, and some of these situations actually happen.

The basic design ideas are solid, I think. But you should get rid of all the fancy features and go back to basics. We can always add those features later should that become necessary. But removing features is much, much harder.

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG, part of Cisco
#6 lore raw

RE: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Hello,

Quoted 126 lines · Hans Verkuil

Hi Michal,

Thanks for working on this, we definitely need something along these lines.

Quoted 17 lines · On Monday 26 July 2010 16:40:30 Michal Nazarewicz wrote:

The Contiguous Memory Allocator framework is a set of APIs for allocating physically contiguous chunks of memory.

Various chips require contiguous blocks of memory to operate. Those chips include devices such as cameras, hardware video decoders and encoders, etc.

The code is highly modular and customisable to suit the needs of various users. Set of regions reserved for CMA can be configured on run-time and it is easy to add custom allocator algorithms if one has such need.

For more details see Documentation/contiguous-memory.txt.

Quoted 10 lines
---
 Documentation/00-INDEX                             |    2 +
 .../ABI/testing/sysfs-kernel-mm-contiguous         |    9 +
 Documentation/contiguous-memory.txt                |  646 +++++++++++
 Documentation/kernel-parameters.txt                |    4 +
 include/linux/cma.h                                |  445 ++++++++
 mm/Kconfig                                         |   34 +
 mm/Makefile                                        |    3 +
 mm/cma-best-fit.c                                  |  407 +++++++
 mm/cma.c                                           | 1170

++++++++++++++++++++

Documentation/00-INDEX connector/ · line 94

Quoted 13 lines
 9 files changed, 2720 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-contiguous
 create mode 100644 Documentation/contiguous-memory.txt
 create mode 100644 include/linux/cma.h
 create mode 100644 mm/cma-best-fit.c
 create mode 100644 mm/cma.c
Documentation/00-INDEX +0 -0
index 5405f7a..bb50209 100644
@@ -94,6 +94,8 @@ connector/
 	- docs on the netlink based userspace<->kernel space communication

mod.

Review

Quoted 8 lines
Documentation/00-INDEX +2 -0
 console/
 	- documentation on Linux console drivers.
+contiguous-memory.txt
+	- documentation on physically-contiguous memory allocation framework.
 cpu-freq/
 	- info on CPU frequency and voltage scaling.
 cpu-hotplug.txt
(quoted diff) +0 -0

b/Documentation/ABI/testing/sysfs-kernel-mm-contiguous

Review

Quoted 15 lines

new file mode 100644
index 0000000..05e2f6a

Documentation/ABI/testing/sysfs-kernel-mm-contiguous new file +9 -0
@@ -0,0 +1,9 @@
+What:		/sys/kernel/mm/contiguous/
+Date:		July 2008
+Contact:	Michal Nazarewicz <[email protected]>
+Description:
+		/sys/kernel/mm/contiguous/ contains two files: asterisk and
+		map.  They are used to configure the Contiguous Memory
+		Allocator framework.
+
+		For details see Documentation/contiguous-memory.txt.
(quoted diff) +0 -0

b/Documentation/contiguous-memory.txt

Quoted 40 lines

new file mode 100644
index 0000000..6eb1295

Documentation/contiguous-memory.txt new file +35 -0
@@ -0,0 +1,646 @@
+                                                             -*- org -*-
+
+* Contiguous Memory Allocator
+
+   The Contiguous Memory Allocator (CMA) is a framework, which allows
+   setting up a machine-specific configuration for physically-contiguous
+   memory management. Memory for devices is then allocated according
+   to that configuration.
+
+   The main role of the framework is not to allocate memory, but to
+   parse and manage memory configurations, as well as to act as an
+   in-between between device drivers and pluggable allocators. It is
+   thus not tied to any memory allocation method or strategy.
+
+** Why is it needed?
+
+    Various devices on embedded systems have no scatter-getter and/or
+    IO map support and as such require contiguous blocks of memory to
+    operate.  They include devices such as cameras, hardware video
+    decoders and encoders, etc.
+
+    Such devices often require big memory buffers (a full HD frame is,
+    for instance, more then 2 mega pixels large, i.e. more than 6 MB
+    of memory), which makes mechanisms such as kmalloc() ineffective.
+
+    Some embedded devices impose additional requirements on the
+    buffers, e.g. they can operate only on buffers allocated in
+    particular location/memory bank (if system has more than one
+    memory bank) or buffers aligned to a particular memory boundary.
+
+    Development of embedded devices have seen a big rise recently
+    (especially in the V4L area) and many such drivers include their
+    own memory allocation code. Most of them use bootmem-based methods.
+    CMA framework is an attempt to unify contiguous memory allocation
+    mechanisms and provide a simple API for device drivers, while

Documentation/contiguous-memory.txt line 45

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+    staying as customisable and modular as possible.
+
+** Design
+
+    The main design goal for the CMA was to provide a customisable and
+    modular framework, which could be configured to suit the needs of
+    individual systems.  Configuration specifies a list of memory
+    regions, which then are assigned to devices.  Memory regions can
+    be shared among many device drivers or assigned exclusively to
+    one.  This has been achieved in the following ways:

OK, I like the idea of regions, i.e. defining memory areas with specific properties or uses.

Quoted 6 lines

But why should it be possible to define regions through kernel parameters? Regions are typically fixed for a particular platform and can be setup in the
platform specific code. Actually, one region could be setup by default:
DMA-able memory. That would be very handy in fact for many PCI-based TV capture drivers.

IMHO this is a just desktop-point-of-view. In embedded world things are a bit different. Most SoCs have a some kind of common system memory and usually all build-in peripherals are able to DMA to any part of it (there is no DMA specific hardware zone).

Quoted 2 lines

I think that the only thing that you want to set in the kernel params is the size of each region.

Keeping it as a kernel parameter is very handy for development. But I agree that we might make it dependent on some Kconfig entry. This way a platform setup code would provide default region description just as an array of the region structures and we will get rid of the parsing code in the release versions.

Quoted 2 lines

The same with assigning regions to drivers: why would you want to do that? The driver should know which regions it can use (with possible fallbacks).

I'm sorry, but this is again a little 'desktop-centric point-of-view'. On desktop it is perfectly acceptable to have a separate memory region for each device. In embedded world memory is a precious resource. Of course we can go the 'separate memory region for each device' way, but we observed that at least some memory can be recovered if we decide to share memory regions for some of the devices.

Assigning regions to the drivers is a way to describe how memory can be shared. This is something that is independent from the actual drivers. Device drivers cannot and mustn't have such knowledge.

Quoted 16 lines

And it can know that provided regions are setup by the platform code and not created dynamically. This will simplify things enormously.

Documentation/contiguous-memory.txt line 56

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+    1. The core of the CMA does not handle allocation of memory and
+       management of free space.  Dedicated allocators are used for
+       that purpose.
+
+       This way, if the provided solution does not match demands
+       imposed on a given system, one can develop a new algorithm and
+       easily plug it into the CMA framework.
+
+       The presented solution includes an implementation of a best-fit
+       algorithm.

Again, do we really need user-settable per-region allocators? Just provide one with the option to later choose others through the kernel Kconfig files.

#7 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Quoted 137 lines · On Tuesday 27 July 2010 09:41:40 Marek Szyprowski wrote:

Hello,

Quoted 126 lines · Hans Verkuil

Hi Michal,

Thanks for working on this, we definitely need something along these lines.

Quoted 17 lines · On Monday 26 July 2010 16:40:30 Michal Nazarewicz wrote:

The Contiguous Memory Allocator framework is a set of APIs for allocating physically contiguous chunks of memory.

Various chips require contiguous blocks of memory to operate. Those chips include devices such as cameras, hardware video decoders and encoders, etc.

The code is highly modular and customisable to suit the needs of various users. Set of regions reserved for CMA can be configured on run-time and it is easy to add custom allocator algorithms if one has such need.

For more details see Documentation/contiguous-memory.txt.

Quoted 10 lines
---
 Documentation/00-INDEX                             |    2 +
 .../ABI/testing/sysfs-kernel-mm-contiguous         |    9 +
 Documentation/contiguous-memory.txt                |  646 +++++++++++
 Documentation/kernel-parameters.txt                |    4 +
 include/linux/cma.h                                |  445 ++++++++
 mm/Kconfig                                         |   34 +
 mm/Makefile                                        |    3 +
 mm/cma-best-fit.c                                  |  407 +++++++
 mm/cma.c                                           | 1170

++++++++++++++++++++

Documentation/00-INDEX connector/ · line 94

Quoted 13 lines
 9 files changed, 2720 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-contiguous
 create mode 100644 Documentation/contiguous-memory.txt
 create mode 100644 include/linux/cma.h
 create mode 100644 mm/cma-best-fit.c
 create mode 100644 mm/cma.c
Documentation/00-INDEX +0 -0
index 5405f7a..bb50209 100644
@@ -94,6 +94,8 @@ connector/
 	- docs on the netlink based userspace<->kernel space communication

mod.

Review

Quoted 8 lines
Documentation/00-INDEX +2 -0
 console/
 	- documentation on Linux console drivers.
+contiguous-memory.txt
+	- documentation on physically-contiguous memory allocation framework.
 cpu-freq/
 	- info on CPU frequency and voltage scaling.
 cpu-hotplug.txt
(quoted diff) +0 -0

b/Documentation/ABI/testing/sysfs-kernel-mm-contiguous

Review

Quoted 15 lines

new file mode 100644
index 0000000..05e2f6a

Documentation/ABI/testing/sysfs-kernel-mm-contiguous new file +9 -0
@@ -0,0 +1,9 @@
+What:		/sys/kernel/mm/contiguous/
+Date:		July 2008
+Contact:	Michal Nazarewicz <[email protected]>
+Description:
+		/sys/kernel/mm/contiguous/ contains two files: asterisk and
+		map.  They are used to configure the Contiguous Memory
+		Allocator framework.
+
+		For details see Documentation/contiguous-memory.txt.
(quoted diff) +0 -0

b/Documentation/contiguous-memory.txt

Quoted 40 lines

new file mode 100644
index 0000000..6eb1295

Documentation/contiguous-memory.txt new file +35 -0
@@ -0,0 +1,646 @@
+                                                             -*- org -*-
+
+* Contiguous Memory Allocator
+
+   The Contiguous Memory Allocator (CMA) is a framework, which allows
+   setting up a machine-specific configuration for physically-contiguous
+   memory management. Memory for devices is then allocated according
+   to that configuration.
+
+   The main role of the framework is not to allocate memory, but to
+   parse and manage memory configurations, as well as to act as an
+   in-between between device drivers and pluggable allocators. It is
+   thus not tied to any memory allocation method or strategy.
+
+** Why is it needed?
+
+    Various devices on embedded systems have no scatter-getter and/or
+    IO map support and as such require contiguous blocks of memory to
+    operate.  They include devices such as cameras, hardware video
+    decoders and encoders, etc.
+
+    Such devices often require big memory buffers (a full HD frame is,
+    for instance, more then 2 mega pixels large, i.e. more than 6 MB
+    of memory), which makes mechanisms such as kmalloc() ineffective.
+
+    Some embedded devices impose additional requirements on the
+    buffers, e.g. they can operate only on buffers allocated in
+    particular location/memory bank (if system has more than one
+    memory bank) or buffers aligned to a particular memory boundary.
+
+    Development of embedded devices have seen a big rise recently
+    (especially in the V4L area) and many such drivers include their
+    own memory allocation code. Most of them use bootmem-based methods.
+    CMA framework is an attempt to unify contiguous memory allocation
+    mechanisms and provide a simple API for device drivers, while

Documentation/contiguous-memory.txt line 45

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+    staying as customisable and modular as possible.
+
+** Design
+
+    The main design goal for the CMA was to provide a customisable and
+    modular framework, which could be configured to suit the needs of
+    individual systems.  Configuration specifies a list of memory
+    regions, which then are assigned to devices.  Memory regions can
+    be shared among many device drivers or assigned exclusively to
+    one.  This has been achieved in the following ways:

OK, I like the idea of regions, i.e. defining memory areas with specific properties or uses.

Quoted 6 lines

But why should it be possible to define regions through kernel parameters? Regions are typically fixed for a particular platform and can be setup in the
platform specific code. Actually, one region could be setup by default:
DMA-able memory. That would be very handy in fact for many PCI-based TV capture drivers.

Quoted 4 lines

IMHO this is a just desktop-point-of-view. In embedded world things are a bit different. Most SoCs have a some kind of common system memory and usually all build-in peripherals are able to DMA to any part of it (there is no DMA specific hardware zone).

There are still desktops in the world, you know :-) It's nice if cma can also be used there. Having a region called 'dma' or something like that would be useful for drivers. And yes, for most embedded systems that would be the equivalent of common memory.

Quoted 8 lines
Quoted 2 lines

I think that the only thing that you want to set in the kernel params is the size of each region.

Keeping it as a kernel parameter is very handy for development. But I agree that we might make it dependent on some Kconfig entry. This way a platform setup code would provide default region description just as an array of the region structures and we will get rid of the parsing code in the release versions.

I think that would help, yes.

Quoted 10 lines
Quoted 2 lines

The same with assigning regions to drivers: why would you want to do that? The driver should know which regions it can use (with possible fallbacks).

I'm sorry, but this is again a little 'desktop-centric point-of-view'. On desktop it is perfectly acceptable to have a separate memory region for each device. In embedded world memory is a precious resource. Of course we can go the 'separate memory region for each device' way, but we observed that at least some memory can be recovered if we decide to share memory regions for some of the devices.

Quoted 3 lines

Assigning regions to the drivers is a way to describe how memory can be shared. This is something that is independent from the actual drivers. Device drivers cannot and mustn't have such knowledge.

You misunderstand me. I wasn't promoting separate memory regions for each device. Quite the opposite: initially I would make regions shared by default. I actually do not see why device drivers can't have the knowledge to allocate their memory from particular regions. We are talking about embedded systems where these types of memory allocations tend to be well defined. E.g. the video capture driver needs to allocate X video buffers at boot time. For most devices that will be the dma region, in the case of samsung devices that will be regions for particular memory banks.

The only time reserving regions for specific devices might be necessary is if you can get memory fragmentation if multiple drivers use the same region. But if drivers just allocate the memory at boot, then shared regions work fine.

Quoted 18 lines
Quoted 4 lines

And it can know that provided regions are setup by the platform code and not created dynamically. This will simplify things enormously.

Quoted 13 lines

Documentation/contiguous-memory.txt line 56

Quoted 10 lines
Documentation/contiguous-memory.txt new file +10 -0
+    1. The core of the CMA does not handle allocation of memory and
+       management of free space.  Dedicated allocators are used for
+       that purpose.
+
+       This way, if the provided solution does not match demands
+       imposed on a given system, one can develop a new algorithm and
+       easily plug it into the CMA framework.
+
+       The presented solution includes an implementation of a best-fit
+       algorithm.

Again, do we really need user-settable per-region allocators? Just provide one with the option to later choose others through the kernel Kconfig files.

Quoted 5 lines

From our experience, yes. Different allocators can cope with different memory usage scenarios better or worse. This results in higher or lower memory fragmentation. System use cases are something that kernel or drivers are definitely not aware, so only user space can tune this parameter to get the best possible system behavior.

Can you describe some of those usage scenarios? The cases I am familiar with are the standard: 'need to allocate X buffers of Y MB memory each on boot' and these buffers are only freed when the module is unloaded. While you can get fragmentation in that case if you creatively load and unload modules, this is not the normal usage in a deployed system.

We already have several 'normal' memory allocators (slab, slub, slob, sleb, whatever. I've lost track there.) and I'm not very enthusiastic about the idea of adding another bunch of them. Especially not in a first release.

Keep it simple. Just fix the core problem first, even if you are not yet able to squeeze the last byte of memory from your system. Later you can always add features as needed and optimize it further.

Quoted 44 lines
Quoted 12 lines

We can always add more complex scenarios later, but for an initial version I'd keep it simple.

Quoted 9 lines
Documentation/contiguous-memory.txt new file +9 -0
+
+    2. CMA allows a run-time configuration of the memory regions it
+       will use to allocate chunks of memory from.  The set of memory
+       regions is given on command line so it can be easily changed
+       without the need for recompiling the kernel.
+
+       Each region has it's own size, alignment demand, a start
+       address (physical address where it should be placed) and an
+       allocator algorithm assigned to the region.
Quoted 8 lines

Documentation/contiguous-memory.txt line 70

Quoted 5 lines
Documentation/contiguous-memory.txt new file +5 -0
+
+       This means that there can be different algorithms running at
+       the same time, if different devices on the platform have
+       distinct memory usage characteristics and different algorithm
+       match those the best way.

Seems overengineering to me. Just ensure that the code can be extended later to such hypothetical scenarios. They are hypothetical, right?

Not really. Having the possibility to reconfigure memory configuration without kernel recompilation is very handy when one is tuning the configuration for the specific use case.

Quoted 14 lines

Documentation/contiguous-memory.txt line 84

Quoted 13 lines
Documentation/contiguous-memory.txt new file +13 -0
+    3. When requesting memory, devices have to introduce themselves.
+       This way CMA knows who the memory is allocated for.  This
+       allows for the system architect to specify which memory regions
+       each device should use.
+
+       3a. Devices can also specify a "kind" of memory they want.
+           This makes it possible to configure the system in such
+           a way, that a single device may get memory from different
+           memory regions, depending on the "kind" of memory it
+           requested.  For example, a video codec driver might want to
+           allocate some shared buffers from the first memory bank and
+           the other from the second to get the highest possible
+           memory throughput.
Quoted 4 lines

Not sure I understand this. Isn't this just two regions, one for each memory bank,
and the driver requests some buffers from one region and some from the other?

Quoted 8 lines

Right.

Quoted 1 line

Not sure how a 'kind of memory' features in this.

This 'kind' is a just cookie or a label used by the driver to distinguish requests for both memory banks. This functionality is essential for our hardware (just for hardware video codec we have 3 'kinds' of memory: memory bank A, memory bank B and special region for the firmware).

Let me see if I understand: driver D needs memory from bank A, so it specifies e.g. label "a" when requesting memory. And the kernel parameters are then supposed to define a region for memory bank A (say region "bank-a" and tell the cma that requests from driver D for memory of kind "a" should go to region "bank-a".

Of course, the driver can also just request memory from the platform-defined region "bank-a" directly rather than having to rely on userspace provided parameters.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Documentation/contiguous-memory.txt new file +7 -0
+    4. For greater flexibility and extensibility, the framework allows
+       device drivers to register private regions of reserved memory
+       which then may be used only by them.
+
+       As an effect, if a driver would not use the rest of the CMA
+       interface, it can still use CMA allocators and other
+       mechanisms.
Quoted 8 lines
Quoted 1 line

Why would you? Is there an actual driver that will need this?

This feature has been added after posting v1 of this rfc/patch. Jonathan Corbet suggested in <http://article.gmane.org/gmane.linux.kernel.mm/50689>
that viafb driver might register its own private memory and use cma just as an allocator. IMHO this is a good idea, this way we might remove a bunch of custom allocators from the drivers (yes, there are such all over the kernel).

It goes back to whether we want per-device (or per-driver) regions or just have global regions. What I have seen in practice is that these drivers just need X amount of contiguous memory on boot. Having just a single region (as it will be for most systems) to carve the buffers from is just as efficient if not more than creating separate regions for each driver. Only if you start freeing and reallocating memory later on will you get into trouble.

But if you do that, then you are trying to duplicate the behavior of the normal allocators in my opinion. I really don't think we want to go there.

Quoted 27 lines
Quoted 10 lines

Documentation/contiguous-memory.txt line 101

Quoted 9 lines
Documentation/contiguous-memory.txt new file +8 -0
+       4a. Early in boot process, device drivers can also request the
+           CMA framework to a reserve a region of memory for them
+           which then will be used as a private region.
+
+           This way, drivers do not need to directly call bootmem,
+           memblock or similar early allocator but merely register an
+           early region and the framework will handle the rest
+           including choosing the right early allocator.
Quoted 8 lines

The whole concept of private regions seems unnecessary to me.

<big snip>

It looks to me as if you tried to think of all possible hypothetical situations
and write a framework for that. Of course, you may know more than I do, and some of these situations actually happen.

Not exactly. We tried to design a solution that would cover all requirements for OUR (quite specific) embedded hardware. However we didn't want to tie it only to our platform. We just generalized most of our requirements so they can be reused for other systems.

Quoted 3 lines

The basic design ideas are solid, I think. But you should get rid of all the fancy features and go back to basics. We can always add those features later should that become necessary. But removing features is much, much

Quoted 7 lines

harder.

Well, please keep in mind that we cannot remove features that are essential for our solution. We know that a simple framework have some advantages (well, the most important one is the fact that it is easy to understand), but making it too simple would render it useless from our point of view (if it would not provide functionality required by our drivers and hardware).

Why not give a list of the requirements that your hardware has? The only requirement unique to your hardware that I am aware of is the need for buffers in specific memory banks (and apparently a special memory area for firmware).

So besides a 'dma' region (which is probably sufficient for most systems) you would also have a 'banka' and 'bankb' region and perhaps a 'fw' region.

In the kernel parameters the user can specify the sizes for these regions that are reserved at boot time and drivers will use cma to get their memory from the relevant region using a simple allocator.

If there are other requirements in your hardware (or other hardware) that should be taken into account for an initial version of the cma, then I'm interested in hearing about it.

One note: since the first version of the patch wasn't posted to linux-media I didn't follow the replies to that. If there are things in that thread that I should read, then just mail me some links.

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG, part of Cisco
#8 lore raw

RE: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Hello,

Quoted 152 lines · Hans Verkuil
Quoted 16 lines · On Tuesday 27 July 2010 09:41:40 Marek Szyprowski wrote:
Quoted 15 lines · Hans Verkuil
Quoted 14 lines · On Monday 26 July 2010 16:40:30 Michal Nazarewicz wrote:

The Contiguous Memory Allocator framework is a set of APIs for allocating physically contiguous chunks of memory.

Various chips require contiguous blocks of memory to operate. Those chips include devices such as cameras, hardware video decoders and encoders, etc.

The code is highly modular and customisable to suit the needs of various users. Set of regions reserved for CMA can be configured on run-time and it is easy to add custom allocator algorithms if one has such need.

For more details see Documentation/contiguous-memory.txt.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
---
 Documentation/00-INDEX                             |    2 +
 .../ABI/testing/sysfs-kernel-mm-contiguous         |    9 +
 Documentation/contiguous-memory.txt                |  646

+++++++++++

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
 Documentation/kernel-parameters.txt                |    4 +
 include/linux/cma.h                                |  445 ++++++++
 mm/Kconfig                                         |   34 +
 mm/Makefile                                        |    3 +
 mm/cma-best-fit.c                                  |  407 +++++++
 mm/cma.c                                           | 1170

++++++++++++++++++++

Quoted 2 lines
 9 files changed, 2720 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-

contiguous

Quoted 11 lines
Quoted 11 lines

Documentation/00-INDEX connector/ · line 94

Quoted 11 lines
 create mode 100644 Documentation/contiguous-memory.txt
 create mode 100644 include/linux/cma.h
 create mode 100644 mm/cma-best-fit.c
 create mode 100644 mm/cma.c
Documentation/00-INDEX +0 -0
index 5405f7a..bb50209 100644
@@ -94,6 +94,8 @@ connector/
 	- docs on the netlink based userspace<->kernel space

communication

Quoted 5 lines
Quoted 5 lines

mod.

Documentation/00-INDEX connector/

Quoted 4 lines
Documentation/00-INDEX +2 -0
 console/
 	- documentation on Linux console drivers.
+contiguous-memory.txt
+	- documentation on physically-contiguous memory allocation

framework.

Quoted 5 lines
Quoted 5 lines

Review

Quoted 4 lines
Documentation/00-INDEX +0 -0
 cpu-freq/
 	- info on CPU frequency and voltage scaling.
 cpu-hotplug.txt
(quoted diff) +0 -0

b/Documentation/ABI/testing/sysfs-kernel-mm-contiguous

Quoted 10 lines
Quoted 10 lines

Documentation/ABI/testing/sysfs-kernel-mm-contiguous line 5

Quoted 10 lines

new file mode 100644
index 0000000..05e2f6a

Documentation/ABI/testing/sysfs-kernel-mm-contiguous new file +5 -0
@@ -0,0 +1,9 @@
+What:		/sys/kernel/mm/contiguous/
+Date:		July 2008
+Contact:	Michal Nazarewicz <[email protected]>
+Description:
+		/sys/kernel/mm/contiguous/ contains two files: asterisk

and

Quoted 6 lines
Quoted 6 lines

Review

Quoted 5 lines
Documentation/ABI/testing/sysfs-kernel-mm-contiguous new file +4 -0
+		map.  They are used to configure the Contiguous Memory
+		Allocator framework.
+
+		For details see Documentation/contiguous-memory.txt.
(quoted diff) +0 -0

b/Documentation/contiguous-memory.txt

Quoted 6 lines
Quoted 6 lines

Documentation/contiguous-memory.txt line 1

Quoted 6 lines

new file mode 100644
index 0000000..6eb1295

Documentation/contiguous-memory.txt new file +1 -0
@@ -0,0 +1,646 @@
+                                                             -*- org

-*-

Quoted 4 lines
Quoted 4 lines

Documentation/contiguous-memory.txt

Quoted 4 lines
Documentation/contiguous-memory.txt new file +4 -0
+
+* Contiguous Memory Allocator
+
+   The Contiguous Memory Allocator (CMA) is a framework, which

allows

Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 6

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+   setting up a machine-specific configuration for physically-

contiguous

Quoted 11 lines
Quoted 11 lines

Documentation/contiguous-memory.txt

Quoted 11 lines
Documentation/contiguous-memory.txt new file +11 -0
+   memory management. Memory for devices is then allocated according
+   to that configuration.
+
+   The main role of the framework is not to allocate memory, but to
+   parse and manage memory configurations, as well as to act as an
+   in-between between device drivers and pluggable allocators. It is
+   thus not tied to any memory allocation method or strategy.
+
+** Why is it needed?
+
+    Various devices on embedded systems have no scatter-getter

and/or

Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 18

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+    IO map support and as such require contiguous blocks of memory

to

Quoted 4 lines
Quoted 4 lines

Documentation/contiguous-memory.txt

Quoted 4 lines
Documentation/contiguous-memory.txt new file +4 -0
+    operate.  They include devices such as cameras, hardware video
+    decoders and encoders, etc.
+
+    Such devices often require big memory buffers (a full HD frame

is,

Quoted 11 lines
Quoted 11 lines

Documentation/contiguous-memory.txt

Quoted 11 lines
Documentation/contiguous-memory.txt new file +11 -0
+    for instance, more then 2 mega pixels large, i.e. more than 6 MB
+    of memory), which makes mechanisms such as kmalloc() ineffective.
+
+    Some embedded devices impose additional requirements on the
+    buffers, e.g. they can operate only on buffers allocated in
+    particular location/memory bank (if system has more than one
+    memory bank) or buffers aligned to a particular memory boundary.
+
+    Development of embedded devices have seen a big rise recently
+    (especially in the V4L area) and many such drivers include their
+    own memory allocation code. Most of them use bootmem-based

methods.

Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 34

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+    CMA framework is an attempt to unify contiguous memory

allocation

Quoted 6 lines
Quoted 6 lines

Documentation/contiguous-memory.txt

Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+    mechanisms and provide a simple API for device drivers, while
+    staying as customisable and modular as possible.
+
+** Design
+
+    The main design goal for the CMA was to provide a customisable

and

Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 41

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+    modular framework, which could be configured to suit the needs

of

Quoted 6 lines
Quoted 6 lines

Documentation/contiguous-memory.txt line 45

Quoted 4 lines
Documentation/contiguous-memory.txt new file +4 -0
+    individual systems.  Configuration specifies a list of memory
+    regions, which then are assigned to devices.  Memory regions can
+    be shared among many device drivers or assigned exclusively to
+    one.  This has been achieved in the following ways:

OK, I like the idea of regions, i.e. defining memory areas with

specific

Quoted 3 lines
Quoted 3 lines

properties or uses.

But why should it be possible to define regions through kernel

parameters?

Quoted 1 line
Quoted 1 line

Regions are typically fixed for a particular platform and can be setup

in

Quoted 8 lines
Quoted 4 lines

the
platform specific code. Actually, one region could be setup by default:
DMA-able memory. That would be very handy in fact for many PCI-based TV capture drivers.

IMHO this is a just desktop-point-of-view. In embedded world things are a bit different. Most SoCs have a some kind of common system memory and usually all build-in peripherals are able to DMA to any part of it (there

Quoted 7 lines

is

Quoted 1 line

no DMA specific hardware zone).

There are still desktops in the world, you know :-) It's nice if cma can also be used there. Having a region called 'dma' or something like that would be useful for drivers. And yes, for most embedded systems that would be the equivalent of common memory.

I really see no problem using CMA on desktops, although in most cases only a few drivers will be able to make use of it.

Quoted 34 lines
Quoted 1 line
Quoted 1 line

I think that the only thing that you want to set in the kernel params

is

Quoted 3 lines
Quoted 1 line

the size of each region.

Keeping it as a kernel parameter is very handy for development. But I

agree

Quoted 1 line

that we might make it dependent on some Kconfig entry. This way a

platform

Quoted 1 line

setup code would provide default region description just as an array of

the

Quoted 2 lines

region structures and we will get rid of the parsing code in the release versions.

I think that would help, yes.

Quoted 1 line
Quoted 1 line

The same with assigning regions to drivers: why would you want to do

that?

Quoted 1 line
Quoted 1 line

The driver should know which regions it can use (with possible

fallbacks).

Quoted 3 lines

I'm sorry, but this is again a little 'desktop-centric point-of-view'. On desktop it is perfectly acceptable to have a separate memory region for

each

Quoted 1 line

device. In embedded world memory is a precious resource. Of course we can

go

Quoted 2 lines

the 'separate memory region for each device' way, but we observed that at least some memory can be recovered if we decide to share memory regions

for

Quoted 5 lines

some of the devices.

Assigning regions to the drivers is a way to describe how memory can be shared. This is something that is independent from the actual drivers. Device drivers cannot and mustn't have such knowledge.

Quoted 8 lines

You misunderstand me. I wasn't promoting separate memory regions for each device. Quite the opposite: initially I would make regions shared by default. I actually do not see why device drivers can't have the knowledge to allocate their memory from particular regions. We are talking about embedded systems where these types of memory allocations tend to be well defined. E.g. the video capture driver needs to allocate X video buffers at boot time. For most devices that will be the dma region, in the case of samsung devices that will be regions for particular memory banks.

Let me introduce one more example. As you may know we have 3 video-processor capture devices (Samsung FIMC) and a hardware codec (like Samsung MFC). FIMC can capture video data from camera sensor and accelerate common video processing tasks (like up/down scaling and color space conversion). Two FIMC and MFC are require for things like HD video encoding or decoding with online display/preview. This task require huge video buffers that are usually allocated and freed at the same time. The third FIMC can be used for gfx acceleration (color space conversion and scaling are quite common tasks in GUI). This latter task usually introduces a lot of memory fragmentation, as gfx surfaces are usually quite small (definitely smaller than HD frames or 8MPix picture from camera). It would be really wise to get that third FIMC device to use memory buffer that will be shared with 3D accelerator (which has quite similar usage scenarios and suffers from similar memory fragmentation).

Quoted 33 lines

The only time reserving regions for specific devices might be necessary is if you can get memory fragmentation if multiple drivers use the same region. But if drivers just allocate the memory at boot, then shared regions work fine.

Quoted 2 lines
Quoted 2 lines

And it can know that provided regions are setup by the platform code

and

Quoted 8 lines
Quoted 8 lines

not created dynamically. This will simplify things enormously.

Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+    1. The core of the CMA does not handle allocation of memory and
+       management of free space.  Dedicated allocators are used for
+       that purpose.
+
+       This way, if the provided solution does not match demands
+       imposed on a given system, one can develop a new algorithm

and

Quoted 3 lines
Quoted 3 lines

Documentation/contiguous-memory.txt

Quoted 3 lines
Documentation/contiguous-memory.txt new file +3 -0
+       easily plug it into the CMA framework.
+
+       The presented solution includes an implementation of a best-

fit

Quoted 3 lines
Quoted 3 lines

Documentation/contiguous-memory.txt line 56

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+       algorithm.

Again, do we really need user-settable per-region allocators? Just

provide

Quoted 1 line
Quoted 1 line

one with the option to later choose others through the kernel Kconfig

files.

Quoted 2 lines

From our experience, yes. Different allocators can cope with different

memory

Quoted 3 lines

usage scenarios better or worse. This results in higher or lower memory fragmentation. System use cases are something that kernel or drivers are definitely not aware, so only user space can tune this parameter to get

the

Quoted 8 lines
Quoted 1 line

best possible system behavior.

Can you describe some of those usage scenarios? The cases I am familiar with
are the standard: 'need to allocate X buffers of Y MB memory each on boot' and these buffers are only freed when the module is unloaded. While you can get fragmentation in that case if you creatively load and unload modules, this is not the normal usage in a deployed system.

We don't want to allocate X buffers of Y MB memory each on boot. Instead we want to just reserve XX MB memory and then dynamically allocate buffers from it. This enables us to perform the following 2 tasks:
1. movie decoding in HD-quality (only one instance)
2. two instances of SD-quality movie decoding and SD-quality move encoding (example: video conference)

We know that these two use cases are exclusive, so they can use the same reserved memory.

Quoted 7 lines

We already have several 'normal' memory allocators (slab, slub, slob, sleb, whatever. I've lost track there.) and I'm not very enthusiastic about the idea of adding another bunch of them. Especially not in a first release.

Keep it simple. Just fix the core problem first, even if you are not yet able to squeeze the last byte of memory from your system. Later you can always add features as needed and optimize it further.

We are trying to keep it as simple as possible :)

Quoted 67 lines
Quoted 1 line
Quoted 1 line

We can always add more complex scenarios later, but for an initial

version

Quoted 5 lines
Quoted 5 lines

I'd keep it simple.

Quoted 3 lines
Documentation/contiguous-memory.txt new file +3 -0
+
+    2. CMA allows a run-time configuration of the memory regions it
+       will use to allocate chunks of memory from.  The set of

memory

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+       regions is given on command line so it can be easily changed
+       without the need for recompiling the kernel.
+
+       Each region has it's own size, alignment demand, a start
+       address (physical address where it should be placed) and an
+       allocator algorithm assigned to the region.
Quoted 8 lines

Documentation/contiguous-memory.txt line 70

Quoted 5 lines
Documentation/contiguous-memory.txt new file +5 -0
+
+       This means that there can be different algorithms running at
+       the same time, if different devices on the platform have
+       distinct memory usage characteristics and different algorithm
+       match those the best way.

Seems overengineering to me. Just ensure that the code can be extended later to such hypothetical scenarios. They are hypothetical, right?

Quoted 7 lines

Not really. Having the possibility to reconfigure memory configuration without kernel recompilation is very handy when one is tuning the configuration for the specific use case.

Quoted 3 lines
Quoted 3 lines
Documentation/contiguous-memory.txt new file +3 -0
+    3. When requesting memory, devices have to introduce themselves.
+       This way CMA knows who the memory is allocated for.  This
+       allows for the system architect to specify which memory

regions

Quoted 7 lines
Quoted 7 lines

Documentation/contiguous-memory.txt

Quoted 7 lines
Documentation/contiguous-memory.txt new file +7 -0
+       each device should use.
+
+       3a. Devices can also specify a "kind" of memory they want.
+           This makes it possible to configure the system in such
+           a way, that a single device may get memory from different
+           memory regions, depending on the "kind" of memory it
+           requested.  For example, a video codec driver might want

to

Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 80

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+           allocate some shared buffers from the first memory bank

and

Quoted 8 lines
Quoted 7 lines

Documentation/contiguous-memory.txt line 84

Quoted 2 lines
Documentation/contiguous-memory.txt new file +2 -0
+           the other from the second to get the highest possible
+           memory throughput.

Not sure I understand this. Isn't this just two regions, one for each memory bank,
and the driver requests some buffers from one region and some from the other?

Quoted 7 lines

Right.

Quoted 1 line

Not sure how a 'kind of memory' features in this.

This 'kind' is a just cookie or a label used by the driver to distinguish requests for both memory banks. This functionality is essential for our hardware (just for hardware video codec we have 3 'kinds' of memory:

memory

Quoted 1 line

bank A, memory bank B and special region for the firmware).

Let me see if I understand: driver D needs memory from bank A, so it specifies
e.g. label "a" when requesting memory. And the kernel parameters are then supposed
to define a region for memory bank A (say region "bank-a" and tell the cma that
requests from driver D for memory of kind "a" should go to region "bank-a".

Quoted 4 lines

Of course, the driver can also just request memory from the platform- defined
region "bank-a" directly rather than having to rely on userspace provided parameters.

Right. Regions provided by kernel command line are considered to be alternative to the default configuration for tuning/debugging purposes.

Quoted 14 lines
Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 83

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+    4. For greater flexibility and extensibility, the framework

allows

Quoted 11 lines
Quoted 8 lines

Documentation/contiguous-memory.txt line 92

Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+       device drivers to register private regions of reserved memory
+       which then may be used only by them.
+
+       As an effect, if a driver would not use the rest of the CMA
+       interface, it can still use CMA allocators and other
+       mechanisms.

Why would you? Is there an actual driver that will need this?

This feature has been added after posting v1 of this rfc/patch. Jonathan Corbet suggested in

<http://article.gmane.org/gmane.linux.kernel.mm/50689>

Quoted 8 lines
Quoted 2 lines

that viafb driver might register its own private memory and use cma just as an allocator. IMHO this is a good idea, this way we might remove a

bunch

Quoted 2 lines

of custom allocators from the drivers (yes, there are such all over the kernel).

It goes back to whether we want per-device (or per-driver) regions or just have global regions.

IMHO both types of regions can coexist and we can make use of their advantages. I see no point in limiting CMA only to one type of the regions.

Quoted 8 lines

What I have seen in practice is that these drivers just
need X amount of contiguous memory on boot. Having just a single region (as it will be for most systems) to carve the buffers from is just as efficient if not more than creating separate regions for each driver. Only if you start freeing and reallocating memory later on will you get into trouble.

But if you do that, then you are trying to duplicate the behavior of the normal allocators in my opinion. I really don't think we want to go there.

Well, then why some drivers like viafb or omapfb have custom memory allocators? This case shows that there is still a need for another one.

Quoted 41 lines
Quoted 1 line
Quoted 1 line

Documentation/contiguous-memory.txt line 90

Quoted 1 line
Documentation/contiguous-memory.txt new file +1 -0
+       4a. Early in boot process, device drivers can also request

the

Quoted 5 lines
Quoted 5 lines

Documentation/contiguous-memory.txt

Quoted 5 lines
Documentation/contiguous-memory.txt new file +5 -0
+           CMA framework to a reserve a region of memory for them
+           which then will be used as a private region.
+
+           This way, drivers do not need to directly call bootmem,
+           memblock or similar early allocator but merely register

an

Quoted 5 lines
Quoted 5 lines

Documentation/contiguous-memory.txt line 101

Quoted 2 lines
Documentation/contiguous-memory.txt new file +2 -0
+           early region and the framework will handle the rest
+           including choosing the right early allocator.

The whole concept of private regions seems unnecessary to me.

Quoted 8 lines
Quoted 6 lines

<big snip>

It looks to me as if you tried to think of all possible hypothetical situations
and write a framework for that. Of course, you may know more than I do, and some of these situations actually happen.

Not exactly. We tried to design a solution that would cover all

requirements

Quoted 1 line

for OUR (quite specific) embedded hardware. However we didn't want to tie

it

Quoted 1 line

only to our platform. We just generalized most of our requirements so

they can

Quoted 3 lines

be reused for other systems.

Quoted 1 line

The basic design ideas are solid, I think. But you should get rid of

all

Quoted 1 line
Quoted 1 line

the fancy features and go back to basics. We can always add those

features

Quoted 4 lines
Quoted 1 line

later should that become necessary. But removing features is much, much

harder.

Well, please keep in mind that we cannot remove features that are

essential

Quoted 4 lines

for our solution. We know that a simple framework have some advantages (well, the most important one is the fact that it is easy to understand), but making it too simple would render it useless from our point of view (if it would not provide functionality required by our drivers and

Quoted 6 lines

hardware).

Why not give a list of the requirements that your hardware has? The only requirement unique to your hardware that I am aware of is the need for buffers in specific memory banks (and apparently a special memory area for firmware).

Yes, this is a real hardware requirement that cannot be dropped. Other requirements come from the system design (software).

Quoted 2 lines

So besides a 'dma' region (which is probably sufficient for most systems) you would also have a 'banka' and 'bankb' region and perhaps a 'fw' region.

Actually this DMA region is either just a 'banka' or 'bankb'. There is no special requirements for the common dma memory and no need for the separate 'dma' region in our case. We just want to make it shared with at least camera capture buffers.

Quoted 7 lines

In the kernel parameters the user can specify the sizes for these regions that are reserved at boot time and drivers will use cma to get their memory from the relevant region using a simple allocator.

If there are other requirements in your hardware (or other hardware) that should be taken into account for an initial version of the cma, then I'm interested in hearing about it.

We will try to summarize our requirements again and repost them soon.

Quoted 3 lines

One note: since the first version of the patch wasn't posted to linux-media I didn't follow the replies to that. If there are things in that thread that I should read, then just mail me some links.

I'm very sorry that the first version has been posted only to linux-mm. The discussion was mainly about the way how the memory configuration and device mapping should be provided (first version used only kernel parameters for that). We should probably also CC a few more kernel subsystems that might benefit from a common memory allocator, but we already were a bit scared by the number of people and lists in this series. We really don't want to get a 'spammer' label ;)

Best regards
--
Marek Szyprowski
Samsung Poland R&D Center

#9 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Hi Marek,

On Wednesday 28 July 2010 11:04:56 Marek Szyprowski wrote:

<snip>

Quoted 7 lines
Quoted 4 lines

There are still desktops in the world, you know :-) It's nice if cma can also be used there. Having a region called 'dma' or something like that would be useful for drivers. And yes, for most embedded systems that would be the equivalent of common memory.

I really see no problem using CMA on desktops, although in most cases only a few drivers will be able to make use of it.

As past maintainer of ivtv I can testify that I know at least one driver that could definitely benefit from CMA.

Quoted 44 lines
Quoted 35 lines
Quoted 2 lines
Quoted 2 lines

I think that the only thing that you want to set in the kernel params

is

Quoted 3 lines
Quoted 1 line

the size of each region.

Keeping it as a kernel parameter is very handy for development. But I

agree

Quoted 1 line

that we might make it dependent on some Kconfig entry. This way a

platform

Quoted 1 line

setup code would provide default region description just as an array of

the

Quoted 2 lines

region structures and we will get rid of the parsing code in the release versions.

I think that would help, yes.

Quoted 1 line
Quoted 1 line

The same with assigning regions to drivers: why would you want to do

that?

Quoted 1 line
Quoted 1 line

The driver should know which regions it can use (with possible

fallbacks).

Quoted 3 lines

I'm sorry, but this is again a little 'desktop-centric point-of-view'. On desktop it is perfectly acceptable to have a separate memory region for

each

Quoted 1 line

device. In embedded world memory is a precious resource. Of course we can

go

Quoted 2 lines

the 'separate memory region for each device' way, but we observed that at least some memory can be recovered if we decide to share memory regions

for

Quoted 5 lines

some of the devices.

Assigning regions to the drivers is a way to describe how memory can be shared. This is something that is independent from the actual drivers. Device drivers cannot and mustn't have such knowledge.

Quoted 8 lines

You misunderstand me. I wasn't promoting separate memory regions for each device. Quite the opposite: initially I would make regions shared by default. I actually do not see why device drivers can't have the knowledge to allocate their memory from particular regions. We are talking about embedded systems where these types of memory allocations tend to be well defined. E.g. the video capture driver needs to allocate X video buffers at boot time. For most devices that will be the dma region, in the case of samsung devices that will be regions for particular memory banks.

Quoted 14 lines

Let me introduce one more example. As you may know we have 3 video-processor capture devices (Samsung FIMC) and a hardware codec (like Samsung MFC). FIMC can capture video data from camera sensor and accelerate common video processing tasks (like up/down scaling and color space conversion). Two FIMC and MFC are require for things like HD video encoding or decoding with online display/preview. This task require huge video buffers that are usually allocated and freed at the same time. The third FIMC can be used for gfx acceleration (color space conversion and scaling are quite common tasks in GUI). This latter task usually introduces a lot of memory fragmentation, as gfx surfaces are usually quite small (definitely smaller than HD frames or 8MPix picture from camera). It would be really wise to get that third FIMC device to use memory buffer that will be shared with 3D accelerator (which has quite similar usage scenarios and suffers from similar memory fragmentation).

OK, I understand. And I assume both gfx and 3D acceleration need to use a specific region? If they can use any type of memory, then this might be more appropriate for kmalloc and friends.

Quoted 49 lines
Quoted 33 lines

The only time reserving regions for specific devices might be necessary is if you can get memory fragmentation if multiple drivers use the same region. But if drivers just allocate the memory at boot, then shared regions work fine.

Quoted 2 lines
Quoted 2 lines

And it can know that provided regions are setup by the platform code

and

Quoted 8 lines
Quoted 8 lines

not created dynamically. This will simplify things enormously.

Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+    1. The core of the CMA does not handle allocation of memory and
+       management of free space.  Dedicated allocators are used for
+       that purpose.
+
+       This way, if the provided solution does not match demands
+       imposed on a given system, one can develop a new algorithm

and

Quoted 3 lines
Quoted 3 lines

Documentation/contiguous-memory.txt

Quoted 3 lines
Documentation/contiguous-memory.txt new file +3 -0
+       easily plug it into the CMA framework.
+
+       The presented solution includes an implementation of a best-

fit

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

+ algorithm.

Again, do we really need user-settable per-region allocators? Just

provide

Quoted 1 line
Quoted 1 line

one with the option to later choose others through the kernel Kconfig

files.

Quoted 2 lines

From our experience, yes. Different allocators can cope with different

memory

Quoted 3 lines

usage scenarios better or worse. This results in higher or lower memory fragmentation. System use cases are something that kernel or drivers are definitely not aware, so only user space can tune this parameter to get

the

Quoted 8 lines
Quoted 1 line

best possible system behavior.

Can you describe some of those usage scenarios? The cases I am familiar with
are the standard: 'need to allocate X buffers of Y MB memory each on boot' and these buffers are only freed when the module is unloaded. While you can get fragmentation in that case if you creatively load and unload modules, this is not the normal usage in a deployed system.

We don't want to allocate X buffers of Y MB memory each on boot. Instead we want to just reserve XX MB memory and then dynamically allocate buffers from it. This enables us to perform the following 2 tasks:
1. movie decoding in HD-quality (only one instance)
2. two instances of SD-quality movie decoding and SD-quality move encoding (example: video conference)

Quoted 2 lines

We know that these two use cases are exclusive, so they can use the same reserved memory.

When I said 'allocating X buffers of Y MB memory' I meant that you need to allocate a known amount of memory (X * Y MB in this case). So the boot args say e.g. dma=40MB and the driver just allocates X buffers from that region.

Switching to SD quality requires releasing those buffers and instead allocating a number of smaller buffers from the same region.

For these use-cases the allocator can be very simple and probably covers most use-cases.

Anyway, I'm no expert on memory allocators and people on the linux-mm list are no doubt much more qualified to discuss this. My main concern is that of trying to add too much for a first release. It is simply easier to start simple and extend as needed. That makes it easier to be accepted in the mainline.

Quoted 118 lines
Quoted 8 lines

We already have several 'normal' memory allocators (slab, slub, slob, sleb, whatever. I've lost track there.) and I'm not very enthusiastic about the idea of adding another bunch of them. Especially not in a first release.

Keep it simple. Just fix the core problem first, even if you are not yet able to squeeze the last byte of memory from your system. Later you can always add features as needed and optimize it further.

We are trying to keep it as simple as possible :)

Quoted 67 lines
Quoted 1 line
Quoted 1 line

We can always add more complex scenarios later, but for an initial

version

Quoted 5 lines
Quoted 5 lines

I'd keep it simple.

Quoted 3 lines
Documentation/contiguous-memory.txt new file +3 -0
+
+    2. CMA allows a run-time configuration of the memory regions it
+       will use to allocate chunks of memory from.  The set of

memory

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+       regions is given on command line so it can be easily changed
+       without the need for recompiling the kernel.
+
+       Each region has it's own size, alignment demand, a start
+       address (physical address where it should be placed) and an
+       allocator algorithm assigned to the region.
Quoted 8 lines

Documentation/contiguous-memory.txt line 70

Quoted 5 lines
Documentation/contiguous-memory.txt new file +5 -0
+
+       This means that there can be different algorithms running at
+       the same time, if different devices on the platform have
+       distinct memory usage characteristics and different algorithm
+       match those the best way.

Seems overengineering to me. Just ensure that the code can be extended later to such hypothetical scenarios. They are hypothetical, right?

Quoted 7 lines

Not really. Having the possibility to reconfigure memory configuration without kernel recompilation is very handy when one is tuning the configuration for the specific use case.

Quoted 3 lines
Quoted 3 lines
Documentation/contiguous-memory.txt new file +3 -0
+    3. When requesting memory, devices have to introduce themselves.
+       This way CMA knows who the memory is allocated for.  This
+       allows for the system architect to specify which memory

regions

Quoted 7 lines
Quoted 7 lines

Documentation/contiguous-memory.txt

Quoted 7 lines
Documentation/contiguous-memory.txt new file +7 -0
+       each device should use.
+
+       3a. Devices can also specify a "kind" of memory they want.
+           This makes it possible to configure the system in such
+           a way, that a single device may get memory from different
+           memory regions, depending on the "kind" of memory it
+           requested.  For example, a video codec driver might want

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

+ allocate some shared buffers from the first memory bank

and

Quoted 8 lines
Quoted 7 lines

Documentation/contiguous-memory.txt line 84

Quoted 2 lines
Documentation/contiguous-memory.txt new file +2 -0
+           the other from the second to get the highest possible
+           memory throughput.

Not sure I understand this. Isn't this just two regions, one for each memory bank,
and the driver requests some buffers from one region and some from the other?

Quoted 7 lines

Right.

Quoted 1 line

Not sure how a 'kind of memory' features in this.

This 'kind' is a just cookie or a label used by the driver to distinguish requests for both memory banks. This functionality is essential for our hardware (just for hardware video codec we have 3 'kinds' of memory:

memory

Quoted 1 line

bank A, memory bank B and special region for the firmware).

Let me see if I understand: driver D needs memory from bank A, so it specifies
e.g. label "a" when requesting memory. And the kernel parameters are then supposed
to define a region for memory bank A (say region "bank-a" and tell the cma that
requests from driver D for memory of kind "a" should go to region "bank-a".

Quoted 4 lines

Of course, the driver can also just request memory from the platform- defined
region "bank-a" directly rather than having to rely on userspace provided parameters.

Right. Regions provided by kernel command line are considered to be alternative to the default configuration for tuning/debugging purposes.

Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

+ 4. For greater flexibility and extensibility, the framework

allows

Quoted 11 lines
Quoted 8 lines

Documentation/contiguous-memory.txt line 92

Quoted 6 lines
Documentation/contiguous-memory.txt new file +6 -0
+       device drivers to register private regions of reserved memory
+       which then may be used only by them.
+
+       As an effect, if a driver would not use the rest of the CMA
+       interface, it can still use CMA allocators and other
+       mechanisms.

Why would you? Is there an actual driver that will need this?

This feature has been added after posting v1 of this rfc/patch. Jonathan Corbet suggested in

<http://article.gmane.org/gmane.linux.kernel.mm/50689>

Quoted 8 lines
Quoted 2 lines

that viafb driver might register its own private memory and use cma just as an allocator. IMHO this is a good idea, this way we might remove a

bunch

Quoted 2 lines

of custom allocators from the drivers (yes, there are such all over the kernel).

It goes back to whether we want per-device (or per-driver) regions or just have global regions.

IMHO both types of regions can coexist and we can make use of their advantages. I see no point in limiting CMA only to one type of the regions.

Quoted 5 lines

What I have seen in practice is that these drivers just
need X amount of contiguous memory on boot. Having just a single region (as it will be for most systems) to carve the buffers from is just as efficient if not more than creating separate regions for each driver. Only if you start freeing and reallocating memory later on will you get into trouble.

Quoted 5 lines
Quoted 2 lines

But if you do that, then you are trying to duplicate the behavior of the normal allocators in my opinion. I really don't think we want to go there.

Well, then why some drivers like viafb or omapfb have custom memory allocators? This case shows that there is still a need for another one.

I have no idea why they have allocators. Someone would have to analyze that code. Just because they use allocators doesn't necessarily mean that they really need them :-)

Quoted 74 lines
Quoted 42 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ 4a. Early in boot process, device drivers can also request

the

Quoted 5 lines
Quoted 5 lines

Documentation/contiguous-memory.txt

Quoted 5 lines
Documentation/contiguous-memory.txt new file +5 -0
+           CMA framework to a reserve a region of memory for them
+           which then will be used as a private region.
+
+           This way, drivers do not need to directly call bootmem,
+           memblock or similar early allocator but merely register

an

Quoted 5 lines
Quoted 5 lines

Documentation/contiguous-memory.txt line 101

Quoted 2 lines
Documentation/contiguous-memory.txt new file +2 -0
+           early region and the framework will handle the rest
+           including choosing the right early allocator.

The whole concept of private regions seems unnecessary to me.

Quoted 8 lines
Quoted 6 lines

<big snip>

It looks to me as if you tried to think of all possible hypothetical situations
and write a framework for that. Of course, you may know more than I do, and some of these situations actually happen.

Not exactly. We tried to design a solution that would cover all

requirements

Quoted 1 line

for OUR (quite specific) embedded hardware. However we didn't want to tie

it

Quoted 1 line

only to our platform. We just generalized most of our requirements so

they can

Quoted 3 lines

be reused for other systems.

Quoted 1 line

The basic design ideas are solid, I think. But you should get rid of

all

Quoted 1 line
Quoted 1 line

the fancy features and go back to basics. We can always add those

features

Quoted 4 lines
Quoted 1 line

later should that become necessary. But removing features is much, much

harder.

Well, please keep in mind that we cannot remove features that are

essential

Quoted 4 lines

for our solution. We know that a simple framework have some advantages (well, the most important one is the fact that it is easy to understand), but making it too simple would render it useless from our point of view (if it would not provide functionality required by our drivers and

Quoted 6 lines

hardware).

Why not give a list of the requirements that your hardware has? The only requirement unique to your hardware that I am aware of is the need for buffers in specific memory banks (and apparently a special memory area for firmware).

Yes, this is a real hardware requirement that cannot be dropped. Other requirements come from the system design (software).

Quoted 2 lines

So besides a 'dma' region (which is probably sufficient for most systems) you would also have a 'banka' and 'bankb' region and perhaps a 'fw' region.

Actually this DMA region is either just a 'banka' or 'bankb'. There is no special requirements for the common dma memory and no need for the separate 'dma' region in our case. We just want to make it shared with at least camera capture buffers.

Quoted 7 lines

In the kernel parameters the user can specify the sizes for these regions that are reserved at boot time and drivers will use cma to get their memory from the relevant region using a simple allocator.

If there are other requirements in your hardware (or other hardware) that should be taken into account for an initial version of the cma, then I'm interested in hearing about it.

We will try to summarize our requirements again and repost them soon.

Quoted 3 lines

One note: since the first version of the patch wasn't posted to linux-media I didn't follow the replies to that. If there are things in that thread that I should read, then just mail me some links.

Quoted 7 lines

I'm very sorry that the first version has been posted only to linux-mm. The discussion was mainly about the way how the memory configuration and device mapping should be provided (first version used only kernel parameters for that). We should probably also CC a few more kernel subsystems that might benefit from a common memory allocator, but we already were a bit scared by the number of people and lists in this series. We really don't want to get a 'spammer' label ;)

You probably got the main interested parties now.

Regarding regions and shared and per-driver buffers: I've been thinking about this a bit more and I have a proposal of my own.

There are two different aspects to this: first there is the hardware aspect: if the hardware needs memory from specific memory banks or with specific requirements (e.g. DMAable), then those regions should be setup in the platform code. There you know the memory sizes/alignments/etc. since that is hw dependent. The other reason is that drivers need to be able to tell CMA that they need to allocate from such regions. You can't have a driver refer to a region that is specified through kernel parameters, that would create a very ugly dependency.

The other aspect is how to setup buffers. A global buffer is simply setup by assigning a size to the region: "banka=20MB". Unless specified otherwise any driver that needs memory from banka will use that global banka buffer.

Alternatively, you can set aside memory from a region specifically for drivers:
banka/foo=30MB. This allocated 30 MB from region banka specifically for driver foo.

You can also share that with another driver:

banka/foo,bar=30MB

Now this 30 MB buffer is shared between drivers foo and bar.

The nice thing about this is that the driver will still only refer to region banka as setup by the platform code.

And in the more general case you can have two standard regions: dma and common. So drivers can rely on the presence of a dma region when allocating buffers.

What would make this even better is that CMA has the option to try and allocate additional memory on the fly if its memory pool becomes empty. E.g. if the dma pool is full, then it can try to do a kmalloc(..., GFP_KERNEL | __GFP_DMA).

This allows you to setup the dma and common regions with size 0. So allocating from the dma region would effectively be the same as doing a kmalloc. Unless the user sets up a dma area in the kernel parameters.

Obviously this is probably impossible if you need memory from specific memory banks, so this is something that is not available for every region.

The nice thing about this is that it is very flexible for end users. For example, most users of the ivtv driver wouldn't have to do anything since most of the time it is able to assign the necessary buffers. But some users have multiple ivtv-based capture boards in their PC, and then it can become harder to have ivtv obtain the needed buffers. In that case they can preallocate the buffers by setting dma/ivtv=500MB or something like that.

That would be a really nice feature...

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG, part of Cisco
#10 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Quoted 15 lines · On Wednesday 28 July 2010 11:04:56 Marek Szyprowski wrote:
Quoted 14 lines

Let me introduce one more example. As you may know we have 3 video-processor capture devices (Samsung FIMC) and a hardware codec (like Samsung MFC). FIMC can capture video data from camera sensor and accelerate common video processing tasks (like up/down scaling and color space conversion). Two FIMC and MFC are require for things like HD video encoding or decoding with online display/preview. This task require huge video buffers that are usually allocated and freed at the same time. The third FIMC can be used for gfx acceleration (color space conversion and scaling are quite common tasks in GUI). This latter task usually introduces a lot of memory fragmentation, as gfx surfaces are usually quite small (definitely smaller than HD frames or 8MPix picture from camera). It would be really wise to get that third FIMC device to use memory buffer that will be shared with 3D accelerator (which has quite similar usage scenarios and suffers from similar memory fragmentation).

Quoted 3 lines · Hans Verkuil

OK, I understand. And I assume both gfx and 3D acceleration need to use a specific region? If they can use any type of memory, then this might be more appropriate for kmalloc and friends.

I've been thinking about providing a "fake" region with a "fake" allocator which would allow in a generic way passing requests to kmalloc() and friends. Such regions could prove valuable for small allocations in things like 3D accelerator.

But as you've said, it's better to provide something small first and later add to it so I'm postponing implementation of this feature.

Note, however, that 3D accelerator does not operate only on small chunks of memory. A 1024x1024 texture is 1 Mipx. RGB makes it 3MiB. With mipmap it's 4MiB. Even 512x512 texture can reach 1MiB this way. It ma be impossible to allocate such chunks with just a kmalloc().

Quoted 7 lines
Quoted 7 lines

We don't want to allocate X buffers of Y MB memory each on boot. Instead we want to just reserve XX MB memory and then dynamically allocate buffers from it. This enables us to perform the following 2 tasks:
1. movie decoding in HD-quality (only one instance)
2. two instances of SD-quality movie decoding and SD-quality move encoding (example: video conference)

Quoted 6 lines
Quoted 2 lines

We know that these two use cases are exclusive, so they can use the same reserved memory.

When I said 'allocating X buffers of Y MB memory' I meant that you need to allocate a known amount of memory (X * Y MB in this case). So the boot args say e.g. dma=40MB and the driver just allocates X buffers from that region.

But the point is that driver does not allocate memory at boot time. If video codec would allocate memory at boot time no one else could use it even if the codec is not used. The idea is to make other devices use the memory when codec is idle. For instance, one could work on huge JPEG images and need buffers for a hardware JPEG codec.

Or have I misunderstood your point?

Quoted 2 lines

Switching to SD quality requires releasing those buffers and instead allocating a number of smaller buffers from the same region.

Our intention is that driver would allocate buffers only when needed so the buffers would be freed when video codec driver is released. So when the device is opened (whatever that would mean for a particular device) it would allocate enough memory for the requested task.

Quoted 2 lines

For these use-cases the allocator can be very simple and probably covers most use-cases.

Yes, this is our experience. The best-fit algorithm, even though simple, seem to handle use cases tested on our system with little fragmentation.

Quoted 4 lines

Anyway, I'm no expert on memory allocators and people on the linux-mm list are no doubt much more qualified to discuss this. My main concern is that of trying to add too much for a first release. It is simply easier to start simple and extend as needed. That makes it easier to be accepted in the mainline.

I'm trying to keep it as simple as possible :) still making it useful for us.

In particular we need a way to specify where different regions reside (different memory banks, etc.) as well as specify which drivers should use which regions. What's more, we need the notion of a "kind" of memory as one driver may need memory buffers from different regions (ie. MFC needs to allocate buffers from both banks).

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/contiguous-memory.txt new file +8 -0
+    2. CMA allows a run-time configuration of the memory regions it
+       will use to allocate chunks of memory from.  The set of memory
+       regions is given on command line so it can be easily changed
+       without the need for recompiling the kernel.
+
+       Each region has it's own size, alignment demand, a start
+       address (physical address where it should be placed) and an
+       allocator algorithm assigned to the region.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Documentation/contiguous-memory.txt line 70

Quoted 5 lines
Documentation/contiguous-memory.txt new file +5 -0
+
+       This means that there can be different algorithms running at
+       the same time, if different devices on the platform have
+       distinct memory usage characteristics and different algorithm
+       match those the best way.

Seems overengineering to me. Just ensure that the code can be extended later to such hypothetical scenarios. They are hypothetical, right?

1. Everyone seem to hate the command line interface that was present in the first and second version of the patch. As such, I've made it optional (via Kconfig option) in the third version (not posted yet), which unfortunately makes platform initialisation code longer and more complicated but hopefully more people will be happy. ;)

2. We need to specify size, alignment and start address so those are not hypothetical.

3. The algorithms are somehow hypothetical (we haven't tried using a different allocator as of you) but I think it's much easier to design the whole system with them in mind and implement them in the first version then later add code for them.

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Documentation/contiguous-memory.txt new file +7 -0
+    4. For greater flexibility and extensibility, the framework allows
+       device drivers to register private regions of reserved memory
+       which then may be used only by them.
+
+       As an effect, if a driver would not use the rest of the CMA
+       interface, it can still use CMA allocators and other
+       mechanisms.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Why would you? Is there an actual driver that will need this?

This feature has been added after posting v1 of this rfc/patch. Jonathan Corbet suggested in
<http://article.gmane.org/gmane.linux.kernel.mm/50689>
that viafb driver might register its own private memory and use cma just as an allocator.

I may also add that adding this actually made me refactor the code a bit making it more readable in the end I think. :)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

What I have seen in practice is that these drivers just
need X amount of contiguous memory on boot. Having just a single region (as it will be for most systems) to carve the buffers from is just as efficient if not more than creating separate regions for each driver. Only if you start freeing and reallocating memory later on will you get into trouble.

But if you do that, then you are trying to duplicate the behavior of the normal allocators in my opinion. I really don't think we want to go there.

Please note that kmalloc() was not designed to handle big chunks of memory and vmalloc() does not give a contiguous memory blocks. This is usually reason enough for a custom allocator that operates on a big region of memory reserved at boot time.

For instance, if some driver operates on buffers that are between 512 KiB and 4 MiB (as I've shown above such sizes could well be required for textures) it needs to reserve some big region of contiguous memory and then manage it by itself.

One of CMA's goals is to give a common API for drivers that need such allocators.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Documentation/contiguous-memory.txt line 101

Quoted 8 lines
Documentation/contiguous-memory.txt new file +8 -0
+       4a. Early in boot process, device drivers can also request the
+           CMA framework to a reserve a region of memory for them
+           which then will be used as a private region.
+
+           This way, drivers do not need to directly call bootmem,
+           memblock or similar early allocator but merely register an
+           early region and the framework will handle the rest
+           including choosing the right early allocator.

The whole concept of private regions seems unnecessary to me.

This particular thing was suggested by someone I think. Or maybe someone wrote something that make me think about it? Someone suggested that drivers may want to just grab some region of memory and have it for themselves. Even though I'd rather see them using the other set of CMA APIs but nonetheless it may prove useful for someone.

This is especially true for devices with their own memory which only their driver should have access to. I admit that it is a bit hypothetical though.

At any rate, with a changes made between the first and the second (this one) versions of the patch private regions were actually trivial to add. This merely mimics the way regions are reserved at boot time so the code is simply identical to what platform initialisation code may use. The only thing that make private regions special is the fact that they have no name.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It looks to me as if you tried to think of all possible hypothetical situations and write a framework for that.

Not exactly... The first version of the patch provided fewer features and this was mostly what we needed on our platform with maybe a few features that weren't a must.

After posting we received some comments and suggestions which made my change the code a bit making it more flexible and dynamic at the same time letting more features in.

Quoted 15 lines

Regarding regions and shared and per-driver buffers: I've been thinking about this a bit more and I have a proposal of my own.

There are two different aspects to this: first there is the hardware aspect: if the hardware needs memory from specific memory banks or with specific requirements (e.g. DMAable), then those regions should be setup in the platform code. There you know the memory sizes/alignments/etc. since that is hw dependent. The other reason is that drivers need to be able to tell CMA that they need to allocate from such regions. You can't have a driver refer to a region that is specified through kernel parameters, that would create a very ugly dependency.

The other aspect is how to setup buffers. A global buffer is simply setup by assigning a size to the region: "banka=20MB". Unless specified otherwise any driver that needs memory from banka will use that global banka buffer.

Quoted 8 lines

Alternatively, you can set aside memory from a region specifically for drivers:
banka/foo=30MB. This allocated 30 MB from region banka specifically for driver foo.

You can also share that with another driver:

banka/foo,bar=30MB

Now this 30 MB buffer is shared between drivers foo and bar.

Let me rephrase it to see if I got it correct:

You propose that platform will define what types of memory it has. For instance banka for a the first bank, bankb for the second memory bank, dma for DMA-able memory, etc. Those definitions would be merely informative and by themselves they would not reserve any memory.

Later, it would be possible to specify regions of memory of those types. For instance:

banka=20M; banka/foo,bar=30M

would register two regions in the memory type "banka" such that the first is 20 MiB and used by all drivers expect for driver foo and bar which would use the second region of 30 MiB?

Quoted 2 lines

The nice thing about this is that the driver will still only refer to region banka as setup by the platform code.

So the driver would request a memory type "banka" and then get a chunk from one of the abovementioned regions?

I somehow like the simplicity of that but I see some disadvantages:

1. Imagine a video decoder which for best performance should operate on some buffers from the first and some buffers from the second bank. However, if the buffers are from the incorrect bank it will still work, only slower. In such situations you cannot specify that when driver foo requests memory type "banka" then it should first try memory type "banka" but if allocation failed there try "bankb".

2. What if the device handled by the above driver were run on a platform with only one memory bank? The driver would still refer to "banka" and "bankb" but there would be no such types in the system.

3. What if there were one driver, initially written for platform X which used names "banka" and "bankb", and another driver, initially written for platform Y which used names "bank1" and "bank2". How would you make them work on a single platform with two memory banks?

4. This is hypothetical, but the "kind" defined by CMA could be used to specify characteristics that are not hardware dependent. For instance some driver could use kind "bulk" for some big, several MiB buffers and "control" for small less then MiB buffers. Regions for those kinds could be of the same type of memory but it could be valuable splitting those to two regions to minimise fragmentation.

Quoted 2 lines

And in the more general case you can have two standard regions: dma and common. So drivers can rely on the presence of a dma region when allocating buffers.

I think that driver should not care about or know region names at all.

Quoted 3 lines

What would make this even better is that CMA has the option to try and allocate additional memory on the fly if its memory pool becomes empty. E.g. if the dma pool is full, then it can try to do a kmalloc(..., GFP_KERNEL | __GFP_DMA).

As I've said somewhere above, I was thinking about something like it.

Quoted 7 lines

This allows you to setup the dma and common regions with size 0. So allocating from the dma region would effectively be the same as doing a kmalloc. Unless the user sets up a dma area in the kernel parameters.

Obviously this is probably impossible if you need memory from specific memory banks, so this is something that is not available for every region.

Quoted 8 lines

The nice thing about this is that it is very flexible for end users. For example, most users of the ivtv driver wouldn't have to do anything since most of the time it is able to assign the necessary buffers. But some users have multiple ivtv-based capture boards in their PC, and then it can become harder to have ivtv obtain the needed buffers. In that case they can preallocate the buffers by setting dma/ivtv=500MB or something like that.

That would be a really nice feature...

I think the main difference between your proposal and what is in CMA is that you propose that platform define types of memory and later on user will be able to define regions of given type of memory. This means that drivers would have to be aware of the names of the types and specify the type name witch each allocation.

The CMA solution however, lets drivers define their own kinds of memory and later on platform initialisation code map drivers with their kinds to regions.

Have I got it right?

Signature · 4 lines
Best regards,                                        _     _
| Humble Liege of Serenely Enlightened Majesty of  o' \,=./ `o
| Computer Science,  Michał "mina86" Nazarewicz       (o o)
+----[mina86*mina86.com]---[mina86*jabber.org]----ooO--(_)--Ooo--
#11 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

On Monday 02 August 2010 17:51:39 Michał Nazarewicz wrote:

<snip>

Quoted 14 lines
Quoted 7 lines
Quoted 7 lines

We don't want to allocate X buffers of Y MB memory each on boot. Instead we want to just reserve XX MB memory and then dynamically allocate buffers from it. This enables us to perform the following 2 tasks:
1. movie decoding in HD-quality (only one instance)
2. two instances of SD-quality movie decoding and SD-quality move encoding (example: video conference)

Quoted 6 lines
Quoted 2 lines

We know that these two use cases are exclusive, so they can use the same reserved memory.

When I said 'allocating X buffers of Y MB memory' I meant that you need to allocate a known amount of memory (X * Y MB in this case). So the boot args say e.g. dma=40MB and the driver just allocates X buffers from that region.

Quoted 7 lines

But the point is that driver does not allocate memory at boot time. If video codec would allocate memory at boot time no one else could use it even if the codec is not used. The idea is to make other devices use the memory when codec is idle. For instance, one could work on huge JPEG images and need buffers for a hardware JPEG codec.

Or have I misunderstood your point?

I think we are talking about the same thing. A region like dma=40MB would be shared by all drivers that want to allocate from it.

Quoted 7 lines
Quoted 2 lines

Switching to SD quality requires releasing those buffers and instead allocating a number of smaller buffers from the same region.

Our intention is that driver would allocate buffers only when needed so the buffers would be freed when video codec driver is released. So when the device is opened (whatever that would mean for a particular device) it would allocate enough memory for the requested task.

Right.

Quoted 5 lines
Quoted 2 lines

For these use-cases the allocator can be very simple and probably covers most use-cases.

Yes, this is our experience. The best-fit algorithm, even though simple, seem to handle use cases tested on our system with little fragmentation.

That's what I expected as well.

Quoted 30 lines
Quoted 4 lines

Anyway, I'm no expert on memory allocators and people on the linux-mm list are no doubt much more qualified to discuss this. My main concern is that of trying to add too much for a first release. It is simply easier to start simple and extend as needed. That makes it easier to be accepted in the mainline.

I'm trying to keep it as simple as possible :) still making it useful for us.

In particular we need a way to specify where different regions reside (different memory banks, etc.) as well as specify which drivers should use which regions. What's more, we need the notion of a "kind" of memory as one driver may need memory buffers from different regions (ie. MFC needs to allocate buffers from both banks).

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +8 -0
+    2. CMA allows a run-time configuration of the memory regions it
+       will use to allocate chunks of memory from.  The set of memory
+       regions is given on command line so it can be easily changed
+       without the need for recompiling the kernel.
+
+       Each region has it's own size, alignment demand, a start
+       address (physical address where it should be placed) and an
+       allocator algorithm assigned to the region.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+       This means that there can be different algorithms running at
+       the same time, if different devices on the platform have
+       distinct memory usage characteristics and different algorithm
+       match those the best way.

Seems overengineering to me. Just ensure that the code can be extended later to such hypothetical scenarios. They are hypothetical, right?

Quoted 5 lines

1. Everyone seem to hate the command line interface that was present in the first and second version of the patch. As such, I've made it optional (via Kconfig option) in the third version (not posted yet), which unfortunately makes platform initialisation code longer and more complicated but hopefully more people will be happy. ;)

I strongly recommend that it is simple dropped from the first version. That will increase the chances of getting it merged. And once merged, interfaces like this can be discussed at leisure.

Quoted 2 lines

2. We need to specify size, alignment and start address so those are not hypothetical.

Agreed. But this is platform code, you should not have to pass this info through boot args.

Quoted 4 lines

3. The algorithms are somehow hypothetical (we haven't tried using a different allocator as of you) but I think it's much easier to design the whole system with them in mind and implement them in the first version then later add code for them.

I agree with that as long as the extra code needed to do so it within limits. I've seen too often that people design for a future that never happens. That leads to code that is never used and will make it hard to future generations of developers to figure out what the purpose was of that code.

<snip>

Quoted 22 lines
Quoted 22 lines

Regarding regions and shared and per-driver buffers: I've been thinking about this a bit more and I have a proposal of my own.

There are two different aspects to this: first there is the hardware aspect: if the hardware needs memory from specific memory banks or with specific requirements (e.g. DMAable), then those regions should be setup in the platform code. There you know the memory sizes/alignments/etc. since that is hw dependent. The other reason is that drivers need to be able to tell CMA that they need to allocate from such regions. You can't have a driver refer to a region that is specified through kernel parameters, that would create a very ugly dependency.

The other aspect is how to setup buffers. A global buffer is simply setup by assigning a size to the region: "banka=20MB". Unless specified otherwise any driver that needs memory from banka will use that global banka buffer.

Alternatively, you can set aside memory from a region specifically for drivers:
banka/foo=30MB. This allocated 30 MB from region banka specifically for driver foo.

You can also share that with another driver:

banka/foo,bar=30MB

Quoted 8 lines
Quoted 1 line

Now this 30 MB buffer is shared between drivers foo and bar.

Let me rephrase it to see if I got it correct:

You propose that platform will define what types of memory it has. For instance banka for a the first bank, bankb for the second memory bank, dma for DMA-able memory, etc. Those definitions would be merely informative and by themselves they would not reserve any memory.

Right. It might be an option that they do reserve a minimal amount of memory. If you know that you always need at least X MB of memory to get the system running, then that might be useful.

Quoted 8 lines

Later, it would be possible to specify regions of memory of those types. For instance:

   banka=20M; banka/foo,bar=30M

would register two regions in the memory type "banka" such that the first is 20 MiB and used by all drivers expect for driver foo and bar which would use the second region of 30 MiB?

Right.

Quoted 5 lines
Quoted 2 lines

The nice thing about this is that the driver will still only refer to region banka as setup by the platform code.

So the driver would request a memory type "banka" and then get a chunk from one of the abovementioned regions?

Right.

Quoted 7 lines

I somehow like the simplicity of that but I see some disadvantages:

1. Imagine a video decoder which for best performance should operate on some buffers from the first and some buffers from the second bank. However, if the buffers are from the incorrect bank it will still work, only slower. In such situations you cannot specify that when driver foo requests memory type "banka" then it should first try memory type "banka" but if allocation failed there try "bankb".

Not quite sure I understand the problem here. Isn't that something for the driver to decide? If it can only work with buffers from banka, then it will just fail if it cannot allocate the required buffers. On the other hand, if it can also work with buffers from bankb if banka is full, then it can just use bankb as fallback.

This type of behavior is very much driver specific and as such should be done by the driver and not through user supplied kernel parameters IMHO.

Quoted 3 lines

2. What if the device handled by the above driver were run on a platform with only one memory bank? The driver would still refer to "banka" and "bankb" but there would be no such types in the system.

First of all, any driver that needs specific memory banks is highly platform specific and is extremely unlikely to work anywhere else.

But this can also handled in the driver itself. Either through config #ifdefs or by using e.g. a dma region as fallback.

Quoted 4 lines

3. What if there were one driver, initially written for platform X which used names "banka" and "bankb", and another driver, initially written for platform Y which used names "bank1" and "bank2". How would you make them work on a single platform with two memory banks?

Sorry, I don't understand the question. I think I would refer to my answer to the previous question, but I'm not sure if that covers this.

Quoted 6 lines

4. This is hypothetical, but the "kind" defined by CMA could be used to specify characteristics that are not hardware dependent. For instance some driver could use kind "bulk" for some big, several MiB buffers and "control" for small less then MiB buffers. Regions for those kinds could be of the same type of memory but it could be valuable splitting those to two regions to minimise fragmentation.

That's actually a good point. I can imagine this.

I would not implement this for a first version. But one way this could be done is by something like this:

dma/foo(kind)=20MB where '(kind)' is optional. The big problem I have with this is that this means that you need to know what 'kinds' of memory a particular driver needs.

This can always be added later. For an initial release I wouldn't do this.

Quoted 4 lines
Quoted 2 lines

And in the more general case you can have two standard regions: dma and common. So drivers can rely on the presence of a dma region when allocating buffers.

I think that driver should not care about or know region names at all.

A region is very similar to the last argument to kmalloc. And drivers most definitely need to know about regions, just like they need to specify the correct GFP flags.

In fact, it's the only thing that need to know.

Quoted 5 lines
Quoted 3 lines

What would make this even better is that CMA has the option to try and allocate additional memory on the fly if its memory pool becomes empty. E.g. if the dma pool is full, then it can try to do a kmalloc(..., GFP_KERNEL | __GFP_DMA).

As I've said somewhere above, I was thinking about something like it.

Cool.

Quoted 21 lines
Quoted 7 lines

This allows you to setup the dma and common regions with size 0. So allocating from the dma region would effectively be the same as doing a kmalloc. Unless the user sets up a dma area in the kernel parameters.

Obviously this is probably impossible if you need memory from specific memory banks, so this is something that is not available for every region.

Quoted 8 lines

The nice thing about this is that it is very flexible for end users. For example, most users of the ivtv driver wouldn't have to do anything since most of the time it is able to assign the necessary buffers. But some users have multiple ivtv-based capture boards in their PC, and then it can become harder to have ivtv obtain the needed buffers. In that case they can preallocate the buffers by setting dma/ivtv=500MB or something like that.

That would be a really nice feature...

I think the main difference between your proposal and what is in CMA is that you propose that platform define types of memory and later on user will be able to define regions of given type of memory. This means that drivers would have to be aware of the names of the types and specify the type name witch each allocation.

Quoted 4 lines

The CMA solution however, lets drivers define their own kinds of memory and later on platform initialisation code map drivers with their kinds to regions.

Have I got it right?

I think so, yes. The disadvantage of the CMA solution is that if you have a number of drivers, each with their own kinds of memory, you get very complex mappings. And remember that these drivers are not limited to the hardware inside the SoC, but can also include e.g. USB drivers. You can't predict what USB device the end user will connect to the device, so you would have to be able to handle any mapping that any USB driver might need.

I really think this is the wrong approach.

Regards,

Hans

Signature · 7 lines
Hans Verkuil - video4linux developer - sponsored by TANDBERG, part of Cisco

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#12 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Hello Hans,

I've just posted updated patchset. It changes the way regions are reserved somehow so our discussion is not entirely applicable to a new version I think.

I preserved the original "map" there. I came to a conclusion that your approach is not that different from what I had in mind but I noticed that with your syntax it's impossible to specify the order of regions to try. For instance that driver should first try region "foo" and then region "bar" and not the other way around.

I'm looking forward to hearing your comments on the newest version of CMA.

Signature · 4 lines
Best regards,                                        _     _
| Humble Liege of Serenely Enlightened Majesty of  o' \,=./ `o
| Computer Science,  Michał "mina86" Nazarewicz       (o o)
+----[mina86*mina86.com]---[mina86*jabber.org]----ooO--(_)--Ooo--
#13 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

In reply to Michal Nazarewicz

Documentation/contiguous-memory.txt line 20

Quoted 6 lines · Michal Nazarewicz
Documentation/contiguous-memory.txt new file +6 -0
+** Why is it needed?
+
+    Various devices on embedded systems have no scatter-getter and/or
+    IO map support and as such require contiguous blocks of memory to
+    operate.  They include devices such as cameras, hardware video
+    decoders and encoders, etc.

Yes, this is becoming quite a big problem - and many ARM SoCs suffer from the existing memory allocators being extremely inadequate for their use.

One of the areas I've been working on is sorting out the DMA coherent allocator so we don't violate the architecture requirements for ARMv6 and ARMv7 CPUs (which basically prohibits multiple mappings of memory with different attributes.)

One of the ideas that I've thought about for this is to reserve an amount of contiguous memory at boot time to fill the entire DMA coherent mapping, marking the memory in the main kernel memory map as 'no access', and allocate directly from the DMA coherent region.

However, discussing this with people who have the problem you're trying to solve indicates that they do not want to set aside an amount of memory as they perceive this to be a waste of resources.

This concern also applies to 'cma'.

include/linux/cma.h line 32

Quoted 7 lines
include/linux/cma.h new file +7 -0
+/*
+ * Don't call it directly, use cma_alloc(), cma_alloc_from() or
+ * cma_alloc_from_region().
+ */
+dma_addr_t __must_check
+__cma_alloc(const struct device *dev, const char *kind,
+	    size_t size, dma_addr_t alignment);

Does this really always return DMA-able memory (memory which can be DMA'd to/from without DMA-mapping etc?)

As it returns a dma_addr_t, it's returning a cookie for the memory which will be suitable for writing directly to the device 'dev' doing the DMA. (NB: DMA addresses may not be the same as physical addresses, especially if the device is on a downstream bus. We have ARM platforms which have different bus offsets.)

How does one obtain the CPU address of this memory in order for the CPU to access it?

include/linux/cma.h line 58

Quoted 5 lines
include/linux/cma.h new file +5 -0
+static inline dma_addr_t __must_check
+cma_alloc(const struct device *dev, const char *kind,
+	  size_t size, dma_addr_t alignment)
+{
+	return dev ? -EINVAL : __cma_alloc(dev, kind, size, alignment);

So I can't use this to allocate memory for anything but a NULL device?

include/linux/cma.h line 94

Quoted 4 lines
include/linux/cma.h new file +4 -0
+static inline int
+cma_info(struct cma_info *info, const struct device *dev, const char *kind)
+{
+	return dev ? -EINVAL : __cma_info(info, dev, kind);

This won't return information for anything but a NULL device?

#14 lore raw

RE: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Hello,

Quoted 17 lines · Russell King - ARM Linux

Documentation/contiguous-memory.txt line 20

Quoted 6 lines · Michal Nazarewicz
Documentation/contiguous-memory.txt new file +6 -0
+** Why is it needed?
+
+    Various devices on embedded systems have no scatter-getter and/or
+    IO map support and as such require contiguous blocks of memory to
+    operate.  They include devices such as cameras, hardware video
+    decoders and encoders, etc.

Yes, this is becoming quite a big problem - and many ARM SoCs suffer from the existing memory allocators being extremely inadequate for their use.

One of the areas I've been working on is sorting out the DMA coherent allocator so we don't violate the architecture requirements for ARMv6 and ARMv7 CPUs (which basically prohibits multiple mappings of memory with different attributes.)

Quoted 8 lines

One of the ideas that I've thought about for this is to reserve an amount of contiguous memory at boot time to fill the entire DMA coherent mapping, marking the memory in the main kernel memory map as 'no access', and allocate directly from the DMA coherent region.

However, discussing this with people who have the problem you're trying to solve indicates that they do not want to set aside an amount of memory as they perceive this to be a waste of resources.

Assuming your board have only 128MB of physical memory (quite common case for some embedded boards), leaving 16MB unused just for DMA coherent area is a huge waste imho.

Quoted 1 line

This concern also applies to 'cma'.

Yes, we know. We plan to recover some of that 'wasted' memory by providing a way to allocate some kind of virtual swap device on it. This is just an idea, no related works has been started yet.

Quoted 12 lines

include/linux/cma.h line 32

Quoted 8 lines
include/linux/cma.h new file +7 -0
+/*
+ * Don't call it directly, use cma_alloc(), cma_alloc_from() or
+ * cma_alloc_from_region().
+ */
+dma_addr_t __must_check
+__cma_alloc(const struct device *dev, const char *kind,
+	    size_t size, dma_addr_t alignment);

Does this really always return DMA-able memory (memory which can be DMA'd to/from without DMA-mapping etc?)

Quoted 8 lines

As it returns a dma_addr_t, it's returning a cookie for the memory which will be suitable for writing directly to the device 'dev' doing the DMA. (NB: DMA addresses may not be the same as physical addresses, especially if the device is on a downstream bus. We have ARM platforms which have different bus offsets.)

How does one obtain the CPU address of this memory in order for the CPU to access it?

Right, we did not cover such case. In CMA approach we tried to separate memory allocation from the memory mapping into user/kernel space. Mapping a buffer is much more complicated process that cannot be handled in a generic way, so we decided to leave this for the device drivers. Usually video processing devices also don't need in-kernel mapping for such buffers at all.

Best regards
--
Marek Szyprowski
Samsung Poland R&D Center

maintainer
#15 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Quoted 9 lines · On Tue, 27 Jul 2010 14:45:58 +0200 Marek Szyprowski <[email protected]>…
Quoted 2 lines

How does one obtain the CPU address of this memory in order for the CPU to access it?

Right, we did not cover such case. In CMA approach we tried to separate memory allocation from the memory mapping into user/kernel space. Mapping a buffer is much more complicated process that cannot be handled in a generic way, so we decided to leave this for the device drivers. Usually video processing devices also don't need in-kernel mapping for such buffers at all.

Still...that *is* why I suggested an interface which would return both the DMA address and a kernel-space virtual address, just like the DMA API does... Either that, or just return the void * kernel address and let drivers do the DMA mapping themselves. Returning only the dma_addr_t address will make the interface difficult to use in many situations.

jon

#16 lore raw

RE: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

Hello,

Quoted 13 lines · Jonathan Corbet
Quoted 9 lines · On Tue, 27 Jul 2010 14:45:58 +0200 Marek Szyprowski <[email protected]>…
Quoted 2 lines

How does one obtain the CPU address of this memory in order for the CPU to access it?

Right, we did not cover such case. In CMA approach we tried to separate memory allocation from the memory mapping into user/kernel space. Mapping a buffer is much more complicated process that cannot be handled in a generic way, so we decided to leave this for the device drivers. Usually video processing devices also don't need in-kernel mapping for such buffers at all.

Quoted 6 lines

Still...that *is* why I suggested an interface which would return both the DMA address and a kernel-space virtual address, just like the DMA API does... Either that, or just return the void * kernel address and let drivers do the DMA mapping themselves. Returning only the dma_addr_t address will make the interface difficult to use in many situations.

As I said, drivers usually don't need in-kernel mapping for video buffers. Is there really a need for creating such mapping?

Best regards
--
Marek Szyprowski
Samsung Poland R&D Center

#17 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

In reply to Jonathan Corbet
Quoted 13 lines · On Tue, 27 Jul 2010 06:58:42 -0600 Jonathan Corbet <[email protected]> wrote:
Quoted 9 lines · On Tue, 27 Jul 2010 14:45:58 +0200 Marek Szyprowski <[email protected]>…
Quoted 2 lines

How does one obtain the CPU address of this memory in order for the CPU to access it?

Right, we did not cover such case. In CMA approach we tried to separate memory allocation from the memory mapping into user/kernel space. Mapping a buffer is much more complicated process that cannot be handled in a generic way, so we decided to leave this for the device drivers. Usually video processing devices also don't need in-kernel mapping for such buffers at all.

Quoted 3 lines

Still...that *is* why I suggested an interface which would return both the DMA address and a kernel-space virtual address, just like the DMA API does... Either that, or just return the void * kernel address and

The DMA API for coherent memory (dma_alloc_coherent) returns both an DMA address and a kernel-space virtual address because it does both allocation and mapping.

However, other DMA API (dma_map_*) returns only an DMA address because it does only mapping.

I think that if we need new API for coherent memory, we could unify it with the DMA API for coherent memory.

IMO, it's cleaner to having two separate APIs for allocation and mapping (except for coherent memory). The drivers have been working in that way.

#18 lore raw

Re: [PATCHv2 2/4] mm: cma: Contiguous Memory Allocator added

In reply to Russell King - ARM Linux
Quoted 5 lines
Quoted 5 lines
include/linux/cma.h new file +5 -0
+static inline dma_addr_t __must_check
+cma_alloc(const struct device *dev, const char *kind,
+	  size_t size, dma_addr_t alignment)
+{
+	return dev ? -EINVAL : __cma_alloc(dev, kind, size, alignment);
Quoted 8 lines · Russell King - ARM Linux

So I can't use this to allocate memory for anything but a NULL device?

include/linux/cma.h line 94

Quoted 4 lines
include/linux/cma.h new file +4 -0
+static inline int
+cma_info(struct cma_info *info, const struct device *dev, const char *kind)
+{
+	return dev ? -EINVAL : __cma_info(info, dev, kind);

This won't return information for anything but a NULL device?

Obviously a bug. Thanks for spotting that.

Signature · 4 lines
Best regards,                                        _     _
| Humble Liege of Serenely Enlightened Majesty of  o' \,=./ `o
| Computer Science,  Michał "mina86" Nazarewicz       (o o)
+----[mina86*mina86.com]---[mina86*jabber.org]----ooO--(_)--Ooo--