Skip to contentSkip to conversation

ARM: OMPA4+: is it expected dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(64)); to fail?

DISCUSSION [email protected] 3 participants 7 messages Mar 5, 2015 – Mar 10, 2015 Memory management

lore
Share & tools
lore
#0 lore raw

Hi All,

Now I can see very interesting behavior related to dma_coerce_mask_and_coherent() and friends which I'd like to explain and clarify.

Below is set of questions I have (why - I explained below):
- Is expected dma_coerce_mask_and_coherent(DMA_BIT_MASK(64)) and friends to fail on 32 bits HW?

- What is expected value for max_pfn: max_phys_pfn or max_phys_pfn + 1?

- What is expected value for struct memblock_region->size: mem_range_size or mem_range_size - 1?

- What is expected value to be returned by memblock_end_of_DRAM():
@base + @size(max_phys_addr + 1) or @base + @size - 1(max_phys_addr)?

I'm working with BeaglBoard-X15 (AM572x/DRA7xx) board and have following code in OMAP ASOC driver which is failed SOMETIMES during the boot with error -EIO.
=== to omap-pcm.c:
omap_pcm_new() {
...

	ret = dma_coerce_mask_and_coherent(card->dev, DMA_BIT_MASK(64));
^^ failed sometimes
	if (ret)
		return ret;
}

What I can see is that dma_coerce_mask_and_coherent() and etc may fail or succeed depending on - max_pfn value.
-> max_pfn value depends on memblock configuration
max_pfn = max_high = PFN_DOWN(memblock_end_of_DRAM());
|- PFN_DOWN(memblock.memory.regions[last_idx].base + memblock.memory.regions[last_idx].size)

-> memblock configuration depends on
a) CONFIG_ARM_LPAE=y|n (my system really works with 32 bit address space)
b) RAM configuration

Example 1 CONFIG_ARM_LPAE=n:

	memory {
		device_type = "memory";
		reg = <0x80000000 0x60000000>; /* 1536 MB */
	};

memblock will be configured as:

	memory.cnt  = 0x1
	memory[0x0]     [0x00000080000000-0x000000dfffffff], 0x60000000 bytes flags: 0x0
							     ^^^^^^^^^^

max_pfn = 0x000E0000

Example 2 CONFIG_ARM_LPAE=n:

	memory {
		device_type = "memory";
		reg = <0x80000000 0x80000000>;
	};

memblock will be configured as:

	memory.cnt  = 0x1
	memory[0x0]     [0x00000080000000-0x000000fffffffe], 0x7fffffff bytes flags: 0x0
							     ^^^^^^^^^^

max_pfn = 0x000FFFFF

Example 3 CONFIG_ARM_LPAE=y (but system really works with 32 bit address space):

	memory {
		device_type = "memory";
		reg = <0x80000000 0x80000000>;
	};

memblock will be configured as:

	memory.cnt  = 0x1
	memory[0x0]     [0x00000080000000-0x000000ffffffff], 0x80000000 bytes flags: 0x0
							     ^^^^^^^^^^

max_pfn = 0x00100000

The dma_coerce_mask_and_coherent() will fail in case 'Example 3' and succeed in cases 1,2. dma-mapping.c --> __dma_supported()

	if (sizeof(mask) != sizeof(dma_addr_t) && <== true for all OMAP4+
	    mask > (dma_addr_t)~0 &&		<== true for DMA_BIT_MASK(64)
	    dma_to_pfn(dev, ~0) < max_pfn) {  <== true only for Example 3

I've tracked down patch which changes memblock behavior to:
commit eb18f1b5bfb99b1d7d2f5d792e6ee5c9b7d89330

Author: Tejun Heo <[email protected]>
Date:   Thu Dec 8 10:22:07 2011 -0800

memblock: Make memblock functions handle overflowing range @size

This commit is pretty old :( and it doesn't takes into account LPAE mode where phys_addr_t is 64 bit, but physically accessible addresses <= 40 bit (memblock_cap_size()).

The issue with omap-pcm was simply fixed by using DMA_BIT_MASK(32), but It seems problem is wider and above behavior of dma_set_maskX() and memblock confused me a bit.

I'd be very appreciated for any comments/clarification on questions I've listed at the beginning of my e-mail - there are no patches from my side as I'd like to understand expected behavior of the kernel first (especially taking into account that any memblock changes might affect on@least half of arches).

Thanks.

Additional info:
memblock: Make memblock functions handle overflowing range @size
https://lkml.org/lkml/2011/7/26/235
[alsa-devel] [PATCH] ASoC: omap-pcm: Lower the dma coherent mask to 32bits
http://mailman.alsa-project.org/pipermail/alsa-devel/2013-December/069817.html

Signature · 2 lines
regards,
-grygorii
#1 lore raw
Quoted 5 lines · Grygorii.Strashko at linaro.org

Now I can see very interesting behavior related to dma_coerce_mask_and_coherent() and friends which I'd like to explain and clarify.

Below is set of questions I have (why - I explained below):
- Is expected dma_coerce_mask_and_coherent(DMA_BIT_MASK(64)) and friends to fail on 32 bits HW?

Not really.

Quoted 1 line

- What is expected value for max_pfn: max_phys_pfn or max_phys_pfn + 1?

mm/page_owner.c:

        /* Find an allocated page */
        for (; pfn < max_pfn; pfn++) {
drivers/base/platform.c:    u32 low_totalram = ((max_pfn - 1) << PAGE_SHIFT);
drivers/base/platform.c:    u32 high_totalram = ((max_pfn - 1) >> (32 - PAGE_SHIFT));

So, there's ample evidence that max_pfn is one more than the greatest pfn which may be used in the system.

Quoted 1 line

- What is expected value for struct memblock_region->size: mem_range_size or mem_range_size - 1?

A size is a size - it's a number of bytes contained within the region. If it is value 1, then there is exactly one byte in the region. If there are 0x7fffffff, then there are 2G-1 bytes in the region, not 2G.

Quoted 2 lines

- What is expected value to be returned by memblock_end_of_DRAM():
@base + @size(max_phys_addr + 1) or @base + @size - 1(max_phys_addr)?

The last address plus one in the system. However, there's a problem here. On a 32-bit system, phys_addr_t may be 32-bit. If it is 32-bit, then "last address plus one" could be zero, which makes no sense. Hence, it is artificially reduced to 0xfffff000, thereby omitting the final page.

Quoted 10 lines

Example 3 CONFIG_ARM_LPAE=y (but system really works with 32 bit address space):

	memory {
		device_type = "memory";
		reg = <0x80000000 0x80000000>;
	};

memblock will be configured as:

	memory.cnt  = 0x1
	memory[0x0]     [0x00000080000000-0x000000ffffffff], 0x80000000 bytes flags: 0x0
							     ^^^^^^^^^^
Quoted 7 lines

max_pfn = 0x00100000

The dma_coerce_mask_and_coherent() will fail in case 'Example 3' and succeed in cases 1,2. dma-mapping.c --> __dma_supported()

	if (sizeof(mask) != sizeof(dma_addr_t) && <== true for all OMAP4+
	    mask > (dma_addr_t)~0 &&		<== true for DMA_BIT_MASK(64)
	    dma_to_pfn(dev, ~0) < max_pfn) {  <== true only for Example 3

Hmm, I think this may make more sense to be "< max_pfn - 1" here, as that would be better suited to our intention.

The result of dma_to_pfn(dev, ~0) is the maximum PFN which we could address via DMA, but we're comparing it with the maximum PFN in the system plus 1 - so we need to subtract one from it.

Please think about this and test this out; I'm not back to normal yet (post-op) so I could very well not be thinking straight yet.

Thanks.

Signature · 2 lines
FTTC broadband for 0.8mile line: [email protected] down 400kbps up
according to speedtest.net.
#2 lore raw

Hi Russell,

Quoted 18 lines · Russell King - ARM Linux
Quoted 5 lines · Grygorii.Strashko at linaro.org

Now I can see very interesting behavior related to dma_coerce_mask_and_coherent() and friends which I'd like to explain and clarify.

Below is set of questions I have (why - I explained below):
- Is expected dma_coerce_mask_and_coherent(DMA_BIT_MASK(64)) and friends to fail on 32 bits HW?

Not really.

Quoted 1 line

- What is expected value for max_pfn: max_phys_pfn or max_phys_pfn + 1?

mm/page_owner.c:

         /* Find an allocated page */
         for (; pfn < max_pfn; pfn++) {
drivers/base/platform.c:    u32 low_totalram = ((max_pfn - 1) << PAGE_SHIFT);
drivers/base/platform.c:    u32 high_totalram = ((max_pfn - 1) >> (32 - PAGE_SHIFT));
Quoted 8 lines

So, there's ample evidence that max_pfn is one more than the greatest pfn which may be used in the system.

Quoted 1 line

- What is expected value for struct memblock_region->size: mem_range_size or mem_range_size - 1?

A size is a size - it's a number of bytes contained within the region. If it is value 1, then there is exactly one byte in the region. If there are 0x7fffffff, then there are 2G-1 bytes in the region, not 2G.

Thanks - it seems clear now.

Quoted 7 lines
Quoted 2 lines

- What is expected value to be returned by memblock_end_of_DRAM():

   @base + @size(max_phys_addr + 1) or @base + @size - 1(max_phys_addr)?

The last address plus one in the system. However, there's a problem here. On a 32-bit system, phys_addr_t may be 32-bit. If it is 32-bit, then "last address plus one" could be zero, which makes no sense. Hence, it is artificially reduced to 0xfffff000, thereby omitting the final page.

^ this part seems not fully true now, because for ARM32 + DT the fdt.c->early_init_dt_add_memory_arch() is called instead of arm_add_memory()
and it works in a different way a bit.

For example, I don't see below message when reg = <0x80000000 0x80000000>:
"Truncating memory at 0x80000000 to fit in 32-bit physical address space"

instead memblock silently configured as
memory.cnt = 0x1
memory[0x0].base = 0x80000000
memory[0x0].size = 0x7fffffff

Quoted 19 lines
Quoted 13 lines

Example 3 CONFIG_ARM_LPAE=y (but system really works with 32 bit address space):

	memory {
		device_type = "memory";
		reg = <0x80000000 0x80000000>;
	};
   memblock will be configured as:
	memory.cnt  = 0x1
	memory[0x0]     [0x00000080000000-0x000000ffffffff], 0x80000000 bytes flags: 0x0
							     ^^^^^^^^^^
   max_pfn = 0x00100000
Quoted 5 lines

The dma_coerce_mask_and_coherent() will fail in case 'Example 3' and succeed in cases 1,2. dma-mapping.c --> __dma_supported()

	if (sizeof(mask) != sizeof(dma_addr_t) && <== true for all OMAP4+
	    mask > (dma_addr_t)~0 &&		<== true for DMA_BIT_MASK(64)
	    dma_to_pfn(dev, ~0) < max_pfn) {  <== true only for Example 3
Quoted 6 lines

Hmm, I think this may make more sense to be "< max_pfn - 1" here, as that would be better suited to our intention.

The result of dma_to_pfn(dev, ~0) is the maximum PFN which we could address via DMA, but we're comparing it with the maximum PFN in the system plus 1 - so we need to subtract one from it.

Ok. I'll try it.

Quoted 3 lines

Please think about this and test this out; I'm not back to normal yet (post-op) so I could very well not be thinking straight yet.

Thanks for your comments. I hope you feel better.

Signature · 2 lines
regards,
-grygorii
#5 lore raw
In reply to [email protected]
Quoted 7 lines · On Thursday 05 March 2015 20:55:07 Grygorii.Strashko at linaro.org wrote:

Hi All,

Now I can see very interesting behavior related to dma_coerce_mask_and_coherent() and friends which I'd like to explain and clarify.

Below is set of questions I have (why - I explained below):
- Is expected dma_coerce_mask_and_coherent(DMA_BIT_MASK(64)) and friends to fail on 32 bits HW?

No. dma_coerce_mask_and_coherent() is meant to ignore the actual mask. It's usually considered a bug to use this function for that reason.

Quoted 13 lines

- What is expected value for max_pfn: max_phys_pfn or max_phys_pfn + 1?

- What is expected value for struct memblock_region->size: mem_range_size or mem_range_size - 1?

- What is expected value to be returned by memblock_end_of_DRAM():
@base + @size(max_phys_addr + 1) or @base + @size - 1(max_phys_addr)?

I'm working with BeaglBoard-X15 (AM572x/DRA7xx) board and have following code in OMAP ASOC driver which is failed SOMETIMES during the boot with error -EIO.
=== to omap-pcm.c:
omap_pcm_new() {
...

Quoted 5 lines
	ret = dma_coerce_mask_and_coherent(card->dev, DMA_BIT_MASK(64));
^^ failed sometimes
	if (ret)
		return ret;
}

The code should be fixed to use dma_set_mask_and_coherent(), which is expected to fail if the bus is incapable of addressing all RAM within the mask.

Quoted 4 lines

I'd be very appreciated for any comments/clarification on questions I've listed at the beginning of my e-mail - there are no patches from my side as I'd like to understand expected behavior of the kernel first (especially taking into account that any memblock changes might affect on at least half of arches).

Is the device you have actually 64-bit capable?

Is the bus it is connected to 64-bit wide?

Does the dma-ranges property of the parent bus reflect the correct address width?

Arnd

#3 lore raw
In reply to [email protected]
Quoted 10 lines · Grygorii.Strashko at linaro.org

Hi Russell,

Quoted 7 lines · Russell King - ARM Linux
Quoted 5 lines · Grygorii.Strashko at linaro.org

The dma_coerce_mask_and_coherent() will fail in case 'Example 3' and succeed in cases 1,2. dma-mapping.c --> __dma_supported()

	if (sizeof(mask) != sizeof(dma_addr_t) && <== true for all OMAP4+
	    mask > (dma_addr_t)~0 &&		<== true for DMA_BIT_MASK(64)
	    dma_to_pfn(dev, ~0) < max_pfn) {  <== true only for Example 3
Quoted 8 lines
Quoted 6 lines

Hmm, I think this may make more sense to be "< max_pfn - 1" here, as that would be better suited to our intention.

The result of dma_to_pfn(dev, ~0) is the maximum PFN which we could address via DMA, but we're comparing it with the maximum PFN in the system plus 1 - so we need to subtract one from it.

Ok. I'll try it.

Any news on this - I think it is a real off-by-one bug which we should fix in any case.

Thanks.

Signature · 2 lines
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.
#4 lore raw

Hi Russell,

Quoted 12 lines · Russell King - ARM Linux
Quoted 11 lines · Grygorii.Strashko at linaro.org
Quoted 10 lines · Russell King - ARM Linux
Quoted 5 lines · Grygorii.Strashko at linaro.org

The dma_coerce_mask_and_coherent() will fail in case 'Example 3' and succeed in cases 1,2. dma-mapping.c --> __dma_supported()

	if (sizeof(mask) != sizeof(dma_addr_t) && <== true for all OMAP4+
	    mask > (dma_addr_t)~0 &&		<== true for DMA_BIT_MASK(64)
	    dma_to_pfn(dev, ~0) < max_pfn) {  <== true only for Example 3

Hmm, I think this may make more sense to be "< max_pfn - 1" here, as that would be better suited to our intention.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The result of dma_to_pfn(dev, ~0) is the maximum PFN which we could address via DMA, but we're comparing it with the maximum PFN in the system plus 1 - so we need to subtract one from it.

Ok. I'll try it.

Any news on this - I think it is a real off-by-one bug which we should fix in any case.

Sorry for delay, there was a day-off on my side.

As per my test results - with above change
dma_coerce_mask_and_coherent(DMA_BIT_MASK(64)) and friends will succeed always.

=========== Test results:

==== Test case 1:
Input data:
- RAM: start = 0x80000000 size = 0x80000000
- CONFIG_ARM_LPAE=n and sizeof(phys_addr_t) = 4

a) NO changes:
memory registered within memblock as:
memory.cnt = 0x1
memory[0x0] [0x00000080000000-0x000000fffffffe], 0x7fffffff bytes flags: 0x0

max_pfn = 0xFFFFF
max_mapnr = 0x7FFFF

dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); -- succeeded

b) with change in __dma_supported():

        if (sizeof(mask) != sizeof(dma_addr_t) &&
            mask > (dma_addr_t)~0 &&
-           dma_to_pfn(dev, ~0) < max_pfn) {
+           dma_to_pfn(dev, ~0) < (max_pfn - 1)) {
                if (warn) {

memory registered within memblock as:
memory.cnt = 0x1
memory[0x0] [0x00000080000000-0x000000fffffffe], 0x7fffffff bytes flags: 0x0

max_pfn = 0xFFFFF
max_mapnr = 0x7FFFF

dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); -- succeeded

==== Test case 2:
Input data:
- RAM: start = 0x80000000 size = 0x80000000
- CONFIG_ARM_LPAE=y and sizeof(phys_addr_t) = 8

a) NO changes:
memory registered within memblock as:
memory.cnt = 0x1
memory[0x0] [0x00000080000000-0x000000ffffffff], 0x80000000 bytes flags: 0x0

max_pfn = 0x100000
max_mapnr = 0x80000

dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); -- failed

[    5.468470] asoc-simple-card sound at 0: Coherent DMA mask 0xffffffffffffffff is larger than dma_addr_t allows
[    5.478706] asoc-simple-card sound at 0: Driver did not use or check the return value from dma_set_coherent_mask()?
[    5.496620] davinci-mcasp 48468000.mcasp: ASoC: pcm constructor failed: -5
[    5.503844] asoc-simple-card sound@0: ASoC: can't create pcm davinci-mcasp.0-tlv320aic3x-hifi :-5

b) with change in __dma_supported():

        if (sizeof(mask) != sizeof(dma_addr_t) &&
            mask > (dma_addr_t)~0 &&
-           dma_to_pfn(dev, ~0) < max_pfn) {
+           dma_to_pfn(dev, ~0) < (max_pfn - 1)) {
                if (warn) {

memory registered within memblock as:
memory.cnt = 0x1
memory[0x0] [0x00000080000000-0x000000ffffffff], 0x80000000 bytes flags: 0x0

max_pfn = 0x100000
max_mapnr = 0x80000

dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); -- succeeded

regards,
-grygorii

Signature · 2 lines
regards,
-grygorii
#6 lore raw
In reply to Arnd Bergmann

Hi Arnd,

Quoted 34 lines · Arnd Bergmann
Quoted 7 lines · On Thursday 05 March 2015 20:55:07 Grygorii.Strashko at linaro.org wrote:

Hi All,

Now I can see very interesting behavior related to dma_coerce_mask_and_coherent() and friends which I'd like to explain and clarify.

Below is set of questions I have (why - I explained below):
- Is expected dma_coerce_mask_and_coherent(DMA_BIT_MASK(64)) and friends to fail on 32 bits HW?

No. dma_coerce_mask_and_coherent() is meant to ignore the actual mask. It's usually considered a bug to use this function for that reason.

Quoted 13 lines

- What is expected value for max_pfn: max_phys_pfn or max_phys_pfn + 1?

- What is expected value for struct memblock_region->size: mem_range_size or mem_range_size - 1?

- What is expected value to be returned by memblock_end_of_DRAM():

   @base + @size(max_phys_addr + 1) or @base + @size - 1(max_phys_addr)?

I'm working with BeaglBoard-X15 (AM572x/DRA7xx) board and have following code in OMAP ASOC driver which is failed SOMETIMES during the boot with error -EIO.
=== to omap-pcm.c:
omap_pcm_new() {
...

Quoted 5 lines
	ret = dma_coerce_mask_and_coherent(card->dev, DMA_BIT_MASK(64));
^^ failed sometimes
	if (ret)
		return ret;
}

The code should be fixed to use dma_set_mask_and_coherent(), which is expected to fail if the bus is incapable of addressing all RAM within the mask.

Quoted 8 lines
Quoted 4 lines

I'd be very appreciated for any comments/clarification on questions I've listed at the beginning of my e-mail - there are no patches from my side as I'd like to understand expected behavior of the kernel first (especially taking into account that any memblock changes might affect on at least half of arches).

Is the device you have actually 64-bit capable?

Is the bus it is connected to 64-bit wide?

As I mentioned before - The device was fixed by switching to use 32 bit mask "The issue with omap-pcm was simply fixed by using DMA_BIT_MASK(32), ".

Quoted 2 lines

Does the dma-ranges property of the parent bus reflect the correct address width?

dma-ranges is not used and all devices are created with default mask DMA_BIT_MASK(32);

My goal was to clarify above questions (first of all), because on my HW I can see different values of max_pfn, max_mapnr and memblock configuration depending on CONFIG_ARM_LPAE=n|y and when RAM is defined as: start = 0x80000000 size = 0x80000000. (and also between kernels 3.14 and LKML).

Looks like such RAM configuration is a corner case, which is not always handled as expected (and how is it expected to be handled?).
For example:
before commit ARM: 8025/1: Get rid of meminfo
- registered RAM start = 0x80000000 size = 0x80000000 will be adjusted by arm_add_memory() and final RAM configuration will be start = 0x80000000 size = 0x7FFFF000 after this commit:
- code will try to register start = 0x80000000 size = 0x80000000, but memblock will adjust it to start = 0x80000000 size = 0x7fffffff.

Signature · 2 lines
regards,
-grygorii