vmalloc_reserve with no highmem

PATCH Florian Fainelli 4 participants 8 messages Oct 19, 2015 – Oct 25, 2015 Arm (32-bit)

Share & tools

Florian Fainelli

author

2015-10-19 23:20 UTC #0 lore raw

Hi Russell, Laura,

Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.

What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.

The maths look like this for this particular board (512MB)

high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000

so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000

in sanity_check_meminfo(), high_memory is unconditionally assigned with arm_lowmem_limit's VA.

The following quick and dirty patch seems to do it for me, but I am not confident this is remotely the correct approach here:

arch/arm/mm/mmu.c +8 -1

index 14428d2..e196ea4 100644
@@ -1196,7 +1211,14 @@ void __init sanity_check_meminfo(void)
        }
 #endif
        meminfo.nr_banks = j;
-       high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       if (vmalloc_limit > arm_lowmem_limit)
+               high_memory = vmalloc_min - VMALLOC_OFFSET;
+       else
+               high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       pr_info("%s: high_memory: 0x%p, arm_lowmem_limit: 0x%llx\n",
+               __func__, high_memory, arm_lowmem_limit);

BTW, even when there is highmem available, setting vmalloc= on the command-line is off by VMALLOC_OFFSET, the way early_vmalloc() computes things, is that intended?

Thanks!

Signature · 1 line

Florian

Laura Abbott

2015-10-20 18:52 UTC #1 lore raw

Quoted 50 lines · Florian Fainelli

Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000
in sanity_check_meminfo(), high_memory is unconditionally assigned with arm_lowmem_limit's VA.
The following quick and dirty patch seems to do it for me, but I am not confident this is remotely the correct approach here:
arch/arm/mm/mmu.c +8 -1
index 14428d2..e196ea4 100644
@@ -1196,7 +1211,14 @@ void __init sanity_check_meminfo(void)
         }
  #endif
         meminfo.nr_banks = j;
-       high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       if (vmalloc_limit > arm_lowmem_limit)
+               high_memory = vmalloc_min - VMALLOC_OFFSET;
+       else
+               high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       pr_info("%s: high_memory: 0x%p, arm_lowmem_limit: 0x%llx\n",
+               __func__, high_memory, arm_lowmem_limit);
 
BTW, even when there is highmem available, setting vmalloc= on the command-line is off by VMALLOC_OFFSET, the way early_vmalloc() computes things, is that intended?

Quoted 2 lines

Thanks!

Which baseline are you testing off of? Meminfo was removed a while ago so I'm not sure where the meminfo.nr_banks context is coming from.

This is how arm currently defines VMALLOC ranges

#define VMALLOC_OFFSET          (8*1024*1024)
#define VMALLOC_START           (((unsigned long)high_memory + VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1))
#define VMALLOC_END             0xff000000UL

VMALLOC_START is based off of the value of high_memory. high_memory is supposed to be the limit of the kernel direct mapped ram region so setting it to something above that is asking for trouble. It seems like this is working as intended. You'd have to decouple VMALLOC_START and high_memory to make this work properly.

Having a vmalloc_start be greater than the end of direct mapped RAM seems a little unusual. This would be leaving a hole in the virtual space. Is this what you are intending or is there another use case for adjusting the vmalloc space as you described?

Thanks,
Laura

Florian Fainelli

author

2015-10-20 19:17 UTC #2 lore raw

Quoted 51 lines · Laura Abbott

Quoted 50 lines · Florian Fainelli
Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000
in sanity_check_meminfo(), high_memory is unconditionally assigned with arm_lowmem_limit's VA.
The following quick and dirty patch seems to do it for me, but I am not confident this is remotely the correct approach here:
arch/arm/mm/mmu.c +8 -1
index 14428d2..e196ea4 100644
@@ -1196,7 +1211,14 @@ void __init sanity_check_meminfo(void)
         }
  #endif
         meminfo.nr_banks = j;
-       high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       if (vmalloc_limit > arm_lowmem_limit)
+               high_memory = vmalloc_min - VMALLOC_OFFSET;
+       else
+               high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       pr_info("%s: high_memory: 0x%p, arm_lowmem_limit: 0x%llx\n",
+               __func__, high_memory, arm_lowmem_limit);
 
BTW, even when there is highmem available, setting vmalloc= on the command-line is off by VMALLOC_OFFSET, the way early_vmalloc() computes things, is that intended?

Quoted 6 lines

Quoted 2 lines
Thanks!
Which baseline are you testing off of? Meminfo was removed a while ago so I'm not sure where the meminfo.nr_banks context is coming from.

This is 3.14, but I can forward port and test the change on 4.3-rcX as well where this is still relevant.

Quoted 11 lines

This is how arm currently defines VMALLOC ranges
#define VMALLOC_OFFSET (8*1024*1024)
#define VMALLOC_START (((unsigned long)high_memory +
VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1))
#define VMALLOC_END 0xff000000UL
VMALLOC_START is based off of the value of high_memory. high_memory is supposed to be the limit of the kernel direct mapped ram region so setting it to something above that is asking for trouble.

You're right, that does not seem right.

Quoted 3 lines

It seems like this is
working as intended. You'd have to decouple VMALLOC_START and high_memory to make this work properly.

That does not sound that trivial, but I can probably tinker with something locally which is not keyed of high_memory for now.

Quoted 5 lines

Having a vmalloc_start be greater than the end of direct mapped RAM seems a little unusual. This would be leaving a hole in the virtual space. Is this what you are intending or is there another use case for adjusting the vmalloc space as you described?

The intended use case is something like you have 512MB of physical memory, you want to give 248MB of this for vmalloc and no more, and the remainder to a statically carved out region, used for non Linux-managed allocations.

Do we agree though, that if you are setting vmalloc on the command-line we should be reserving the amount that is being indicated?

Signature · 1 line

Florian

Russell King - ARM Linux

2015-10-20 19:25 UTC #3 lore raw

Quoted 15 lines · Florian Fainelli

Quoted 14 lines · Laura Abbott
Quoted 13 lines · Florian Fainelli
Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000

Correct, high_memory will be 0xe0000000, which will set VMALLOC_START to 0xe0800000, and you'll have lots of vmalloc space available. That's intentional.

vmalloc= sets the _minimum_ vmalloc space that's available, not the absolute amount of space.

Quoted 10 lines

Quoted 8 lines
#define VMALLOC_OFFSET (8*1024*1024)
#define VMALLOC_START (((unsigned long)high_memory +
VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1))
#define VMALLOC_END 0xff000000UL
VMALLOC_START is based off of the value of high_memory. high_memory is supposed to be the limit of the kernel direct mapped ram region so setting it to something above that is asking for trouble.
You're right, that does not seem right.

That's correct. high_memory is the last address of mapped lowmem. We then have an 8MB space. The quote above is correct.

I don't see what the problem is.

Signature · 2 lines

FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Florian Fainelli

author

2015-10-20 19:32 UTC #4 lore raw

Quoted 25 lines · Russell King - ARM Linux

Quoted 15 lines · Florian Fainelli
Quoted 14 lines · Laura Abbott
Quoted 13 lines · Florian Fainelli
Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000

Quoted 6 lines

Correct, high_memory will be 0xe0000000, which will set VMALLOC_START to 0xe0800000, and you'll have lots of vmalloc space available. That's intentional.
vmalloc= sets the _minimum_ vmalloc space that's available, not the absolute amount of space.

I think we need to update Documentation/kernel-parameters.txt then:

"""
vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact
size of <nn>. This can be used to increase the minimum size (128MB on x86). It can also be used to decrease the size and leave more room for directly mapped kernel RAM.
"""

this makes me understand I should get the absolute value I specified.

Quoted 10 lines

Quoted 10 lines
Quoted 9 lines
#define VMALLOC_OFFSET (8*1024*1024)
#define VMALLOC_START (((unsigned long)high_memory +
VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1))
#define VMALLOC_END 0xff000000UL
VMALLOC_START is based off of the value of high_memory. high_memory is supposed to be the limit of the kernel direct mapped ram region so setting it to something above that is asking for trouble.

Quoted 6 lines

Quoted 1 line
You're right, that does not seem right.
That's correct. high_memory is the last address of mapped lowmem. We then have an 8MB space. The quote above is correct.
I don't see what the problem is.

Might just be documentation vs. actual results displayed by the "Kernel virtual memory layout". A similar "issue" exists if you specify vmalloc= on the command-line and you get a result which is offset by VMALLOC_OFFSET.

I will do a better homework and go back to thinking about this.

Signature · 1 line

Florian

Russell King - ARM Linux

2015-10-20 19:40 UTC #5 lore raw

Quoted 43 lines · Florian Fainelli

Quoted 25 lines · Russell King - ARM Linux
Quoted 15 lines · Florian Fainelli
Quoted 14 lines · Laura Abbott
Quoted 13 lines · Florian Fainelli
Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000
Quoted 6 lines
Correct, high_memory will be 0xe0000000, which will set VMALLOC_START to 0xe0800000, and you'll have lots of vmalloc space available. That's intentional.
vmalloc= sets the _minimum_ vmalloc space that's available, not the absolute amount of space.
I think we need to update Documentation/kernel-parameters.txt then:
"""
vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact
size of <nn>. This can be used to increase the minimum size (128MB on x86). It can also be used to decrease the size and leave more room for directly mapped kernel RAM.
"""

Quoted 1 line

this makes me understand I should get the absolute value I specified.

It's not very well worded, but:

"This can be used to increase the minimum size" is the applicable part here. On ARM we follow x86.

Quoted 5 lines

Might just be documentation vs. actual results displayed by the "Kernel virtual memory layout". A similar "issue" exists if you specify vmalloc= on the command-line and you get a result which is offset by VMALLOC_OFFSET.
I will do a better homework and go back to thinking about this.

x86 also has an 8MB hole.

Signature · 2 lines

FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Gregory Fong

2015-10-25 03:18 UTC #6 lore raw

Quoted 44 lines · Russell King - ARM Linux

Quoted 43 lines · Florian Fainelli
Quoted 25 lines · Russell King - ARM Linux
Quoted 15 lines · Florian Fainelli
Quoted 14 lines · Laura Abbott
Quoted 13 lines · Florian Fainelli
Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000
Quoted 6 lines
Correct, high_memory will be 0xe0000000, which will set VMALLOC_START to 0xe0800000, and you'll have lots of vmalloc space available. That's intentional.
vmalloc= sets the _minimum_ vmalloc space that's available, not the absolute amount of space.
I think we need to update Documentation/kernel-parameters.txt then:
"""
vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact
size of <nn>. This can be used to increase the minimum size (128MB on x86). It can also be used to decrease the size and leave more room for directly mapped kernel RAM.
"""

Quoted 6 lines

Quoted 1 line
this makes me understand I should get the absolute value I specified.
It's not very well worded, but:
"This can be used to increase the minimum size" is the applicable part here. On ARM we follow x86.

It isn't clear from the documentation that there might be some offset that needs to be taken into account. What would you think of a change in wording to:

vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact
size of <nn>. This can be used to increase the minimum size (128MB on x86). It can also be used to decrease the size and leave more room for directly mapped kernel RAM. On some architectures, the vmalloc area includes an offset, resulting in the available vmalloc space being smaller than the size specified here.

Laura Abbott

2015-10-20 19:27 UTC #7 lore raw

In reply to Florian Fainelli

Quoted 89 lines · Florian Fainelli

Quoted 51 lines · Laura Abbott
Quoted 50 lines · Florian Fainelli
Hi Russell, Laura,
Setting vmalloc= on the kernel command-line to define the amount of vmalloc_reserve is not quite working when you have no highmem, as is the case of one my boards which has 512MB or 256M populated on a first bank at PA 0x0.
What happens in that case is that, despite setting vmalloc_reserve, therefore bumping up vmalloc_min to a higher address than high_memory, which is assigned __va(arm_lowmem_limit), we end-up with VMALLOC_START at high_memory + VMALLOC_OFFSET, which yields the amount of physical memory (start at PA 0x0 in my case) - VMALLOC_OFFSET.
The maths look like this for this particular board (512MB)
high_memory = 0x20000000 + PAGE_OFFSET = 0x20000000 + 0xC0000000 = 0xE0000000
vmalloc_min = 0xFF00000 - (248 * 1024 * 1024) = 0xEF800000
so we end-up with VMALLOC_START = high_memory + VMALLOC_OFFSET = 0xE0000000 + 8 * 1024* 1024 = 0xE0800000
in sanity_check_meminfo(), high_memory is unconditionally assigned with arm_lowmem_limit's VA.
The following quick and dirty patch seems to do it for me, but I am not confident this is remotely the correct approach here:
arch/arm/mm/mmu.c +8 -1
index 14428d2..e196ea4 100644
@@ -1196,7 +1211,14 @@ void __init sanity_check_meminfo(void)
          }
   #endif
          meminfo.nr_banks = j;
-       high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       if (vmalloc_limit > arm_lowmem_limit)
+               high_memory = vmalloc_min - VMALLOC_OFFSET;
+       else
+               high_memory = __va(arm_lowmem_limit - 1) + 1;
+
+       pr_info("%s: high_memory: 0x%p, arm_lowmem_limit: 0x%llx\n",
+               __func__, high_memory, arm_lowmem_limit);
 
BTW, even when there is highmem available, setting vmalloc= on the command-line is off by VMALLOC_OFFSET, the way early_vmalloc() computes things, is that intended?
Quoted 6 lines
Quoted 2 lines
Thanks!
Which baseline are you testing off of? Meminfo was removed a while ago so I'm not sure where the meminfo.nr_banks context is coming from.
This is 3.14, but I can forward port and test the change on 4.3-rcX as well where this is still relevant.
Quoted 11 lines
This is how arm currently defines VMALLOC ranges
#define VMALLOC_OFFSET (8*1024*1024)
#define VMALLOC_START (((unsigned long)high_memory +
VMALLOC_OFFSET) & ~(VMALLOC_OFFSET-1))
#define VMALLOC_END 0xff000000UL
VMALLOC_START is based off of the value of high_memory. high_memory is supposed to be the limit of the kernel direct mapped ram region so setting it to something above that is asking for trouble.
You're right, that does not seem right.
Quoted 3 lines
It seems like this is
working as intended. You'd have to decouple VMALLOC_START and high_memory to make this work properly.
That does not sound that trivial, but I can probably tinker with something locally which is not keyed of high_memory for now.
Quoted 5 lines
Having a vmalloc_start be greater than the end of direct mapped RAM seems a little unusual. This would be leaving a hole in the virtual space. Is this what you are intending or is there another use case for adjusting the vmalloc space as you described?

Quoted 4 lines

The intended use case is something like you have 512MB of physical memory, you want to give 248MB of this for vmalloc and no more, and the remainder to a statically carved out region, used for non Linux-managed allocations.

I'm confused here. What does 512MB of physical memory have to do with your virtual address space? I think what you are saying is that you have 512MB of physical memory and you want to carve out 248M of that physical memory for a non-Linux region. If that's the case you would be better off reserving the physical region in devicetree. The virtual space wouldn't be relevant for anything outside of Linux.

Thanks,
Laura