Kernel crash in 2.6.32.6 / bcm1480 with 16k page size
Hi,
I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu. It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
A similar problem was recently fixed for ppc. It turned out to be a problem in ppc specific memory management code, so that fix won't help here.
Has anyone else seen this before ? Any idea where to start looking for the problem ?
Thanks,
Guenter
--------------------------------------
[instrumentation]pcpu_get_vm_areas: VMALLOC_START: 0xc000000000000000 VMALLOC_END: 0xc0007fff00000000 PTRS_PER_PGD: 2048 PTRS_PER_PMD: 2048 PTRS_PER_PTE: 2048 PAGE_SIZE: 0x4000emulate_load_store_insn: Attempted access to supervisor address space: opcode=0x2c addr=0xc0007ffefffc0000
[/instrumentation]
Unhandled kernel unaligned access[#1]:
Cpu 2
$ 0 : 0000000000000000 0000000014001fe0 a8000002fe067b68 0000000000000000
$ 4 : ffffffff80116f10 0000000014001fe3 000000000000003f c0007ffefffc0000
$ 8 : ffffffff807e9998 a8000002fe716228 0000000000000000 fffffffffe000000
$12 : 0000000014001fe1 000000001000001e 6db6db6db6db6db7 0000000000000001
$16 : fffffffffc85ffc0 a8000002fe067b40 ffffffff80810000 0000000000000000
$20 : a8000002fe7161e0 ffffffff80810000 ffffffff80730000 ffffffff80810000
$24 : 0000000000000000 ffffffff8f1bbcdc
$28 : a8000002fe064000 a8000002fe067b10 a8000002fe2e53c0 ffffffff801004e0
Hi : 000000000023d1ad
Lo : 00000000000bf08f
epc : ffffffff80116f38 do_ade+0x248/0x468
Not tainted
ra : ffffffff801004e0 ret_from_exception+0x0/0x20
Status: 14001fe3 KX SX UX KERNEL EXL IE
Cause : 00808014
BadVA : c0007ffefffc0000
PrId : 05041100 (SiByte SB1A)
Modules linked in:
Process swapper (pid: 1, threadinfo=a8000002fe064000, task=a8000002fe063658, tls=0000000000000000)
Stack : 0000000000000100 0000000000000000 ffffffff807e9998 ffffffff80810000
0000000000000000 ffffffff801004e0 0000000000000000 0000000014001fe0
c0007ffefffc0000 a800000003a5c060 c0007ffefffc0040 0000000000000000
0000000000001000 0000000000000000 ffffffff807e9998 a8000002fe716228
0000000000000000 fffffffffe000000 0000000000000000 c0007ffefffc1000
6db6db6db6db6db7 0000000000000001 0000000000000000 ffffffff807e9998
ffffffff80810000 0000000000000000 a8000002fe7161e0 ffffffff80810000
ffffffff80730000 ffffffff80810000 0000000000000000 ffffffff8f1bbcdc
6db6db6db6db6db7 a8000002fe7161e0 a8000002fe064000 a8000002fe067c70
a8000002fe2e53c0 ffffffff801f0d04 0000000014001fe3 000000000023d1ad
...
Call Trace:
[<ffffffff80116f38>] do_ade+0x248/0x468
[<ffffffff801004e0>] ret_from_exception+0x0/0x20
[<ffffffff8010585c>] __bzero+0x38/0x11c
[<ffffffff801f0d04>] pcpu_alloc+0x9b4/0xba0
[<ffffffff804b1460>] snmp_mib_init+0x40/0x80
[<ffffffff804fa6e8>] ipv6_add_dev+0x170/0x3c0
[<ffffffff80797178>] addrconf_init+0x44/0x17c
[<ffffffff8079705c>] inet6_init+0x29c/0x368
[<ffffffff8010fa88>] do_one_initcall+0x38/0x198
[<ffffffff80778350>] kernel_init+0x1ec/0x26c
[<ffffffff80112780>] kernel_thread_helper+0x10/0x18Code: 000210f8 0222102d dc430000 <b0e30000> b4e30007 24030000 1060ffe7 00000000 00000000 Disabling lock debugging due to kernel taint Kernel panic - not syncing: Attempted to kill init!
Rebooting in 5 seconds..
Quoted 8 lines · Guenter Roeck
I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu. It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
A similar problem was recently fixed for ppc. It turned out to be a problem in ppc specific memory management code, so that fix won't help here.
Has anyone else seen this before ? Any idea where to start looking for the problem ?
Supposedly this was working for SB1. I suggest you find an older kernel version that works for your with 16k pages then use git bisect to find the problem.
Ralf
Quoted 5 lines · Ralf Baechle
Quoted 4 lines · Guenter Roeck
I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu. It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
Quoted 9 lines
Quoted 4 lines
A similar problem was recently fixed for ppc. It turned out to be a problem in ppc specific memory management code, so that fix won't help here.
Has anyone else seen this before ? Any idea where to start looking for the problem ?
Supposedly this was working for SB1. I suggest you find an older kernel version that works for your with 16k pages then use git bisect to find the problem.
It used to work with 2.6.27.
However, bisect won't work, because the code in question (per cpu memory allocation) was completely rewritten since then.
The new percpu code tries to allocate memory just below VMALLOC_END. This works on sb1 for a page size of 4k, but not for a page size of 16k and 64k. The value of VMALLOC_END depends on the page size.
ppc had a similar problem. It had nothing to do with the new percpu memory allocation code, but with memory alocation close to VMALLOC_END. In other words, it was a day-one bug which was never noticed because allocating memory in that address space is highly unlikely.
I suspect the same is the case here. I could write some code for 2.6.27, to test the same memory allocation there, but I am quite sure the problem is going to show up there as well.
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Thanks,
Guenter
Quoted 22 lines · Guenter Roeck
Quoted 4 lines · Ralf Baechle
Quoted 3 lines · Guenter Roeck
I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu. It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
Quoted 8 lines
Quoted 4 lines
A similar problem was recently fixed for ppc. It turned out to be a problem in ppc specific memory management code, so that fix won't help here.
Has anyone else seen this before ? Any idea where to start looking for the problem ?
Supposedly this was working for SB1. I suggest you find an older kernel version that works for your with 16k pages then use git bisect to find the problem.
It used to work with 2.6.27.
However, bisect won't work, because the code in question (per cpu memory allocation) was completely rewritten since then.
The new percpu code tries to allocate memory just below
VMALLOC_END. This works on sb1 for a page size of 4k, but not for a page size of 16k and 64k. The value ofVMALLOC_ENDdepends on the page size.
Quoted 9 lines
ppc had a similar problem. It had nothing to do with the new percpu memory allocation code, but with memory alocation close to
VMALLOC_END. In other words, it was a day-one bug which was never noticed because allocating memory in that address space is highly unlikely.I suspect the same is the case here. I could write some code for 2.6.27, to test the same memory allocation there, but I am quite sure the problem is going to show up there as well.
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
David Daney.
Quoted 7 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.
Either way, that's good enought to look into things.
Ralf
Quoted 6 lines · Ralf Baechle
Quoted 4 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Quoted 9 lines
Quoted 3 lines
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.
Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do with the problem, though. IPv6 enabled just means that the percpu code needs to allocate more memory. This memory allocation then crashes.
Quoted 2 lines
Either way, that's good enought to look into things.
Did you see the problem on a bcm1250/1480 or with some other mips core ?
Guenter
Quoted 16 lines · Guenter Roeck
Quoted 6 lines · Ralf Baechle
Quoted 4 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Quoted 9 lines
Quoted 3 lines
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.
Quoted 7 lines
Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do with the problem, though. IPv6 enabled just means that the percpu code needs to allocate more memory. This memory allocation then crashes.
Quoted 2 lines
Either way, that's good enought to look into things.
Did you see the problem on a bcm1250/1480 or with some other mips core ?
That was on R10000.
Ralf
Quoted 22 lines · Ralf Baechle
Quoted 20 lines · Guenter Roeck
Quoted 6 lines · Ralf Baechle
Quoted 4 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Quoted 9 lines
Quoted 3 lines
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do with the problem, though. IPv6 enabled just means that the percpu code needs to allocate more memory. This memory allocation then crashes.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Either way, that's good enought to look into things.
Did you see the problem on a bcm1250/1480 or with some other mips core ?
That was on R10000.
According to Wikipedia, R10k has a 44 bit virtual memory size limit.
So I think we have two options, assuming we go with the approach I used in the patch I sent out yesterday. We could either set the default to 44 bit and override it with larger values for processors like the Octeon, or go with a large default (eg with the 60 bit I proposed in the patch) and override it with smaller values as needed (ie pretty much for all CPUs).
Seems to me if we would have fewer exceptions if we set the default to 44 bit. Also, it would probably be better if the number of bits is too low for a given CPU, since it would not result in a crash. So I would prefer to use 44 bit as default.
Thoughts ?
Guenter
Quoted 37 lines · Guenter Roeck
Quoted 22 lines · Ralf Baechle
Quoted 20 lines · Guenter Roeck
Quoted 6 lines · Ralf Baechle
Quoted 4 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Quoted 9 lines
Quoted 3 lines
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do with the problem, though. IPv6 enabled just means that the percpu code needs to allocate more memory. This memory allocation then crashes.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Either way, that's good enought to look into things.
Did you see the problem on a bcm1250/1480 or with some other mips core ?
That was on R10000.
According to Wikipedia, R10k has a 44 bit virtual memory size limit.
So I think we have two options, assuming we go with the approach I used in the patch I sent out yesterday. We could either set the default to 44 bit and override it with larger values for processors like the Octeon, or go with a large default (eg with the 60 bit I proposed in the patch) and override it with smaller values as needed (ie pretty much for all CPUs).
Quoted 6 lines
Seems to me if we would have fewer exceptions if we set the default to 44 bit. Also, it would probably be better if the number of bits is too low for a given CPU, since it would not result in a crash. So I would prefer to use 44 bit as default.
Thoughts ?
As a followup, this is what I found so far for the various mips64 cores in respect to virtual memory size.
Core Virtual memory size
R10000 44 bit
SB1 44 bit
Octeon 49 bit
MIPS 20Kc 40 bit
MIPS 5Kc/5Kf 42 bit
Loongson 2E/2F 40 bit
Guenter
Quoted 10 lines · Ralf Baechle
Quoted 7 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Quoted 7 lines
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Either way, that's good enought to look into things.
16k page size works for me with the patch below. Not that I have any idea why; this was just a blind test.
It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding available virtual memory per page size may contradict with the definition of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases, but the comments indicate that a system with 16k pages should have _less_ virtual memory available than a system with 4k pages because it only uses a 2 level page table.
Guenter
---------------
git diff arch/mips/include/asm/pgtable-64.h
arch/mips/include/asm/pgtable-64.h +1 -1
@@ -110,7 +110,7 @@#define VMALLOC_START MAP_BASE#define VMALLOC_END \ (VMALLOC_START + \ - PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32)) + (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))#if defined(CONFIG_MODULES ) && defined(KBUILD_64BIT_SYM32 ) && \VMALLOC_START !=CKSSEG /* Load modules into 32bit-compatible segment. */
Quoted 36 lines · Guenter Roeck
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Quoted 7 lines
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Either way, that's good enought to look into things.
16k page size works for me with the patch below. Not that I have any idea why; this was just a blind test.
It seems to me that the notes in
arch/mips/include/asm/pgtable-64.hregarding available virtual memory per page size may contradict with the definition ofVMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases, but the comments indicate that a system with 16k pages should have _less_ virtual memory available than a system with 4k pages because it only uses a 2 level page table.Guenter
---------------
git diff
arch/mips/include/asm/pgtable-64.harch/mips/include/asm/pgtable-64.h +0
-0
arch/mips/include/asm/pgtable-64.h line 116
Quoted 8 lines
arch/mips/include/asm/pgtable-64.h +1
-1@@ -110,7 +110,7 @@#define VMALLOC_START MAP_BASE#define VMALLOC_END \ (VMALLOC_START + \ - PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32)) + (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))#if defined(CONFIG_MODULES ) && defined(KBUILD_64BIT_SYM32 ) && \VMALLOC_START !=CKSSEG /* Load modules into 32bit-compatible segment. */
Although it may fix it, I think something along the lines of this:
In asm/mach-generic/spaces.h:
#define MAX_VIRTUAL_SIZE (1 << some number)
In asm/mach-sibyte/paces.h:
#define MAX_VIRTUAL_SIZE (1 << some other umber)
In arch/mips/include/asm/pgtable-64.h
#define VIRTUAL_SIZE (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE)
#define VMALLOC_END (VMALLOC_START + min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE) - (1UL << 32))
Quoted 54 lines · David Daney
Quoted 36 lines · Guenter Roeck
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Quoted 7 lines
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Either way, that's good enought to look into things.
16k page size works for me with the patch below. Not that I have any idea why; this was just a blind test.
It seems to me that the notes in
arch/mips/include/asm/pgtable-64.hregarding available virtual memory per page size may contradict with the definition ofVMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases, but the comments indicate that a system with 16k pages should have _less_ virtual memory available than a system with 4k pages because it only uses a 2 level page table.Guenter
---------------
git diff
arch/mips/include/asm/pgtable-64.harch/mips/include/asm/pgtable-64.h +0
-0arch/mips/include/asm/pgtable-64.h line 116
Quoted 8 lines
arch/mips/include/asm/pgtable-64.h +1
-1@@ -110,7 +110,7 @@#define VMALLOC_START MAP_BASE#define VMALLOC_END \ (VMALLOC_START + \ - PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32)) + (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))#if defined(CONFIG_MODULES ) && defined(KBUILD_64BIT_SYM32 ) && \VMALLOC_START !=CKSSEG /* Load modules into 32bit-compatible segment. */ Although it may fix it, I think something along the lines of this:
In
asm/mach-generic/spaces.h:
#defineMAX_VIRTUAL_SIZE(1 << some number)In
asm/mach-sibyte/paces.h:
#defineMAX_VIRTUAL_SIZE(1 << some other umber)
Quoted 7 lines
In
arch/mips/include/asm/pgtable-64.h#define
VIRTUAL_SIZE(PTRS_PER_PGD*PTRS_PER_PMD*PTRS_PER_PTE*PAGE_SIZE)#define
VMALLOC_END(VMALLOC_START+min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE)
- (1UL << 32))
Something like that. My patch wasn't supposed to be a proposal for a fix, just a guide to help finding the underlying problem.
It may require some factor related to the page size.
Someone who knows mips and its memory management scheme should have a look, otherwise it would just be a trial-end-error fix.
Guenter
Quoted 58 lines · Guenter Roeck
Quoted 57 lines · David Daney
Quoted 36 lines · Guenter Roeck
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Quoted 7 lines
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Either way, that's good enought to look into things.
16k page size works for me with the patch below. Not that I have any idea why; this was just a blind test.
It seems to me that the notes in
arch/mips/include/asm/pgtable-64.hregarding available virtual memory per page size may contradict with the definition ofVMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases, but the comments indicate that a system with 16k pages should have _less_ virtual memory available than a system with 4k pages because it only uses a 2 level page table.Guenter
---------------
git diff
arch/mips/include/asm/pgtable-64.harch/mips/include/asm/pgtable-64.h +0
-0arch/mips/include/asm/pgtable-64.h line 116
Quoted 8 lines
arch/mips/include/asm/pgtable-64.h +1
-1@@ -110,7 +110,7 @@#define VMALLOC_START MAP_BASE#define VMALLOC_END \ (VMALLOC_START + \ - PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32)) + (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))#if defined(CONFIG_MODULES ) && defined(KBUILD_64BIT_SYM32 ) && \VMALLOC_START !=CKSSEG /* Load modules into 32bit-compatible segment. */ Although it may fix it, I think something along the lines of this:
In
asm/mach-generic/spaces.h:
#defineMAX_VIRTUAL_SIZE(1 << some number)In
asm/mach-sibyte/paces.h:
#defineMAX_VIRTUAL_SIZE(1 << some other umber)In
arch/mips/include/asm/pgtable-64.h#define
VIRTUAL_SIZE(PTRS_PER_PGD*PTRS_PER_PMD*PTRS_PER_PTE*PAGE_SIZE)
Quoted 9 lines
Quoted 2 lines
#define
VMALLOC_END(VMALLOC_START+min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE)
- (1UL << 32))Something like that. My patch wasn't supposed to be a proposal for a fix, just a guide to help finding the underlying problem.
It may require some factor related to the page size.
Someone who knows mips and its memory management scheme should have a look, otherwise it would just be a trial-end-error fix.
I suspect you are hitting a maximum valid address bits limit and getting the Address Exception. Limiting VMALLOC_END so that you don't hit the limit seems to be the solution. I don't have the manual for the sibyte, so I don't know what the limit is. The architecture specification doesn't state a fixed limit, although it tells what should happen when the limit is reached.
David Daney
On Fri, Jan 29, 2010 at 02:28:49PM -0500, David Daney wrote:
...
Quoted 8 lines
I suspect you are hitting a maximum valid address bits limit and getting the Address Exception. Limiting
VMALLOC_ENDso that you don't hit the limit seems to be the solution. I don't have the manual for the sibyte, so I don't know what the limit is. The architecture specification doesn't state a fixed limit, although it tells what should happen when the limit is reached.
You mean there might be a CPU-specific limit ? I hope not - that would be quite messy.
I'll see if I can dig up a sibyte manual.
Guenter
Quoted 8 lines · Guenter Roeck
Quoted 7 lines
I suspect you are hitting a maximum valid address bits limit and getting the Address Exception. Limiting
VMALLOC_ENDso that you don't hit the limit seems to be the solution. I don't have the manual for the sibyte, so I don't know what the limit is. The architecture specification doesn't state a fixed limit, although it tells what should happen when the limit is reached.You mean there might be a CPU-specific limit ? I hope not - that would be quite messy.
The size of the address space can be probed via CP0 registers (for MIPS
architecture processors that is). No need to add any CPU dependencies (except from legacy 64-bit MIPS processors perhaps).
Maciej
Quoted 9 lines · Maciej W. Rozycki
Quoted 7 lines · Guenter Roeck
Quoted 7 lines
I suspect you are hitting a maximum valid address bits limit and getting the Address Exception. Limiting
VMALLOC_ENDso that you don't hit the limit seems to be the solution. I don't have the manual for the sibyte, so I don't know what the limit is. The architecture specification doesn't state a fixed limit, although it tells what should happen when the limit is reached.
Quoted 6 lines
Quoted 1 line
You mean there might be a CPU-specific limit ? I hope not - that would be quite messy.
The size of the address space can be probed via CP0 registers (for MIPS
architecture processors that is). No need to add any CPU dependencies (except from legacy 64-bit MIPS processors perhaps).
That would help. Do you happen to know which CP0 register(s) to look for ? I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.
Guenter
Quoted 6 lines · Guenter Roeck
Quoted 4 lines
The size of the address space can be probed via CP0 registers (for MIPS
architecture processors that is). No need to add any CPU dependencies (except from legacy 64-bit MIPS processors perhaps).That would help. Do you happen to know which CP0 register(s) to look for ? I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.
Write a value with all bits set to c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different TLB scheme anyway.
Similarly it is possible to probe the physical address range in either c0_entrylo0 or c0_entrylo1. This is also of interest on 32-bit processors.
Ralf
Quoted 9 lines · Ralf Baechle
Write a value with all bits set to
c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different TLB scheme anyway.Similarly it is possible to probe the physical address range in either
c0_entrylo0orc0_entrylo1. This is also of interest on 32-bit processors.
Indeed -- IIRC the architecture spec calls this value PABITS. I wasn't
sure about the legacy processors -- if that works with them too, then it's even better.
Maciej
Quoted 7 lines · Maciej W. Rozycki
Quoted 7 lines
Write a value with all bits set to
c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different TLB scheme anyway.
Quoted 6 lines
Quoted 2 lines
Similarly it is possible to probe the physical address range in either
c0_entrylo0orc0_entrylo1. This is also of interest on 32-bit processors.Indeed -- IIRC the architecture spec calls this value PABITS. I wasn't
sure about the legacy processors -- if that works with them too, then it's even better.
The probing method was undocumented until MIPS64 but if you look at the format of the EntryLo register it's always been possible. The R10000 needs special treatment though - it has the UC (Uncache Attribute) field in the bits 62..63 of EntryLo; this field needs to be ignored.
Ralf
Quoted 6 lines · Ralf Baechle
Quoted 3 lines
Indeed -- IIRC the architecture spec calls this value PABITS. I wasn't
sure about the legacy processors -- if that works with them too, then it's even better.The probing method was undocumented until MIPS64 but if you look at the format of the EntryLo register it's always been possible. The R10000
Frankly, I would be concerned about some reserved bits left floating
rather than hardwired to any particular logical state, but the number of chips to check is limited. At least for some of them documentation is explicit the value reported is fixed and we can take it as an indication it is likely actually the case.
Quoted 2 lines
needs special treatment though - it has the UC (Uncache Attribute) field in the bits 62..63 of EntryLo; this field needs to be ignored.
Well, I think masking out everything beyond what would become the bit #63
of the physical address unconditionally is a safe and reasonable way to make the approach automagically work for the R10k too.
Maciej
Quoted 20 lines · Ralf Baechle DL5RB wrote:
Quoted 8 lines · Guenter Roeck
Quoted 1 line
The size of the address space can be probed via CP0 registers (for
MIPS
Quoted 3 lines
architecture processors that is). No need to add any CPU dependencies (except from legacy 64-bit MIPS processors perhaps).
That would help. Do you happen to know which CP0 register(s) to look for ?
I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.Write a value with all bits set to
c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different
TLB scheme anyway.
Quoted 8 lines
Similarly it is possible to probe the physical address range in either
c0_entrylo0orc0_entrylo1. This is also of interest on 32-bit processors.Ralf
Ok, I'll try that and submit a new set of patches if it works.
Guenter
Signature · 2 lines
View this message in context: http://old.nabble.com/Kernel-crash-in-2.6.32.6---bcm1480-with-16k-page-size-tp27358179p27405624.html Sent from the linux-mips main mailing list archive at Nabble.com.
Quoted 9 lines · Ralf Baechle
Quoted 6 lines · Guenter Roeck
Quoted 4 lines
The size of the address space can be probed via CP0 registers (for MIPS
architecture processors that is). No need to add any CPU dependencies (except from legacy 64-bit MIPS processors perhaps).That would help. Do you happen to know which CP0 register(s) to look for ? I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.
Quoted 6 lines
Write a value with all bits set to
c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different TLB scheme anyway.
Are you sure that this doesn't work for the R8000 ? From the user's manual (section 2.1.9, EntryHi) it looks like it should work.
Guenter
Quoted 9 lines · Guenter Roeck
Quoted 6 lines
Write a value with all bits set to
c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different TLB scheme anyway.Are you sure that this doesn't work for the R8000 ? From the user's manual (section 2.1.9, EntryHi) it looks like it should work.
The probe itself will work if it's carefully written not to get fooled by difference in bits 12..18 but that's easy) but still the R8000 has a fairly different TLB architecture which Linux doesn't support - the machines are rare and none of the active developers has one.
Ralf
Quoted 9 lines · Ralf Baechle
Quoted 7 lines · Guenter Roeck
Quoted 6 lines
Write a value with all bits set to
c0_entryhi, then read it back again. The set bits in the VPN2 bitfield will indicate the size of the virtual address range supported. The MIPS64 documentation also calls this value SEGBITS. The nice thing about this probe is that it is supported for all 64-bit MIPS processors except the R8000 which has an entirely different TLB scheme anyway.
Quoted 8 lines
Quoted 2 lines
Are you sure that this doesn't work for the R8000 ? From the user's manual (section 2.1.9, EntryHi) it looks like it should work.
The probe itself will work if it's carefully written not to get fooled by difference in bits 12..18 but that's easy) but still the R8000 has a fairly different TLB architecture which Linux doesn't support - the machines are rare and none of the active developers has one.
Looks like my code should work. I'll send out a patch in a couple of minutes.
Guenter
Quoted 70 lines · David Daney
Quoted 59 lines · Guenter Roeck
Quoted 58 lines · David Daney
Quoted 36 lines · Guenter Roeck
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Quoted 7 lines
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Either way, that's good enought to look into things.
16k page size works for me with the patch below. Not that I have any idea why; this was just a blind test.
It seems to me that the notes in
arch/mips/include/asm/pgtable-64.hregarding available virtual memory per page size may contradict with the definition ofVMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases, but the comments indicate that a system with 16k pages should have _less_ virtual memory available than a system with 4k pages because it only uses a 2 level page table.Guenter
---------------
git diff
arch/mips/include/asm/pgtable-64.harch/mips/include/asm/pgtable-64.h +0
-0arch/mips/include/asm/pgtable-64.h line 116
Quoted 8 lines
arch/mips/include/asm/pgtable-64.h +1
-1@@ -110,7 +110,7 @@#define VMALLOC_START MAP_BASE#define VMALLOC_END \ (VMALLOC_START + \ - PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32)) + (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))#if defined(CONFIG_MODULES ) && defined(KBUILD_64BIT_SYM32 ) && \VMALLOC_START !=CKSSEG /* Load modules into 32bit-compatible segment. */ Although it may fix it, I think something along the lines of this:
In
asm/mach-generic/spaces.h:
#defineMAX_VIRTUAL_SIZE(1 << some number)In
asm/mach-sibyte/paces.h:
#defineMAX_VIRTUAL_SIZE(1 << some other umber)In
arch/mips/include/asm/pgtable-64.h#define
VIRTUAL_SIZE(PTRS_PER_PGD*PTRS_PER_PMD*PTRS_PER_PTE*PAGE_SIZE)Quoted 9 lines
Quoted 2 lines
#define
VMALLOC_END(VMALLOC_START+min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE)
- (1UL << 32))Something like that. My patch wasn't supposed to be a proposal for a fix, just a guide to help finding the underlying problem.
It may require some factor related to the page size.
Someone who knows mips and its memory management scheme should have a look, otherwise it would just be a trial-end-error fix.
Quoted 7 lines
I suspect you are hitting a maximum valid address bits limit and getting the Address Exception. Limiting
VMALLOC_ENDso that you don't hit the limit seems to be the solution. I don't have the manual for the sibyte, so I don't know what the limit is. The architecture specification doesn't state a fixed limit, although it tells what should happen when the limit is reached.
To follow up on this - does Octeon have a fixed VM size limit ? And when you run your tests with larger page sizes, do you have IPV6 enabled ?
Guenter
Quoted 68 lines · Guenter Roeck
Quoted 67 lines · David Daney
Quoted 66 lines · Guenter Roeck
Quoted 53 lines · David Daney
Quoted 36 lines · Guenter Roeck
Quoted 8 lines · Ralf Baechle
Quoted 6 lines · David Daney
Quoted 3 lines
So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Quoted 7 lines
I can reproduce it with today's
14b7baff3eb4b1b46a592630e6f85ded9264798a. 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes. Note, I was testing with a non-16K capable userland so ok means userland is reached.Either way, that's good enought to look into things.
16k page size works for me with the patch below. Not that I have any idea why; this was just a blind test.
It seems to me that the notes in
arch/mips/include/asm/pgtable-64.hregarding available virtual memory per page size may contradict with the definition ofVMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases, but the comments indicate that a system with 16k pages should have _less_ virtual memory available than a system with 4k pages because it only uses a 2 level page table.Guenter
---------------
git diff
arch/mips/include/asm/pgtable-64.harch/mips/include/asm/pgtable-64.h +0
-0arch/mips/include/asm/pgtable-64.h line 116
Quoted 8 lines
arch/mips/include/asm/pgtable-64.h +1
-1@@ -110,7 +110,7 @@#define VMALLOC_START MAP_BASE#define VMALLOC_END \ (VMALLOC_START + \ - PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32)) + (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))#if defined(CONFIG_MODULES ) && defined(KBUILD_64BIT_SYM32 ) && \VMALLOC_START !=CKSSEG /* Load modules into 32bit-compatible segment. */ Although it may fix it, I think something along the lines of this:
In
asm/mach-generic/spaces.h:
#defineMAX_VIRTUAL_SIZE(1 << some number)In
asm/mach-sibyte/paces.h:
#defineMAX_VIRTUAL_SIZE(1 << some other umber)Quoted 6 lines
In
arch/mips/include/asm/pgtable-64.h#define
VIRTUAL_SIZE(PTRS_PER_PGD*PTRS_PER_PMD*PTRS_PER_PTE*PAGE_SIZE)#define
VMALLOC_END(VMALLOC_START+min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE)
- (1UL << 32))Something like that. My patch wasn't supposed to be a proposal for a fix, just a guide to help finding the underlying problem.
It may require some factor related to the page size.
Someone who knows mips and its memory management scheme should have a look, otherwise it would just be a trial-end-error fix.
Quoted 3 lines
Quoted 2 lines
I suspect you are hitting a maximum valid address bits limit and getting the Address Exception. Limiting
VMALLOC_ENDso that you don't hit the limit seems to be the solution. I don't have the manual for the sibyte, so I don't know what the limit is. The architecture specification doesn't state a fixed limit, although it tells what should happen when the limit is reached.To follow up on this - does Octeon have a fixed VM size limit ?
Yes. But it is larger than SB1's 44 bits. For Octeon it is 49 bits.
Quoted 1 line
And when you run your tests with larger page sizes, do you have IPV6 enabled ?
Yes.
But at 16K pages the VM size is 47 bits, so it is under the limit.
At 64K pages I am using 2-level page tables for 42 bits of VM space which is again under the limit.
On SB1 and 3-level tables, you will always exceed the limit for both 16K (47 bits) and 64K (55 bits) page sizes.
David Daney
On Fri, Jan 29, 2010 at 03:23:21PM -0500, David Daney wrote:
...
Quoted 9 lines
Quoted 1 line
To follow up on this - does Octeon have a fixed VM size limit ?
Yes. But it is larger than SB1's 44 bits. For Octeon it is 49 bits.
Quoted 1 line
And when you run your tests with larger page sizes, do you have IPV6 enabled ?
Yes.
Quoted 8 lines
But at 16K pages the VM size is 47 bits, so it is under the limit.
At 64K pages I am using 2-level page tables for 42 bits of VM space which is again under the limit.
On SB1 and 3-level tables, you will always exceed the limit for both 16K (47 bits) and 64K (55 bits) page sizes.
Ok, based on this info I'll send out a proposed patch shortly.
Guenter
Quoted 6 lines · David Daney
Quoted 5 lines · Guenter Roeck
Quoted 4 lines · Ralf Baechle
Quoted 3 lines · Guenter Roeck
I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu. It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
...
Quoted 4 lines
Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 2.6.33-rc*. I have not seen any crashes that can not be easily explained.
Do you have IPV6 enabled ? If not, you won't see the crash. I forgot to mention that earlier.
Guenter