Oops in guest after ioremap() on ARMv7
When running the linux kernel on the ARMv7 envelope model as a guest under the Xen hypervisor there is a oops (see below for an example of the page translation fault) when trying to access ioremap()'d memory.
The same kernel works fine when not running under the hypervisor.
It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.
Calling flush_cache_all() in flush_cache_vmap() makes it work. What isn't being correctly flushed? I see that flush_pmd_entry() and cpu_v7_set_pte_ext() already flush the L1 and L2 translation table entries and I can't think of anything else that would need to be flushed (unless the mapped virtual addresses need to be flushed as well?)
The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a branch predictor flush are required after a translation table entry update. This seems not to be done but adding this didn't seem to help (and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).
I don't see anything in the hypervisor that could be causing this as the fault is occurring at stage 1 and not stage 2 translation.
David
[ 0.200952] Unable to handle kernel paging request at virtual address c8804050 [ 0.203234] pgd = c0004000 [ 0.204147] [c8804050] *pgd=8781a811, *pte=1a000653, *ppte=1a000453 [ 0.206120] Internal error: Oops: 7 [#1] [ 0.207381] CPU: 0 Tainted: G W (3.2.0-rc5.dv #123) [ 0.209338] PC is at v2m_init+0x44/0x100 [ 0.210616] LR is at ioremap_page_range+0x100/0x164 [ 0.212197] pc : [<c0325130>] lr : [<c013e278>] psr: a0000013 [ 0.212235] sp : c781dfa0 ip : 00000001 fp : 00000000 [ 0.215880] r10: 00000000 r9 : 00000000 r8 : 00000000 [ 0.217570] r7 : 00000013 r6 : 00000000 r5 : c0321f1c r4 : c033d184 [ 0.219680] r3 : c8804000 r2 : c0325128 r1 : 1a000613 r0 : c8804000 [ 0.221756] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel [ 0.224159] Control: 10c53c7d Table: 80004059 DAC: 00000015 [ 0.225990] Process swapper (pid: 1, stack limit = 0xc781c268) [ 0.227858] Stack: (0xc781dfa0 to 0xc781e000) [ 0.229325] dfa0: c03250ec c033d184 c0321f1c c0321f38 c781c000 c031f7c0 c033d430 33369eb8 [ 0.231955] dfc0: 00000000 00000000 00000080 c033d184 c033d430 c0009eb8 00000013 00000000 [ 0.234652] dfe0: 00000000 c031f908 00000000 00000000 c031f890 c0009eb8 dfdfdfcf cfdfdfdf [ 0.237304] [<c0325130>] (v2m_init+0x44/0x100) from [<c0321f38>] (customize_machine+0x1c/0x28) [ 0.240099] [<c0321f38>] (customize_machine+0x1c/0x28) from [<c031f7c0>] (do_one_initcall+0x90/0x160) [ 0.243080] [<c031f7c0>] (do_one_initcall+0x90/0x160) from [<c031f908>] (kernel_init+0x78/0x114) [ 0.245958] [<c031f908>] (kernel_init+0x78/0x114) from [<c0009eb8>] (kernel_thread_exit+0x0/0x8) [ 0.248823] Code: e3a02000 ebf3a9b7 e2503000 0a00000a (e5933050)
Quoted 15 lines · David Vrabel
When running the linux kernel on the ARMv7 envelope model as a guest under the Xen hypervisor there is a oops (see below for an example of the page translation fault) when trying to access ioremap()'d memory.
The same kernel works fine when not running under the hypervisor.
It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.Calling
flush_cache_all()inflush_cache_vmap()makes it work. What isn't being correctly flushed? I see thatflush_pmd_entry()andcpu_v7_set_pte_ext()already flush the L1 and L2 translation table entries and I can't think of anything else that would need to be flushed (unless the mapped virtual addresses need to be flushed as well?)
Quoted 7 lines
The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a branch predictor flush are required after a translation table entry update. This seems not to be done but adding this didn't seem to help (and using
local_flush_tlb_all()) inflush_cache_vmap()didn't help either).I don't see anything in the hypervisor that could be causing this as the fault is occurring at stage 1 and not stage 2 translation.
Interesting error, I don't have an immediate idea of what might be wrong, just some questions.
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
Signature · 1 line
Catalin
Quoted 4 lines · Catalin Marinas
Quoted 3 lines · David Vrabel
When running the linux kernel on the ARMv7 envelope model as a guest under the Xen hypervisor there is a oops (see below for an example of the page translation fault) when trying to access ioremap()'d memory.
The translation tables for userspace seem to be also affected. The program repeatedly faults with a translation fault on the same address.
Putting a cache_flush_all() after the call to handle_mm_fault() in__do_page_fault() makes userspace work as well.
Quoted 19 lines
Quoted 11 lines
The same kernel works fine when not running under the hypervisor.
It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.Calling
flush_cache_all()inflush_cache_vmap()makes it work. What isn't being correctly flushed? I see thatflush_pmd_entry()andcpu_v7_set_pte_ext()already flush the L1 and L2 translation table entries and I can't think of anything else that would need to be flushed (unless the mapped virtual addresses need to be flushed as well?)Quoted 7 lines
The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a branch predictor flush are required after a translation table entry update. This seems not to be done but adding this didn't seem to help (and using
local_flush_tlb_all()) inflush_cache_vmap()didn't help either).I don't see anything in the hypervisor that could be causing this as the fault is occurring at stage 1 and not stage 2 translation.
Quoted 7 lines
Interesting error, I don't have an immediate idea of what might be wrong, just some questions.
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer write-back write-allocate cacheable; Normal memory, inner write-back, write-allocate cacheable.
L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer write-back cacheable.
These look sensible to me.
David
Quoted 10 lines · David Vrabel
Quoted 4 lines · Catalin Marinas
Quoted 3 lines · David Vrabel
When running the linux kernel on the ARMv7 envelope model as a guest under the Xen hypervisor there is a oops (see below for an example of the page translation fault) when trying to access ioremap()'d memory.
The translation tables for userspace seem to be also affected. The program repeatedly faults with a translation fault on the same address.
Putting acache_flush_all()after the call tohandle_mm_fault()in__do_page_fault()makes userspace work as well.
With the classic page tables, on A15 we need this patch:
But that's to do with TLBs rather than cache and it only shows on real hardware rather than model.
Quoted 31 lines
Quoted 19 lines
Quoted 11 lines
The same kernel works fine when not running under the hypervisor.
It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.Calling
flush_cache_all()inflush_cache_vmap()makes it work. What isn't being correctly flushed? I see thatflush_pmd_entry()andcpu_v7_set_pte_ext()already flush the L1 and L2 translation table entries and I can't think of anything else that would need to be flushed (unless the mapped virtual addresses need to be flushed as well?)Quoted 7 lines
The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a branch predictor flush are required after a translation table entry update. This seems not to be done but adding this didn't seem to help (and using
local_flush_tlb_all()) inflush_cache_vmap()didn't help either).I don't see anything in the hypervisor that could be causing this as the fault is occurring at stage 1 and not stage 2 translation.
Quoted 7 lines
Interesting error, I don't have an immediate idea of what might be wrong, just some questions.
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
VTCR =
0x80002558which is: Outer Shareable; Normal memory, outer write-back write-allocate cacheable; Normal memory, inner write-back, write-allocate cacheable.
Quoted 5 lines
L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer write-back cacheable.These look sensible to me.
They look fine (UP system). BTW, I assume that the hypervisor also flushes the caches and TLBs for the stage 2 translation tables.
It could as well be a model bug but people are on holiday at the moment (and I'm off shortly as well, until 3rd of January). Could you try to disable the cacheability of the page table walks for both stage 1 (TTBRx with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since Linux does the correct cache flushing and I assume the hypervisor as well, this may work around possible model bug.
Signature · 1 line
Catalin
Quoted 50 lines · Catalin Marinas
Quoted 10 lines · David Vrabel
Quoted 4 lines · Catalin Marinas
Quoted 3 lines · David Vrabel
When running the linux kernel on the ARMv7 envelope model as a guest under the Xen hypervisor there is a oops (see below for an example of the page translation fault) when trying to access ioremap()'d memory.
The translation tables for userspace seem to be also affected. The program repeatedly faults with a translation fault on the same address.
Putting acache_flush_all()after the call tohandle_mm_fault()in__do_page_fault()makes userspace work as well.With the classic page tables, on A15 we need this patch:
But that's to do with TLBs rather than cache and it only shows on real hardware rather than model.
Quoted 31 lines
Quoted 19 lines
Quoted 11 lines
The same kernel works fine when not running under the hypervisor.
It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.Calling
flush_cache_all()inflush_cache_vmap()makes it work. What isn't being correctly flushed? I see thatflush_pmd_entry()andcpu_v7_set_pte_ext()already flush the L1 and L2 translation table entries and I can't think of anything else that would need to be flushed (unless the mapped virtual addresses need to be flushed as well?)Quoted 7 lines
The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a branch predictor flush are required after a translation table entry update. This seems not to be done but adding this didn't seem to help (and using
local_flush_tlb_all()) inflush_cache_vmap()didn't help either).I don't see anything in the hypervisor that could be causing this as the fault is occurring at stage 1 and not stage 2 translation.
Quoted 7 lines
Interesting error, I don't have an immediate idea of what might be wrong, just some questions.
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
VTCR =
0x80002558which is: Outer Shareable; Normal memory, outer write-back write-allocate cacheable; Normal memory, inner write-back, write-allocate cacheable.
Quoted 8 lines
Quoted 5 lines
L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer write-back cacheable.These look sensible to me.
They look fine (UP system). BTW, I assume that the hypervisor also flushes the caches and TLBs for the stage 2 translation tables.
I think so. Cc'ing Ian Campbell who knows the hypervisor side better than me.
Quoted 6 lines
It could as well be a model bug but people are on holiday at the moment (and I'm off shortly as well, until 3rd of January). Could you try to disable the cacheability of the page table walks for both stage 1 (TTBRx with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since Linux does the correct cache flushing and I assume the hypervisor as well, this may work around possible model bug.
I can try this but probably not until the new year.
David
Quoted 55 lines · David Vrabel
Quoted 54 lines · Catalin Marinas
Quoted 10 lines · David Vrabel
Quoted 4 lines · Catalin Marinas
Quoted 3 lines · David Vrabel
When running the linux kernel on the ARMv7 envelope model as a guest under the Xen hypervisor there is a oops (see below for an example of the page translation fault) when trying to access ioremap()'d memory.
The translation tables for userspace seem to be also affected. The program repeatedly faults with a translation fault on the same address.
Putting acache_flush_all()after the call tohandle_mm_fault()in__do_page_fault()makes userspace work as well.With the classic page tables, on A15 we need this patch:
But that's to do with TLBs rather than cache and it only shows on real hardware rather than model.
Quoted 35 lines
Quoted 19 lines
Quoted 11 lines
The same kernel works fine when not running under the hypervisor.
It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.Calling
flush_cache_all()inflush_cache_vmap()makes it work. What isn't being correctly flushed? I see thatflush_pmd_entry()andcpu_v7_set_pte_ext()already flush the L1 and L2 translation table entries and I can't think of anything else that would need to be flushed (unless the mapped virtual addresses need to be flushed as well?)Quoted 7 lines
The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a branch predictor flush are required after a translation table entry update. This seems not to be done but adding this didn't seem to help (and using
local_flush_tlb_all()) inflush_cache_vmap()didn't help either).I don't see anything in the hypervisor that could be causing this as the fault is occurring at stage 1 and not stage 2 translation.
Quoted 7 lines
Interesting error, I don't have an immediate idea of what might be wrong, just some questions.
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
VTCR =
0x80002558which is: Outer Shareable; Normal memory, outer write-back write-allocate cacheable; Normal memory, inner write-back, write-allocate cacheable.L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer write-back cacheable.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
These look sensible to me.
They look fine (UP system). BTW, I assume that the hypervisor also flushes the caches and TLBs for the stage 2 translation tables.
I think so. Cc'ing Ian Campbell who knows the hypervisor side better than me.
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
Obviously we will need to make sure we do appropriate flushes when we start needing to change the p2m of a running guest etc. Currently our write_pte does a flush with DCCMVAC and in general our global flushes are at the more aggressive end of the scale (correctness before optimisation ;-)).
BTW the Xen code is all at
http://xenbits.xen.org/gitweb/?p=people/ianc/xen-unstable.git;a=tree;h=refs/heads/devel/arm;hb=devel/arm
interesting code from this PoV is likely to be xen/arch/arm/{p2m,domain_build}.c and xen/include/asm-arm/page.h
Ian.
Quoted 11 lines
Quoted 7 lines
It could as well be a model bug but people are on holiday at the moment (and I'm off shortly as well, until 3rd of January). Could you try to disable the cacheability of the page table walks for both stage 1 (TTBRx with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since Linux does the correct cache flushing and I assume the hypervisor as well, this may work around possible model bug.
I can try this but probably not until the new year.
David
Quoted 4 lines · Ian Campbell
Quoted 3 lines · David Vrabel
Quoted 2 lines · Catalin Marinas
Quoted 1 line · David Vrabel
On 22/12/11 14:49, Catalin Marinas wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
VTCR =
0x80002558which is: Outer Shareable; Normal memory, outer write-back write-allocate cacheable; Normal memory, inner write-back, write-allocate cacheable.
BTW, it would be better to mark the stage 2 table walks as inner shareable, i.e. VTCR = 0x80003558. That's because in case of SMP you would want page table accesses to snoop the caches of the other CPUs.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer write-back cacheable.These look sensible to me.
They look fine (UP system). BTW, I assume that the hypervisor also flushes the caches and TLBs for the stage 2 translation tables.
Quoted 6 lines
Quoted 2 lines
I think so. Cc'ing Ian Campbell who knows the hypervisor side better than me.
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
Quoted 5 lines
Obviously we will need to make sure we do appropriate flushes when we start needing to change the p2m of a running guest etc. Currently our
write_ptedoes a flush with DCCMVAC and in general our global flushes are at the more aggressive end of the scale (correctness before optimisation ;-)).
You can use DCCMVAU but could also check the coherent walk bits in ID_MMFR3 (assuming that the model works correctly).
Signature · 1 line
Catalin
Quoted 10 lines · Catalin Marinas
Quoted 9 lines · Ian Campbell
Quoted 8 lines · David Vrabel
Quoted 7 lines · Catalin Marinas
Quoted 6 lines · David Vrabel
Quoted 4 lines · Catalin Marinas
What's the value of the VTCR register for this guest? Are the translation table walks marked as cacheable? Also, are the page table attributes Normal Cacheable in the stage 2 translation? The processor chooses the more restrictive attribute between stage 1 and stage 2.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
VTCR =
0x80002558which is: Outer Shareable; Normal memory, outer write-back write-allocate cacheable; Normal memory, inner write-back, write-allocate cacheable.BTW, it would be better to mark the stage 2 table walks as inner shareable, i.e. VTCR =
0x80003558. That's because in case of SMP you would want page table accesses to snoop the caches of the other CPUs.
I did this. It didn't make any difference to the problem (I don't think you were suggesting it would).
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer write-back cacheable.These look sensible to me.
They look fine (UP system). BTW, I assume that the hypervisor also flushes the caches and TLBs for the stage 2 translation tables.
Quoted 6 lines
Quoted 2 lines
I think so. Cc'ing Ian Campbell who knows the hypervisor side better than me.
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
Quoted 5 lines
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
I added a copy of Linux's v7_flush_dcache_all after building the p2m but just before loading the VTTBR and it didn't help.
I turned off cacheability in VTCR and that didn't help either.
Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_ make a difference -- the kernel now boots.
I then tested just that change by itself and it seems to have done the trick.
Does that indicate a model bug?
FWIW ID_MMFR3 is 0x02102111 on the model I've got so you are right that it should to cacheable walks.
Cheers,
Ian.
Quoted 9 lines
Quoted 5 lines
Obviously we will need to make sure we do appropriate flushes when we start needing to change the p2m of a running guest etc. Currently our
write_ptedoes a flush with DCCMVAC and in general our global flushes are at the more aggressive end of the scale (correctness before optimisation ;-)).You can use DCCMVAU but could also check the coherent walk bits in
ID_MMFR3(assuming that the model works correctly).
Quoted 17 lines · Ian Campbell
Quoted 10 lines · Catalin Marinas
Quoted 3 lines · Ian Campbell
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
I added a copy of Linux's
v7_flush_dcache_allafter building the p2m but just before loading the VTTBR and it didn't help.I turned off cacheability in VTCR and that didn't help either.
Quoted 8 lines
Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
TTB_FLAGS_UPandTTB_FLAGS_SMPto useFOO_NCeverywhere) and this _did_ make a difference -- the kernel now boots.I then tested just that change by itself and it seems to have done the trick.
Does that indicate a model bug?
It is possible. The scenario I'm thinking of is that with cacheable PTWs enabled in TTBR, the model wrongly decides to use the intermediate physical address (IPA) to look up the caches and gets the wrong information.
I'll take this to the model guys but most likely they'll ask for an image to load and just run. Could you provide such simple image (minimal filesystem)? I'm not familiar with Xen and building it.
(I'll first ask the model guys, maybe they can spot the error easily without additional debugging).
Signature · 1 line
Catalin
Quoted 27 lines · Catalin Marinas
Quoted 17 lines · Ian Campbell
Quoted 10 lines · Catalin Marinas
Quoted 3 lines · Ian Campbell
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
I added a copy of Linux's
v7_flush_dcache_allafter building the p2m but just before loading the VTTBR and it didn't help.I turned off cacheability in VTCR and that didn't help either.
Quoted 8 lines
Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
TTB_FLAGS_UPandTTB_FLAGS_SMPto useFOO_NCeverywhere) and this _did_ make a difference -- the kernel now boots.I then tested just that change by itself and it seems to have done the trick.
Does that indicate a model bug?
Quoted 8 lines
It is possible. The scenario I'm thinking of is that with cacheable PTWs enabled in TTBR, the model wrongly decides to use the intermediate physical address (IPA) to look up the caches and gets the wrong information.
I'll take this to the model guys but most likely they'll ask for an image to load and just run. Could you provide such simple image (minimal filesystem)?
Sure. It'd be a Xen image & a Linux kernel image. The filesystem I'm using is currently the one which comes from your 3rd party whatsit pack but this bug stops us getting as far as trying to mount it anyway.
Quoted 4 lines
I'm not familiar with Xen and building it.
(I'll first ask the model guys, maybe they can spot the error easily without additional debugging).
OK, I'll wait till I hear back before pull it together.
Ian.
Hi Catalin,
Quoted 32 lines · Catalin Marinas
Quoted 17 lines · Ian Campbell
Quoted 10 lines · Catalin Marinas
Quoted 3 lines · Ian Campbell
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
I added a copy of Linux's
v7_flush_dcache_allafter building the p2m but just before loading the VTTBR and it didn't help.I turned off cacheability in VTCR and that didn't help either.
Quoted 8 lines
Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
TTB_FLAGS_UPandTTB_FLAGS_SMPto useFOO_NCeverywhere) and this _did_ make a difference -- the kernel now boots.I then tested just that change by itself and it seems to have done the trick.
Does that indicate a model bug?
It is possible. The scenario I'm thinking of is that with cacheable PTWs enabled in TTBR, the model wrongly decides to use the intermediate physical address (IPA) to look up the caches and gets the wrong information.
Quoted 6 lines
I'll take this to the model guys but most likely they'll ask for an image to load and just run. Could you provide such simple image (minimal filesystem)? I'm not familiar with Xen and building it.
(I'll first ask the model guys, maybe they can spot the error easily without additional debugging).
Did they conclude anything? Can I provide any more info?
Ian.
Quoted 33 lines · Ian Campbell
Quoted 32 lines · Catalin Marinas
Quoted 17 lines · Ian Campbell
Quoted 10 lines · Catalin Marinas
Quoted 3 lines · Ian Campbell
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
I added a copy of Linux's
v7_flush_dcache_allafter building the p2m but just before loading the VTTBR and it didn't help.I turned off cacheability in VTCR and that didn't help either.
Quoted 8 lines
Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
TTB_FLAGS_UPandTTB_FLAGS_SMPto useFOO_NCeverywhere) and this _did_ make a difference -- the kernel now boots.I then tested just that change by itself and it seems to have done the trick.
Does that indicate a model bug?
It is possible. The scenario I'm thinking of is that with cacheable PTWs enabled in TTBR, the model wrongly decides to use the intermediate physical address (IPA) to look up the caches and gets the wrong information.
Quoted 8 lines
Quoted 6 lines
I'll take this to the model guys but most likely they'll ask for an image to load and just run. Could you provide such simple image (minimal filesystem)? I'm not familiar with Xen and building it.
(I'll first ask the model guys, maybe they can spot the error easily without additional debugging).
Did they conclude anything? Can I provide any more info?
No, they didn't :(. They would like to run some image showing the problem so they can trace the internal state of the model. Unfortunately I didn't have time to look into this.
Is it possible for you to create an ELF image that can be loaded onto the model with some instructions to show the failure?
Thanks.
Signature · 1 line
Catalin
Quoted 41 lines · Catalin Marinas
Quoted 40 lines · Ian Campbell
Quoted 38 lines · Catalin Marinas
Quoted 17 lines · Ian Campbell
Quoted 10 lines · Catalin Marinas
Quoted 3 lines · Ian Campbell
At the moment we build the entire p2m before we ever load the VTTBR or enable stage-2 translations in the HCR. Is that sufficient or do we also need to flush something?
If the model does not correctly implement cacheable page table walks for either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate that it should), the hypervisor would need to clean the D-cache to the point of unification (not necessary to go to the point of coherency) before any of the state 2 translation tables are used.
I added a copy of Linux's
v7_flush_dcache_allafter building the p2m but just before loading the VTTBR and it didn't help.I turned off cacheability in VTCR and that didn't help either.
Quoted 8 lines
Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
TTB_FLAGS_UPandTTB_FLAGS_SMPto useFOO_NCeverywhere) and this _did_ make a difference -- the kernel now boots.I then tested just that change by itself and it seems to have done the trick.
Does that indicate a model bug?
It is possible. The scenario I'm thinking of is that with cacheable PTWs enabled in TTBR, the model wrongly decides to use the intermediate physical address (IPA) to look up the caches and gets the wrong information.
I'll take this to the model guys but most likely they'll ask for an image to load and just run. Could you provide such simple image (minimal filesystem)? I'm not familiar with Xen and building it.
(I'll first ask the model guys, maybe they can spot the error easily without additional debugging).
Quoted 8 lines
Quoted 1 line
Did they conclude anything? Can I provide any more info?
No, they didn't :(. They would like to run some image showing the problem so they can trace the internal state of the model. Unfortunately I didn't have time to look into this.
Is it possible for you to create an ELF image that can be loaded onto the model with some instructions to show the failure?
Yes I'll do that (tomorrow hopefully) and contact you off list with a pointer to the binaries etc.
Thanks!
Ian.
Quoted 6 lines · Catalin Marinas
It could as well be a model bug but people are on holiday at the moment (and I'm off shortly as well, until 3rd of January). Could you try to disable the cacheability of the page table walks for both stage 1 (TTBRx with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since Linux does the correct cache flushing and I assume the hypervisor as well, this may work around possible model bug.
Note that Linux, through and through, expects that page table entries are not negatively cached. Non-present page table entries are expected to be refetched without requiring any kind of TLB flushing at any point.
If later hardware is, then it's incompatible with how Linux works (and with every other CPU which Linux supports.) It'd mean a new raft of endless problems which only affect ARM. So, I'd recommend getting rid of this feature in hardware, not only because of this reason, but also because it's absolutely insane: negative caching probably does _more_ to hurt performance by taking up valuable positive TLB caching space. Not only that, but non-present page table entries aren't used very often, and when they are they're normally part of a slow path.
So I suspect that the amount of traction you'd get to solve such an issue in mainline would be fairly minimal.
Quoted 11 lines · Russell King - ARM Linux
Quoted 6 lines · Catalin Marinas
It could as well be a model bug but people are on holiday at the moment (and I'm off shortly as well, until 3rd of January). Could you try to disable the cacheability of the page table walks for both stage 1 (TTBRx with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since Linux does the correct cache flushing and I assume the hypervisor as well, this may work around possible model bug.
Note that Linux, through and through, expects that page table entries are not negatively cached. ?Non-present page table entries are expected to be refetched without requiring any kind of TLB flushing at any point.
The TLB on ARM does not do any negative caching. Full translations are only stored if the page table walk succeeds. You may get some partial caching (like level 1) if there is a valid entry in the page table but that's not negative caching. For this scenario we have the (pmd|pte)_free_tlb() implementation.
What caching attributes in TTBR/TTBCR/VTCR mean is looking up the page table in the L1 cache so that we don't need to do extra cache flushing.
Signature · 1 line
Catalin