facing undefined inconsistent cache issues on cortex-a9
Hi,
I am using linux-2.6.32 port on spear1300 platform, the port is largely based on realview code.
SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in SMP, but we are currently configuring Linux in non SMP mode, running Linux on the first core only, while second core is spinning forever in a tight loop.
Linux is running with default configuration w.r.t. cache and other things on core1. Since SMP is not enabled this means that SCU and local timers are off, L2 cache is also disabled. While on core2 only instruction cache is on and both MMU and Data cache is off.
We observed a crash while booting. Investigating on the crash we found that this was caused by a wrong value read by the core. Strangely enough, the new value was written few instruction before, but the core was still reading the old one.
What happens is someting like this:
r0 = X ; /* X is the address of a Linux variable */
str r5, [r0]; /* write new value */
...
ldr r5, [r0]; /* read back value - we get the old one (???) */
The value read here, just few lines later the new value was written, is the old one. Obviously we are expecting to find the new one.
Moreover, using the jtag debugger it looks clear that the new value is indeed in the memory, but surprisingly NOT in the data-cache.
If we put a read of X, before the store in order to avoid the write miss when doing 'str r5, [r0]', we see that the problem doesn't appear.
We are completely lost, any pointers would really help, thanks
regards
Shiraz
Quoted 33 lines
-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Shiraz HASHIM
Sent: Monday, May 24, 2010 9:14 AM
To: linux-arm-kernel at lists.infradead.org
Cc: 'Armando VISCONTI'; Vipin KUMAR; Viresh KUMAR
Subject: facing undefined inconsistent cache issues on cortex-a9Hi,
I am using linux-2.6.32 port on spear1300 platform, the port is largely based on realview code.SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in SMP, but we are currently configuring Linux in non SMP mode, running Linux on the first core only, while second core is spinning forever in a tight loop.
Linux is running with default configuration w.r.t. cache and other things on core1. Since SMP is not enabled this means that SCU and local timers are off, L2 cache is also disabled. While on core2 only instruction cache is on and both MMU and Data cache is off.
We observed a crash while booting. Investigating on the crash we found that this was caused by a wrong value read by the core. Strangely enough, the new value was written few instruction before, but the core was still reading the old one.
What happens is someting like this:
r0 = X ; /* X is the address of a Linux variable */
str r5, [r0]; /* write new value */
...
ldr r5, [r0]; /* read back value - we get the old one (???) */
Quoted 8 lines
The value read here, just few lines later the new value was written, is the old one. Obviously we are expecting to find the new one.
Moreover, using the jtag debugger it looks clear that the new value is indeed in the memory, but surprisingly NOT in the data-cache.
If we put a read of X, before the store in order to avoid the write miss when doing 'str r5, [r0]', we see that the problem doesn't appear.
Does this work if disable L1D as well on the processor you are running Linux ??
Regards,
Santosh
Quoted 36 lines · Santosh
Quoted 36 lines
-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Shiraz HASHIM
Sent: Monday, May 24, 2010 9:14 AM
To: linux-arm-kernel at lists.infradead.org
Cc: 'Armando VISCONTI'; Vipin KUMAR; Viresh KUMAR
Subject: facing undefined inconsistent cache issues on cortex-a9Hi,
I am using linux-2.6.32 port on spear1300 platform, the port is largely based on realview code.SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in SMP, but we are currently configuring Linux in non SMP mode, running Linux on the first core only, while second core is spinning forever in a tight loop.
Linux is running with default configuration w.r.t. cache and other things on core1. Since SMP is not enabled this means that SCU and local timers are off, L2 cache is also disabled. While on core2 only instruction cache is on and both MMU and Data cache is off.
We observed a crash while booting. Investigating on the crash we found that this was caused by a wrong value read by the core. Strangely enough, the new value was written few instruction before, but the core was still reading the old one.
What happens is someting like this:
r0 = X ; /* X is the address of a Linux variable */
str r5, [r0]; /* write new value */
...
ldr r5, [r0]; /* read back value - we get the old one (???) */The value read here, just few lines later the new value was written, is the old one. Obviously we are expecting to find the new one.
Quoted 8 lines
Quoted 6 lines
Moreover, using the jtag debugger it looks clear that the new value is indeed in the memory, but surprisingly NOT in the data-cache.
If we put a read of X, before the store in order to avoid the write miss when doing 'str r5, [r0]', we see that the problem doesn't appear.Does this work if disable L1D as well on the processor you are running Linux ??
If we disable the L1 DCache we are never able to boot. But this might be a different problem.
If we disable ICache and leave DCache on it is much more stable, and I think it might be related to timing issue in the data path.
We are using Cortex A9 r2p1.
Any clue?
Arm
Signature · 8 lines
-- "Every step appears to be the unavoidable consequence of the -- preceding one." (A. Einstein) -- Armando Visconti Mobile: (+39) 346 8879146 Senior SW Engineer Fax: (+39) 02 93519290 CPG Work: (+39) 02 93519683 Computer System Division e-mail: armando.visconti at st.com ST Microelectronics TINA: 051 4683
Quoted 50 lines
-----Original Message-----
From: Armando VISCONTI [mailto:armando.visconti at st.com]
Sent: Tuesday, May 25, 2010 5:38 PM
To: Shilimkar, Santosh
Cc: Shiraz HASHIM; linux-arm-kernel at lists.infradead.org; Vipin KUMAR; Viresh KUMAR
Subject: Re: facing undefined inconsistent cache issues on cortex-a9Quoted 42 lines · Santosh
Quoted 42 lines
-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Shiraz HASHIM
Sent: Monday, May 24, 2010 9:14 AM
To: linux-arm-kernel at lists.infradead.org
Cc: 'Armando VISCONTI'; Vipin KUMAR; Viresh KUMAR
Subject: facing undefined inconsistent cache issues on cortex-a9Hi,
I am using linux-2.6.32 port on spear1300 platform, the port is largely based on realview code.SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in SMP, but we are currently configuring Linux in non SMP mode, running Linux on the first core only, while second core is spinning forever in a tight loop.
Linux is running with default configuration w.r.t. cache and other things on core1. Since SMP is not enabled this means that SCU and local timers are off, L2 cache is also disabled. While on core2 only instruction cache is on and both MMU and Data cache is off.
We observed a crash while booting. Investigating on the crash we found that this was caused by a wrong value read by the core. Strangely enough, the new value was written few instruction before, but the core was still reading the old one.
What happens is someting like this:
r0 = X ; /* X is the address of a Linux variable */
str r5, [r0]; /* write new value */
...
ldr r5, [r0]; /* read back value - we get the old one (???) */The value read here, just few lines later the new value was written, is the old one. Obviously we are expecting to find the new one.
Moreover, using the jtag debugger it looks clear that the new value is indeed in the memory, but surprisingly NOT in the data-cache.
If we put a read of X, before the store in order to avoid the write miss when doing 'str r5, [r0]', we see that the problem doesn't appear.
Quoted 8 lines
Quoted 2 lines
Does this work if disable L1D as well on the processor you are running Linux ??
If we disable the L1 DCache we are never able to boot. But this might be a different problem.
If we disable ICache and leave DCache on it is much more stable, and I think it might be related to timing issue in the data path.
For me, it seems that Cache isn't invalidated properly which leads to this issue.
May be you can report the complete crash-log which can give more idea on the issue.
Regards,
Santosh
Quoted 10 lines
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh
Quoted 1 line
Quoted 1 line
Does this work if disable L1D as well on the processor you are running
Linux ??
Quoted 4 lines
Quoted 1 line
If we disable the L1 DCache we are never able to boot. But this might be a different problem.
Quoted 7 lines
Quoted 3 lines
If we disable ICache and leave DCache on it is much more stable, and I think it might be related to timing issue in the data path.
For me, it seems that Cache isn't invalidated properly which leads to this issue.
May be you can report the complete crash-log which can give more idea on the issue.
If you are unable to boot with dcache off then maybe your memory controller or memory part has timing issues. Cache on will trigger burst accesses using different timings. Such an early failure is more likely an issue outside of the arm unless you are doing something really odd.
Regards,
Richard W.
Hello Woodruff,
Quoted 13 lines · Richard
Quoted 13 lines
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh
Quoted 1 line
Quoted 1 line
Does this work if disable L1D as well on the processor you are running
Linux ??
Quoted 7 lines
Quoted 1 line
If we disable the L1 DCache we are never able to boot. But this might be a different problem.
If we disable ICache and leave DCache on it is much more stable, and I think it might be related to timing issue in the data path.
Quoted 6 lines
Quoted 4 lines
For me, it seems that Cache isn't invalidated properly which leads to this issue.
May be you can report the complete crash-log which can give more idea on the issue.If you are unable to boot with dcache off then maybe your memory controller or memory part has timing issues. Cache on will trigger burst accesses using different timings. Such an early failure is more likely an issue outside of the arm unless you are doing something really odd.
thanks for your inputs. We have some updates in this regard.
In our architecture we have 2 ports coming out of the A9SM (multi core cortex) block to the interconnect matrix and the transfers are distributed between them.
When we used the address filtering option of SCU to divide and mutually exclude the traffic between port0 and port1 of A9SM(multi core cortex) block. Like, assigning 1 half of the address space to port 1 and rest goes through port0. This solved the problem with the same set of cache and other system configurations.
We are not using L2 cache and it is kept disabled. The SCU also is only used for address filtering and its synchronisation function is disabled (as we are not bringing up the second core)
Now, we are not able to understand what it means. Do we need to divide the traffic between ports as described and it is normal. Or there is some problem in the way the transfers are handled in the system.
regards
Shiraz
Quoted 10 lines
From: Shiraz HASHIM [mailto:shiraz.hashim at st.com]
Sent: Friday, June 04, 2010 8:55 AM
To: Woodruff, Richardthanks for your inputs. We have some updates in this regard.
In our architecture we have 2 ports coming out of the A9SM (multi core cortex) block to the interconnect matrix and the transfers are distributed between them.
Quoted 14 lines
When we used the address filtering option of SCU to divide and mutually exclude
the traffic between port0 and port1 of A9SM(multi core cortex) block. Like, assigning 1 half of the address space to port 1 and rest goes through port0. This solved the problem with the same set of cache and other system configurations.
We are not using L2 cache and it is kept disabled. The SCU also is only used for
address filtering and its synchronisation function is disabled (as we are not bringing up the second core)
Now, we are not able to understand what it means. Do we need to divide the traffic between ports as described and it is normal. Or there is some problem in
the way the transfers are handled in the system.
Off hand I'm not sure of the right answer. The design I'm working on does not require the filtering feature so I've not looked much at it outside of some errata. I am aware of another design which accesses a bit larger address space which does use filtering.
I'd guess you need to understand your cortex-memmap + AXI-to-memory bus adapter characteristics.
Regards,
Richard W.