Query about: ARM11 MPCore: preemption/task migration cache coherency

I'm using ARM11 MPCore on linux-2.6.34, unfortunately I have random panic/segment fault with task migration. I noticed a patch set [ARM11 MPCore: preemption/task migration cache coherency fixups] to fix such issues in here:

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-October/069851.html

It seems there is no follow ups, is there official patch to fix such issues?

Bill,

Quoted 5 lines · On Wed, May 09, 2012 at 10:11:05AM +0100, bill4carson wrote:

I'm using ARM11 MPCore on linux-2.6.34, unfortunately I have random panic/segment fault with task migration. I noticed a patch set [ARM11 MPCore: preemption/task migration cache coherency fixups] to fix such issues in here:

Quoted 4 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-October/069851.html

It seems there is no follow ups, is there official patch to fix such issues?

Let's be honest: you haven't given us a lot to go on here. Perhaps you could answer the following?

(1) Do you experience the same issues with a more recent kernel?

(2) If you apply the patches linked to above, does it fix your problem?

(3) If you can reproduce on current mainline, do you have a testcase?

(4) Does disabling CONFIG_PREEMPT make the problem disappear?

[10870.314465] Unhandled fault: alignment exception (0x821) at 0xebfffee6
[10870.315347] Unhandled fault: alignment exception (0x821) at 0xebfffee6
[10870.354917] Unhandled fault: alignment exception (0x821) at 0xebfffee6

That should provide us with some information about the problem.

Thanks,

Will

Bill,

Quoted 9 lines · On Wed, May 09, 2012 at 10:11:05AM +0100, bill4carson wrote:

I'm using ARM11 MPCore on linux-2.6.34, unfortunately I have random panic/segment fault with task migration. I noticed a patch set [ARM11 MPCore: preemption/task migration cache coherency fixups] to fix such issues in here:

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-October/069851.html

It seems there is no follow ups, is there official patch to fix such issues?

Let's be honest: you haven't given us a lot to go on here. Perhaps you could answer the following?

(1) Do you experience the same issues with a more recent kernel?
(2) If you apply the patches linked to above, does it fix your problem?
(3) If you can reproduce on current mainline, do you have a testcase?
(4) Does disabling CONFIG_PREEMPT make the problem disappear?

That should provide us with some information about the problem.

But I have no idea on how to accomplish the v6_flush_kern_cache_all, maybe IPI is needed?

arch/arm/mm/cache-v6.S ENDPROC(v6_coherent_kern_range) · line 179

Quoted 13 lines · On Tue, May 29, 2012 at 06:28:11AM +0100, bill4carson wrote:

arch/arm/mm/cache-v6.S +4 -0

@@ -170,6 +170,10 @@ ENDPROC(v6_coherent_kern_range)
   ENTRY(v6_flush_kern_dcache_area)
          add     r1, r0, r1
   1:
+#ifdef CONFIG_SMP
+       ldr     r2, [r0]                        @ read for ownership
+       str     r2, [r0]                        @ write for ownership
+#endif /* CONFIG_SMP */
   #ifdef HARVARD_CACHE
          mcr     p15, 0, r0, c7, c14, 1          @ clean&  invalidate D line
   #else

I don't think the invalidation is needed here, so you probably don't need to hack this function at all.

Quoted 3 lines

But I have no idea on how to accomplish the v6_flush_kern_cache_all, maybe IPI is needed?

We could add an IPI to invalidate the I-caches on the other cores, however I haven't checked to see if we could instead do something on the CPU migration path which avoid the need for the broadcasting.

Will

Quoted 17 lines · Will Deacon

arch/arm/mm/cache-v6.S ENDPROC(v6_coherent_kern_range) · line 179

Quoted 13 lines · On Tue, May 29, 2012 at 06:28:11AM +0100, bill4carson wrote:

arch/arm/mm/cache-v6.S +4 -0

@@ -170,6 +170,10 @@ ENDPROC(v6_coherent_kern_range)
   ENTRY(v6_flush_kern_dcache_area)
          add     r1, r0, r1
   1:
+#ifdef CONFIG_SMP
+       ldr     r2, [r0]                        @ read for ownership
+       str     r2, [r0]                        @ write for ownership
+#endif /* CONFIG_SMP */
   #ifdef HARVARD_CACHE
          mcr     p15, 0, r0, c7, c14, 1          @ clean&  invalidate D line
   #else

I don't think the invalidation is needed here, so you probably don't need to hack this function at all.

1. task t runs on CPU 0, it executes one program in nand flash, so first task t read *part* of this program into its local Dcache, let's call it part_a;

2. task t migrates from CPU0 onto CPU1, in there reads the rest of

    program into its local Dcache, label it part_b;

3. on CPU1, task t flush the whole address range of this program, As with ARM11 MPCore, this flush is locally effective, after flush part_a still resides in CPU0 L1 Dcache.

4. When this program gets executed, CPU 1 fetch its instructions from Icache, SCU will notice this action,

Question:
At this point, is stale part_a in main memory *or* L1 Dcache part_a in CPU0 routed into CPU1 as instruction?

Hi,

Quoted 18 lines · On Wed, May 30, 2012 at 11:01:59AM +0100, bill4carson wrote:

Quoted 17 lines · Will Deacon

arch/arm/mm/cache-v6.S ENDPROC(v6_coherent_kern_range) · line 179

Quoted 13 lines · On Tue, May 29, 2012 at 06:28:11AM +0100, bill4carson wrote:

arch/arm/mm/cache-v6.S +4 -0

@@ -170,6 +170,10 @@ ENDPROC(v6_coherent_kern_range)
    ENTRY(v6_flush_kern_dcache_area)
           add     r1, r0, r1
    1:
+#ifdef CONFIG_SMP
+       ldr     r2, [r0]                        @ read for ownership
+       str     r2, [r0]                        @ write for ownership
+#endif /* CONFIG_SMP */
    #ifdef HARVARD_CACHE
           mcr     p15, 0, r0, c7, c14, 1          @ clean&   invalidate D line
    #else

I don't think the invalidation is needed here, so you probably don't need to hack this function at all.

...

Quoted 11 lines

1. task t runs on CPU 0, it executes one program in nand flash, so first task t read *part* of this program into its local Dcache, let's call it part_a;

2. task t migrates from CPU0 onto CPU1, in there reads the rest of

     program into its local Dcache, label it part_b;

3. on CPU1, task t flush the whole address range of this program, As with ARM11 MPCore, this flush is locally effective, after flush part_a still resides in CPU0 L1 Dcache.

Quoted 6 lines

4. When this program gets executed, CPU 1 fetch its instructions from Icache, SCU will notice this action,

Question:
At this point, is stale part_a in main memory *or* L1 Dcache part_a in CPU0 routed into CPU1 as instruction?

This has been discussed in the past. The first thing is to disable CONFIG_PREEMPT to make sure you are not preempted between page copying and task execution. The code doing the page copy needs to call flush_dcache_page() on the CPU where the copy occurred and Linux does a non-lazy D-cache flush. But there are many situations where Linux doesn't do this (you can google for flush_dcache_page and my name to find a few places where I tried to fix this).

AFAIK, The SCU only snoops the D-cache, not the I-cache. We have a full I-cache invalidation during task migration.

Quoted 2 lines · Catalin Marinas

AFAIK, The SCU only snoops the D-cache, not the I-cache. We have a full I-cache invalidation during task migration.

                             ^^^^^^^^^^^^

Could you please point it to me?

Quoted 38 lines · Catalin Marinas

Quoted 18 lines · On Wed, May 30, 2012 at 11:01:59AM +0100, bill4carson wrote:

Quoted 17 lines · Will Deacon

arch/arm/mm/cache-v6.S ENDPROC(v6_coherent_kern_range) · line 179

Quoted 13 lines · On Tue, May 29, 2012 at 06:28:11AM +0100, bill4carson wrote:

arch/arm/mm/cache-v6.S +4 -0

@@ -170,6 +170,10 @@ ENDPROC(v6_coherent_kern_range)
    ENTRY(v6_flush_kern_dcache_area)
           add     r1, r0, r1
    1:
+#ifdef CONFIG_SMP
+       ldr     r2, [r0]                        @ read for ownership
+       str     r2, [r0]                        @ write for ownership
+#endif /* CONFIG_SMP */
    #ifdef HARVARD_CACHE
           mcr     p15, 0, r0, c7, c14, 1          @ clean&   invalidate D line
    #else

I don't think the invalidation is needed here, so you probably don't need to hack this function at all.

...

Quoted 11 lines

1. task t runs on CPU 0, it executes one program in nand flash, so first task t read *part* of this program into its local Dcache, let's call it part_a;

2. task t migrates from CPU0 onto CPU1, in there reads the rest of

     program into its local Dcache, label it part_b;

3. on CPU1, task t flush the whole address range of this program, As with ARM11 MPCore, this flush is locally effective, after flush part_a still resides in CPU0 L1 Dcache.

Quoted 6 lines

4. When this program gets executed, CPU 1 fetch its instructions from Icache, SCU will notice this action,

Question:
At this point, is stale part_a in main memory *or* L1 Dcache part_a in CPU0 routed into CPU1 as instruction?

Quoted 7 lines

This has been discussed in the past. The first thing is to disable CONFIG_PREEMPT to make sure you are not preempted between page copying and task execution. The code doing the page copy needs to call flush_dcache_page() on the CPU where the copy occurred and Linux does a non-lazy D-cache flush. But there are many situations where Linux doesn't do this (you can google for flush_dcache_page and my name to find a few places where I tried to fix this).

Thanks:)

Quoted 47 lines · On Thu, May 31, 2012 at 04:11:46AM +0100, bill4carson wrote:

Quoted 45 lines · Catalin Marinas

Quoted 18 lines · On Wed, May 30, 2012 at 11:01:59AM +0100, bill4carson wrote:

Quoted 17 lines · Will Deacon

arch/arm/mm/cache-v6.S ENDPROC(v6_coherent_kern_range) · line 179

Quoted 13 lines · On Tue, May 29, 2012 at 06:28:11AM +0100, bill4carson wrote:

arch/arm/mm/cache-v6.S +4 -0

@@ -170,6 +170,10 @@ ENDPROC(v6_coherent_kern_range)
     ENTRY(v6_flush_kern_dcache_area)
            add     r1, r0, r1
     1:
+#ifdef CONFIG_SMP
+       ldr     r2, [r0]                        @ read for ownership
+       str     r2, [r0]                        @ write for ownership
+#endif /* CONFIG_SMP */
     #ifdef HARVARD_CACHE
            mcr     p15, 0, r0, c7, c14, 1          @ clean&    invalidate D line
     #else

I don't think the invalidation is needed here, so you probably don't need to hack this function at all.

...

Quoted 11 lines

1. task t runs on CPU 0, it executes one program in nand flash, so first task t read *part* of this program into its local Dcache, let's call it part_a;

2. task t migrates from CPU0 onto CPU1, in there reads the rest of

      program into its local Dcache, label it part_b;

3. on CPU1, task t flush the whole address range of this program, As with ARM11 MPCore, this flush is locally effective, after flush part_a still resides in CPU0 L1 Dcache.

Quoted 6 lines

4. When this program gets executed, CPU 1 fetch its instructions from Icache, SCU will notice this action,

Question:
At this point, is stale part_a in main memory *or* L1 Dcache part_a in CPU0 routed into CPU1 as instruction?

This has been discussed in the past. The first thing is to disable CONFIG_PREEMPT to make sure you are not preempted between page copying and task execution. The code doing the page copy needs to call flush_dcache_page() on the CPU where the copy occurred and Linux does a non-lazy D-cache flush. But there are many situations where Linux doesn't do this (you can google for flush_dcache_page and my name to find a few places where I tried to fix this).

Quoted 1 line

Thanks:)

BTW, see this as a starting point (and a hack):

http://article.gmane.org/gmane.linux.ports.arm.kernel/51556

Quoted 3 lines · Catalin Marinas

BTW, see this as a starting point (and a hack):

http://article.gmane.org/gmane.linux.ports.arm.kernel/51556

I think we has some mis-understanding here :(

As for:v6_flush_kern_dcache_area/v6_flush_kern_dcache_all these two hooks is supposed to be globally effective, *not* locally!

Hence, there should below fix to make it as globally effective.

(quoted diff) +4 -0

+#ifdef CONFIG_SMP
+ ? ? ? ldr ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
+ ? ? ? str ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownership
+#endif /* CONFIG_SMP */

Or there maybe some other better solution for this issue.

Quoted 19 lines · On 31 May 2012 11:38, bill4carson<[email protected]> wrote:

Quoted 3 lines · Catalin Marinas

BTW, see this as a starting point (and a hack):

http://article.gmane.org/gmane.linux.ports.arm.kernel/51556

I think we has some mis-understanding here :(

As for:v6_flush_kern_dcache_area/v6_flush_kern_dcache_all these two hooks is supposed to be globally effective, *not* locally!

Hence, there should below fix to make it as globally effective.

(quoted diff) +4 -0

+#ifdef CONFIG_SMP
+       ldr     r2, [r0]                        @ read for ownership
+       str     r2, [r0]                        @ write for ownership
+#endif /* CONFIG_SMP */

Quoted 1 line

Or there maybe some other better solution for this issue.

I still didn't fully understand what the problem is. So, to make sure, if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

Quoted 5 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure, if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

This may cause error in some cases for example:

1) Task running on Core-0 loading text section into memory. ? It was preempted and then migrate into Core-1;

2) On Core-1, this task continue loading it and then
? "flush_dcache_page" to make sure the loaded text section write ? into main memory.

3) Task tend to the loaded text section and running it.

If the "flush_dcache_page" was not global effective, there maybe data still in Core-0's data cache, not write into main memory. Thus in step 3, error instruction maybe fetched thus cause strange error.

Quoted 7 lines · On 31 May 2012 13:06, bill4carson<[email protected]> wrote:

Quoted 5 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure, if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

Quoted 4 lines

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

Yes, I know this.

Quoted 4 lines

This may cause error in some cases for example:

1) Task running on Core-0 loading text section into memory. It was preempted and then migrate into Core-1;

BTW, do you have CONFIG_PREEMPT enabled?

To be clear - is your application reading some data from flash and trying to execute or it's the kernel doing the load via the page/prefetch abort mechanism?

If the latter, task running on core 0 gets a prefetch abort when trying to execute some code. The kernel reads the page from flash (via mtd, block layer, VFS) and copies it into RAM. It can be on any CPU as long as it calls flush_dcache_page on the same CPU that copied the data.

No matter where the task was running or migrated to, if the code doing the copy also called flush_dcache_page() on the same core, there is no data left in the D-cache for that page.

Quoted 4 lines

2) On Core-1, this task continue loading it and then "flush_dcache_page" to make sure the loaded text section write into main memory.

The flush_dcache_page() must be called by the code doing the copy. If that copy happened on core 0, the call is done there and not where the task migrated. We don't do lazy flushing on ARM11MPCore.

Quoted 6 lines

3) Task tend to the loaded text section and running it.

If the "flush_dcache_page" was not global effective, there maybe data still in Core-0's data cache, not write into main memory. Thus in step 3, error instruction maybe fetched thus cause strange error.

This can only happen if you have either preempt enabled (so that the kernel code doing the copy is migrated) or the mtd driver or fs do not call flush_dcache_page().

Quoted 8 lines · Catalin Marinas

Quoted 7 lines · On 31 May 2012 13:06, bill4carson<[email protected]> wrote:

Quoted 5 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure, if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

Quoted 6 lines

Quoted 4 lines

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

Yes, I know this.

Then, why not fix "flush_dcache_page" to make it globally effective?

Quoted 7 lines

Quoted 4 lines

This may cause error in some cases for example:

1) Task running on Core-0 loading text section into memory. It was preempted and then migrate into Core-1;

BTW, do you have CONFIG_PREEMPT enabled?

Yes, CONFIG_PREEMPT enabled. Thus cause the task was preempted. :-)

Quoted 3 lines

To be clear - is your application reading some data from flash and trying to execute or it's the kernel doing the load via the page/prefetch abort mechanism?

In my current case, it is yaffs root file system.

Quoted 6 lines

If the latter, task running on core 0 gets a prefetch abort when trying to execute some code. The kernel reads the page from flash (via mtd, block layer, VFS) and copies it into RAM. It can be on any CPU as long as it calls flush_dcache_page on the same CPU that copied the data.

Quoted 3 lines

No matter where the task was running or migrated to, if the code doing the copy also called flush_dcache_page() on the same core, there is no data left in the D-cache for that page.

Yes, I agree with it.
But how to flush the data cache on the same core with PREEMPT enabled?

And, I think according to the design, there is no such operation that guarantee it.

But any way, I think flush_dcache_page should be global effective. If ARMv6 MPCore didn't make it, we should try to accomplish it.

Quoted 9 lines · On Thu, May 31, 2012 at 06:51:22AM +0100, bill4carson wrote:

Quoted 8 lines · Catalin Marinas

Quoted 7 lines · On 31 May 2012 13:06, bill4carson<[email protected]> wrote:

Quoted 5 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure, if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

Yes, I know this.

Then, why not fix "flush_dcache_page" to make it globally effective?

Performance?

And it's also just ARM11MPCore microarchitecture specific.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

This may cause error in some cases for example:

Quoted 6 lines

Quoted 5 lines

Quoted 2 lines

1) Task running on Core-0 loading text section into memory.

    It was preempted and then migrate into Core-1;

BTW, do you have CONFIG_PREEMPT enabled?

Yes, CONFIG_PREEMPT enabled. Thus cause the task was preempted. :-)

I told you that CONFIG_PREEMPT is not supported on ARM11MPCore :).

Quoted 13 lines

Quoted 4 lines

To be clear - is your application reading some data from flash and trying to execute or it's the kernel doing the load via the page/prefetch abort mechanism?

In my current case, it is yaffs root file system.

Quoted 6 lines

If the latter, task running on core 0 gets a prefetch abort when trying to execute some code. The kernel reads the page from flash (via mtd, block layer, VFS) and copies it into RAM. It can be on any CPU as long as it calls flush_dcache_page on the same CPU that copied the data.

Quoted 6 lines

Quoted 3 lines

No matter where the task was running or migrated to, if the code doing the copy also called flush_dcache_page() on the same core, there is no data left in the D-cache for that page.

Yes, I agree with it.
But how to flush the data cache on the same core with PREEMPT enabled?

That's not easily possible. But you may get better results with VOLUNTARY_PREEMPT.

Quoted 2 lines

And, I think according to the design, there is no such operation that guarantee it.

RFO/WFO tricks only work on ARM11MPCore.

Quoted 2 lines

But any way, I think flush_dcache_page should be global effective. If ARMv6 MPCore didn't make it, we should try to accomplish it.

Do some performance tests first.

Quoted 18 lines · Catalin Marinas

Quoted 17 lines · On Thu, May 31, 2012 at 06:51:22AM +0100, bill4carson wrote:

Quoted 16 lines · Catalin Marinas

Quoted 14 lines · On 31 May 2012 13:06, bill4carson<[email protected]> wrote:

Quoted 8 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure,
if you run some applications from flash using a yaffs filesystem, you
get random crashes. Is this correct? If yes, a solution is to actually
call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

Quoted 8 lines

Quoted 3 lines

Quoted 1 line

Yes, I know this.

Then, why not fix "flush_dcache_page" to make it globally effective?

Performance?

And it's also just ARM11MPCore microarchitecture specific.

Quoted 10 lines

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

This may cause error in some cases for example:

1) Task running on Core-0 loading text section into memory.

    It was preempted and then migrate into Core-1;

BTW, do you have CONFIG_PREEMPT enabled?

Yes, CONFIG_PREEMPT enabled. Thus cause the task was preempted. :-)

I told you that CONFIG_PREEMPT is not supported on ARM11MPCore :).

Point!

Is it better to add comment, such as "PREEMPT is not supported for ARM11MPCore" in somewhere(for now, I don't find such place)? then custom will be alerted with such notice when they trying change to PREEMPT.

And again, thanks for your patience with me :)

Quoted 33 lines

Quoted 13 lines

Quoted 4 lines

To be clear - is your application reading some data from flash and trying to execute or it's the kernel doing the load via the page/prefetch abort mechanism?

In my current case, it is yaffs root file system.

Quoted 6 lines

If the latter, task running on core 0 gets a prefetch abort when trying to execute some code. The kernel reads the page from flash (via mtd, block layer, VFS) and copies it into RAM. It can be on any CPU as long as it calls flush_dcache_page on the same CPU that copied the data.

Quoted 6 lines

Quoted 3 lines

No matter where the task was running or migrated to, if the code doing the copy also called flush_dcache_page() on the same core, there is no data left in the D-cache for that page.

Yes, I agree with it.
But how to flush the data cache on the same core with PREEMPT enabled?

That's not easily possible. But you may get better results with VOLUNTARY_PREEMPT.

Quoted 2 lines

And, I think according to the design, there is no such operation that guarantee it.

RFO/WFO tricks only work on ARM11MPCore.

Quoted 2 lines

But any way, I think flush_dcache_page should be global effective. If ARMv6 MPCore didn't make it, we should try to accomplish it.

Do some performance tests first.

Quoted 28 lines · On 2012?05?31? 15:21, bill4carson wrote:

Quoted 27 lines · Catalin Marinas

Quoted 18 lines · On Thu, May 31, 2012 at 06:51:22AM +0100, bill4carson wrote:

Quoted 17 lines · Catalin Marinas

Quoted 14 lines · On 31 May 2012 13:06, bill4carson<[email protected]> ? wrote:

Quoted 7 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure,
if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually
call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

Quoted 4 lines

Quoted 1 line

Yes, I know this.

Then, why not fix "flush_dcache_page" to make it globally effective?

Performance?

Quoted 2 lines

Quoted 2 lines

And it's also just ARM11MPCore microarchitecture specific.

Yes, seems newer CPUs has no such limitation thus this function is global effective naturally. :-)

And , I find Mips's c-r4k also has this issue but it use IPI to make it. Details in arch/mips/mm/c-r4k.c.

Quoted 27 lines · On 31 May 2012 15:46,<[email protected]> wrote:

Quoted 26 lines · On 2012?05?31? 15:21, bill4carson wrote:

Quoted 25 lines · Catalin Marinas

Quoted 11 lines · On Thu, May 31, 2012 at 06:51:22AM +0100, bill4carson wrote:

Quoted 10 lines · Catalin Marinas

Quoted 9 lines · On 31 May 2012 13:06, bill4carson<[email protected]> wrote:

Quoted 7 lines · Catalin Marinas

I still didn't fully understand what the problem is. So, to make sure,
if you run some applications from flash using a yaffs filesystem, you get random crashes. Is this correct? If yes, a solution is to actually
call flush_dcache_page() on the CPU that does the page copying from flash into RAM, which could be the yaffs filesystem.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

The story goes like this:
function "flush_dcache_page" should be global effective but in ARMv6 MPCore, it was not, it was just local effective due to hardware design.

Yes, I know this.

Then, why not fix "flush_dcache_page" to make it globally effective?

Performance?

And it's also just ARM11MPCore microarchitecture specific.

Quoted 5 lines

Yes, seems newer CPUs has no such limitation thus this function is global effective naturally. :-)

And , I find Mips's c-r4k also has this issue but it use IPI to make it. Details in arch/mips/mm/c-r4k.c.

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

Quoted 3 lines · Catalin Marinas

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

I think write for ownership is necessary for flush_dcache_xxx: Read guarantee local data cache get newest data, at the same time write guarantee the data can be flushed into memory.

See Section 7.1 of ARM11 MPCore. Processor Technical Reference Manual(Revision: r2p0) P146/728:
======
Clean Applies to write-back data caches. If the cache line targeted by the Clean operation contains stored data that has not yet been written out to main memory, it is written to main memory, and the line is marked as clean.
======

So I am afraid without the write action, the "clean & invalidate" action later will not write data back to main memory.

Another question here: Why the flush_kern_dcache_xxx in arch/arm/mm/cache-v6 use "clean & invalidate" progress instead of "clean"? Seems clean is enough here.

Quoted 6 lines

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

PREEMPT. :-)

Get it. But currently, I can't find anything related to ARMv6 MPCore conflict with PREEMPT. So if it is also necessary to add something in Documentation and related Kconfig to describe it and make sure PREEMPT can't been enabled on such CPUs?

Quoted 17 lines · snakky.zhang at gmail.com

Quoted 3 lines · Catalin Marinas

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

I think write for ownership is necessary for flush_dcache_xxx: Read guarantee local data cache get newest data, at the same time write guarantee the data can be flushed into memory.

See Section 7.1 of ARM11 MPCore. Processor Technical Reference Manual(Revision: r2p0) P146/728:
======
Clean Applies to write-back data caches. If the cache line targeted by the Clean operation contains stored data that has not yet been written out to main memory, it is written to main memory, and the line is marked as clean.
======

Quoted 2 lines

So I am afraid without the write action, the "clean& invalidate" action later will not write data back to main memory.

If there is a dirty cache line, it will be written to memory by the clean&invalidate operation. If the data in the cache line is in a clean state, it means that it is identical to the main memory (or L2 if present).

With just a read, clean&invalidate would not invalidate (remove) the cache lines from the other CPUs. Doing a write forces the cache line to only be present on the current CPU (though automatically invalidating it on the other CPUs).

Quoted 3 lines

Another question here: Why the flush_kern_dcache_xxx in arch/arm/mm/cache-v6 use "clean& invalidate" progress instead of "clean"? Seems clean is enough here.

I think in the context of VIPT caches clean would be enough.

Quoted 6 lines

Quoted 6 lines

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

Quoted 6 lines

PREEMPT. :-)

Get it. But currently, I can't find anything related to ARMv6 MPCore conflict with PREEMPT. So if it is also necessary to add something in Documentation and related Kconfig to describe it and make sure PREEMPT can't been enabled on such CPUs?

Well, we either get it to work or, if not possible, we add a comment. Let's try the former option first :)

Quoted 6 lines

Yes, seems newer CPUs has no such limitation thus this function is global effective naturally. :-)

And , I find Mips's c-r4k also has this issue but it use IPI to make it. Details in arch/mips/mm/c-r4k.c.

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

Quoted 9 lines

Quoted 5 lines

Yes, seems newer CPUs has no such limitation thus this function is global effective naturally. :-)

And , I find Mips's c-r4k also has this issue but it use IPI to make it. Details in arch/mips/mm/c-r4k.c.

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

Quoted 7 lines

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

And, another confusion for PREEMPT: Even if we disable preempt, with locally effective flush_dcache_xxx, there is still possibility to reproduce such issue(Similar with the previous case):

1) Task running on Core-0 loading text section into memory.

     It was schedule out and then migrate into Core-1;

2) On Core-1, this task continue loading it and then "flush_dcache_page" to make sure the loaded text section write into main memory.
3) Task tend to the loaded text section and running it.

Similar as the previous case, the difference lies in step1 that the task was interrupted by timer interrupt. Thus it still can be switch out and then been migrate to another core. Thus in step2 and 3, this issue may still been reproduced. So, disable preempt can only lower the possibility of this issue but can't avoid it.

Quoted 27 lines · snakky.zhang at gmail.com

Quoted 9 lines

Quoted 5 lines

Yes, seems newer CPUs has no such limitation thus this function is global effective naturally. :-)

And , I find Mips's c-r4k also has this issue but it use IPI to make it. Details in arch/mips/mm/c-r4k.c.

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

Quoted 7 lines

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

And, another confusion for PREEMPT: Even if we disable preempt, with locally effective flush_dcache_xxx, there is still possibility to reproduce such issue(Similar with the previous case):

1) Task running on Core-0 loading text section into memory.

     It was schedule out and then migrate into Core-1;

2) On Core-1, this task continue loading it and then "flush_dcache_page" to make sure the loaded text section write into main memory.
3) Task tend to the loaded text section and running it.

Quoted 5 lines

Similar as the previous case, the difference lies in step1 that the task was interrupted by timer interrupt. Thus it still can be switch out and then been migrate to another core. Thus in step2 and 3, this issue may still been reproduced. So, disable preempt can only lower the possibility of this issue but can't avoid it.

It would work as long as the data copying into the text area (done by the driver and VFS layer) and the flush_dcache_page() sequence are not preemptible. A timer interrupt between data copying and flush_dcache_page() would interrupt a kernel routine which is not preemptible.

Quoted 33 lines · Catalin Marinas

Quoted 27 lines · snakky.zhang at gmail.com

Quoted 9 lines

Quoted 5 lines

Yes, seems newer CPUs has no such limitation thus this function is global effective naturally. :-)

And , I find Mips's c-r4k also has this issue but it use IPI to make it. Details in arch/mips/mm/c-r4k.c.

Rather than IPI we would better use the read-for-ownership trick like in this patch to make flush_dcache_page global (no need for write-for-ownership):

Quoted 7 lines

http://dchs.spinics.net/lists/arm-kernel/msg125075.html

(it may no longer apply, I haven't checked it for some time).

That's the first thing. Secondly you still need preemption disable so that it is not preempted between RFO and the actual cache cleaning.

And, another confusion for PREEMPT: Even if we disable preempt, with locally effective flush_dcache_xxx, there is still possibility to reproduce such issue(Similar with the previous case):

1) Task running on Core-0 loading text section into memory.

      It was schedule out and then migrate into Core-1;

2) On Core-1, this task continue loading it and then "flush_dcache_page" to make sure the loaded text section write into main memory.
3) Task tend to the loaded text section and running it.

Quoted 5 lines

Similar as the previous case, the difference lies in step1 that the task was interrupted by timer interrupt. Thus it still can be switch out and then been migrate to another core. Thus in step2 and 3, this issue may still been reproduced. So, disable preempt can only lower the possibility of this issue but can't avoid it.

Quoted 5 lines

It would work as long as the data copying into the text area (done by the driver and VFS layer) and the flush_dcache_page() sequence are not preemptible. A timer interrupt between data copying and flush_dcache_page() would interrupt a kernel routine which is not preemptible.

And that doesn't matter because on a non-preemptible kernel, timer ticks do _not_ cause the threads to be rescheduled while in kernel mode. If they did, it would be a _preemptible_ kernel.

The only things on a non-preemptible kernel which cause a schedule point are functions which may sleep, so semaphores, waiting for events, etc.

Hi, All

I'm using ARM11 MPCore on linux-2.6.34, unfortunately I have random panic/segment fault with task migration. I noticed a patch set [ARM11 MPCore: preemption/task migration cache coherency fixups] to fix such issues in here:

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-October/069851.html

It seems there is no follow ups, is there official patch to fix such issues?

thanks

Signature · 3 lines

Love each day!

--bill

Hello,

Quoted 8 lines · On May 9, 2012, at 5:11 AM, bill4carson wrote:

Hi, All

I'm using ARM11 MPCore on linux-2.6.34, unfortunately I have random panic/segment fault with task migration. I noticed a patch set [ARM11 MPCore: preemption/task migration cache coherency fixups] to fix such issues in here:

Quoted 4 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-October/069851.html

It seems there is no follow ups, is there official patch to fix such issues?

Apologies for the delayed reply. Yes, I believe those patches are still needed for ARM11 MPCore, even when not using PREEMPT. I vaguely recall at least two cases where problems occurred w/o those patches, 1) LTP stress would randomly fail, and 2) parallel module loading would invariably oops with a random fault, with results similar to what you've reported later in this thread. The most reliable test case to reproduce the failure was the parallel module loading. LTP stress test runs took a long time to trigger the errors.

I'm dusting off the patches and trying to resurrect the test cases to see if I can reproduce the errors on latest kernel.org. I'll followup with results in a day or
so. FWIW, I've got LTP stress running on latest kernel.org now but probably
should have resurrected the parallel module loading test case instead, as LTP stress may take awhile to reproduce the error, if at all.

--
Regards,
George

Quoted 8 lines

thanks

--
Love each day!

--bill

Hi, Geroge

thanks for your updates.

Quoted 40 lines · George G. Davis

Hello,

Quoted 9 lines · On May 9, 2012, at 5:11 AM, bill4carson wrote:

Hi, All

I'm using ARM11 MPCore on linux-2.6.34, unfortunately I have random panic/segment fault with task migration. I noticed a patch set [ARM11 MPCore: preemption/task migration cache coherency fixups] to fix such issues in here:

Quoted 4 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-October/069851.html

It seems there is no follow ups, is there official patch to fix such issues?

Apologies for the delayed reply. ?Yes, I believe those patches are still needed
for ARM11 MPCore, even when not using PREEMPT. ?I vaguely recall at least two cases where problems occurred w/o those patches, 1) LTP stress would randomly fail, and 2) parallel module loading would invariably oops with a random fault, with results similar to what you've reported later in this thread.
The most reliable test case to reproduce the failure was the parallel module
loading. ?LTP stress test runs took a long time to trigger the errors.

I'm dusting off the patches and trying to resurrect the test cases to see if I can
reproduce the errors on latest kernel.org. ?I'll followup with results in a day or
so. ?FWIW, ? I've got LTP stress running on latest kernel.org now but probably
should have resurrected the parallel module loading test case instead, as LTP stress may take awhile to reproduce the error, if at all.

Quoted 15 lines

--
Regards,
George

Quoted 8 lines

thanks

--
Love each day!

--bill

--
Love each day!

--bill

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel