PL310 and QoS logic
Catalin, All,
I work on the SPEAr1300 SoC, which is a device based on Cortex A9 MP (r1p1) and
is using a PL310 L2 cache controller.
We are using the Linux version 2.6.35 and we have also ported some patches coming
from you.
Even with this latest version of kernel we are not able to make the L2 cache working.
It looks like that we are having a problem particularly when using it in combination
with MP.
Just to add a very important information, the situation looks to clearly stabilize
when we set the bit9 of PL310 Auxiliary Control Register to '1'. THis bit is reserved,
and ARM support told us to avoid setting it.
In fact, this bit when set seems to disable a QoS logic whose purpose is to avoid issues
for which one core is not able to release a semaphore because the other core is
continuously trying to take it. So, this QoS logic has been introduce inside the
PL310 to set a kind of priority (a QoS).
Strangely enough this issue looks exactly what we are seeing. It explains all of our observations.
So, it seems that for us writing this bit9 to '1' has the effect of enabling (and not disabling) the QoS logic.
Have you ever heard an issue like that?
Thx,
Armando
Hello Russell, Catalin,
Quoted 30 lines · Armando VISCONTI
Catalin, All,
I work on the SPEAr1300 SoC, which is a device based on Cortex A9 MP (r1p1) and
is using a PL310 L2 cache controller.We are using the Linux version 2.6.35 and we have also ported some patches coming
from you.Even with this latest version of kernel we are not able to make the L2 cache working.
It looks like that we are having a problem particularly when using it in combination
with MP.Just to add a very important information, the situation looks to clearly stabilize
when ?we set the bit9 of PL310 Auxiliary Control Register to '1'. THis bit is reserved,
and ARM support told us to avoid setting it.In fact, this bit when set seems to disable a ?QoS logic whose purpose is to avoid issues
for which one core is not able to release a semaphore because the other core is
continuously trying to take it. So, this QoS logic has been introduce inside the
PL310 to set a kind of priority (a QoS).
Quoted 7 lines
Strangely enough this issue looks exactly what we are seeing. It explains all of our observations.
So, it seems that for us writing this bit9 to '1' has the effect of enabling (and not disabling) the QoS logic.
Have you ever heard an issue like that?
Any help on this.
regards
Shiraz
Hello Catalin,
Quoted 37 lines · shiraz hashim
Hello Russell, Catalin,
Quoted 33 lines · Armando VISCONTI
Catalin, All,
I work on the SPEAr1300 SoC, which is a device based on Cortex A9 MP (r1p1) and
is using a PL310 L2 cache controller.We are using the Linux version 2.6.35 and we have also ported some patches coming
from you.Even with this latest version of kernel we are not able to make the L2 cache working.
It looks like that we are having a problem particularly when using it in combination
with MP.Just to add a very important information, the situation looks to clearly stabilize
when ?we set the bit9 of PL310 Auxiliary Control Register to '1'. THis bit is reserved,
and ARM support told us to avoid setting it.In fact, this bit when set seems to disable a ?QoS logic whose purpose is to avoid issues
for which one core is not able to release a semaphore because the other core is
continuously trying to take it. So, this QoS logic has been introduce inside the
PL310 to set a kind of priority (a QoS).Strangely enough this issue looks exactly what we are seeing. It explains all of our observations.
Quoted 6 lines
Quoted 4 lines
So, it seems that for us writing this bit9 to '1' has the effect of enabling (and not disabling) the QoS logic.
Have you ever heard an issue like that?
Any help on this.
We were going through ARM PL310 errata list and we came across bug #738415, which says
"738415: A cacheable read with address bits [20:5] equal to 0x0000
can be stalled by non-cacheable read traffic targeting other address region 2"
We believe this could be one of the potential problems which we are facing, and the
errata says the implication could be that there could be a deadlock if there are dependent
reads.
Now, how do we ensure that this is the root cause. What is the best way in which I
can prevent kernel from using cache-lines at each 2MB boundary. Have you come across this bug? Any information would be really helpful.
thanks
regards
Shiraz
Quoted 12 lines · shiraz hashim
We were going through ARM PL310 errata list and we came across bug #738415, which says
"738415: A cacheable read with address bits [20:5] equal to0x0000
can be stalled by non-cacheable read traffic targeting other address region 2"We believe this could be one of the potential problems which we are facing, and the
errata says the implication could be that there could be a deadlock if there are dependent
reads.
Quoted 4 lines
Now, how do we ensure that this is the root cause. What is the best way in which I
can prevent kernel from using cache-lines at each 2MB boundary. Have you come across this bug? Any information would be really helpful.
I think more information or conditions for this to happen can be given by support at arm.com. I don't know the hardware details around this.
In general you don't have any cacheable/non-cacheable dependent reads in Linux to be able to trigger the deadlock. That's a situation where one CPU is polling some non-cacheable memory location and a different CPU cannot continue. The only situation I'm aware of is during secondary CPU booting or if you boot Linux with CPU with a maxcpus less than the total number of CPUs (so some of them keep polling the pen_release even at run-time). But that's easily solvable with a WFE in the pen polling loop (if your platform has such booting protocol).
On the RealView platforms, the secondary CPUs are waiting in a WFI in the boot monitor code and waken up one by one via an IPI, so they don't usually wait in the pen polling loop.
If you want to prevent such memory from being used, you can hack the memory initialisation code in Linux to reserve a 4K physical page for every 2MB.
Catalin
Hello Catalin,
Quoted 13 lines · Catalin Marinas
Quoted 12 lines · shiraz hashim
We were going through ARM PL310 errata list and we came across bug #738415, which says
"738415: ? A cacheable read with address bits [20:5] equal to0x0000can be stalled by non-cacheable read traffic targeting other address region 2"We believe this could be one of the potential problems which we are facing, and the
errata says the implication could be that there could be a deadlock if there are dependent
reads.
Quoted 7 lines
Quoted 4 lines
Now, how do we ensure that this is the root cause. What is the best way in which I
can prevent kernel from using cache-lines at each 2MB boundary. Have you come across this bug? Any information would be really helpful.I think more information or conditions for this to happen can be given by support at arm.com. I don't know the hardware details around this.
OK.
Quoted 9 lines
In general you don't have any cacheable/non-cacheable dependent reads in Linux to be able to trigger the deadlock. That's a situation where one CPU is polling some non-cacheable memory location and a different CPU cannot continue. The only situation I'm aware of is during secondary CPU booting or if you boot Linux with CPU with a maxcpus less than the total number of CPUs (so some of them keep polling the
pen_releaseeven at run-time). But that's easily solvable with a WFE in the pen polling loop (if your platform has such booting protocol).
Quoted 7 lines
On the RealView platforms, the secondary CPUs are waiting in a WFI in the boot monitor code and waken up one by one via an IPI, so they don't usually wait in the pen polling loop.
If you want to prevent such memory from being used, you can hack the memory initialisation code in Linux to reserve a 4K physical page for every 2MB.
I am not very sure how to do it. Can I use reserve_bootmem_node from bootmem_init.
Signature · 2 lines
regards Shiraz Hashim