Skip to contentSkip to conversation

Memory performance / Cache problem

DISCUSSION [email protected] 5 participants 13 messages Oct 12, 2009 – Oct 15, 2009

lore
Share & tools
lore
#0 lore raw

Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #1 Mon Oct 12 08:30:58 CEST 2009
CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: OMAP3 EVM
Memory policy: ECC disabled, Data cache writeback
OMAP3430 ES2.1
SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512

I see, you get the message about L2 Cache, which I don't have Do you know how to enable this?
Shoudn't the kernel configure all this things - not rely on bootloader? I am using the U-Boot 2008.10 (TIs PSP)
The old 22 kernel is independet from the uboot in this issue.

Thanks
Steffen

Signature · 2 lines
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
#1 lore raw

Please update to the latest HEAD on the linux-omap pm branch. In the gitweb it shows b7ecc865c5f0885fae4c4401fa78a24084f45c40

Thanks,
-Romit

Quoted 32 lines

-----Original Message-----
From: [email protected] [mailto:linux-omap- [email protected]] On Behalf Of [email protected]
Sent: Monday, October 12, 2009 2:08 PM
To: [email protected]
Subject: RE: Memory performance / Cache problem

Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-
203) ) #1 Mon Oct 12 08:30:58 CEST 2009
CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: OMAP3 EVM
Memory policy: ECC disabled, Data cache writeback
OMAP3430 ES2.1
SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512

I see, you get the message about L2 Cache, which I don't have Do you know how to enable this?
Shoudn't the kernel configure all this things - not rely on bootloader? I am using the U-Boot 2008.10 (TIs PSP)
The old 22 kernel is independet from the uboot in this issue.

Thanks
Steffen
--
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

I am sorry, the code mentioned is from the Android Public Git tree not LO PM. Sorry for the confusion.

Quoted 53 lines

-----Original Message-----
From: [email protected] [mailto:linux-omap- [email protected]] On Behalf Of Dasgupta, Romit
Sent: Monday, October 12, 2009 2:38 PM
To: [email protected]; [email protected]
Subject: RE: Memory performance / Cache problem

Please update to the latest HEAD on the linux-omap pm branch. In the gitweb it shows
b7ecc865c5f0885fae4c4401fa78a24084f45c40

Thanks,
-Romit

Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:linux-omap- [email protected]] On Behalf Of [email protected]
Sent: Monday, October 12, 2009 2:08 PM
To: [email protected]
Subject: RE: Memory performance / Cache problem

Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite

2009q1-

Quoted 15 lines

203) ) #1 Mon Oct 12 08:30:58 CEST 2009
CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: OMAP3 EVM
Memory policy: ECC disabled, Data cache writeback
OMAP3430 ES2.1
SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512

I see, you get the message about L2 Cache, which I don't have Do you know how to enable this?
Shoudn't the kernel configure all this things - not rely on bootloader? I am using the U-Boot 2008.10 (TIs PSP)
The old 22 kernel is independet from the uboot in this issue.

Quoted 9 lines

Thanks
Steffen
--
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to [email protected]
Quoted 19 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of [email protected]
Sent: Monday, October 12, 2009 2:08 PM
To: [email protected]
Subject: RE: Memory performance / Cache problem

Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #1 Mon Oct 12 08:30:58 CEST 2009
CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: OMAP3 EVM
Memory policy: ECC disabled, Data cache writeback OMAP3430 ES2.1
SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000 Built 1 zonelists in Zone order, mobility grouping on. Total
pages: 32512

Quoted 6 lines

I see, you get the message about L2 Cache, which I don't have Do you know how to enable this?
Shoudn't the kernel configure all this things - not rely on bootloader?
I am using the U-Boot 2008.10 (TIs PSP)
The old 22 kernel is independet from the uboot in this issue.

Can you upgrade to a newer u-boot? Either from the PSP release OR u-boot tree hosted at git.denx.de (atleast 2009.03)?

Also, it will be good to see the sample program you are using.

~sanjeev

Quoted 15 lines

Thanks
Steffen

Signature · 9 lines
Jetzt kostenlos herunterladen: Internet Explorer 8 und 
Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! 
http://portal.gmx.net/de/go/atbrowser
--
To unsubscribe from this list: send the line "unsubscribe 
linux-omap" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw
Quoted 8 lines

Can you upgrade to a newer u-boot? Either from the PSP release OR u-boot tree hosted at git.denx.de (atleast 2009.03)?

Also, it will be good to see the sample program you are using.

~sanjeev

There is no newer u-boot from TI available. There is a SDK 02.01.03.11
but it contains the same uboot 2008.10 with the only addition of the second generation of EVM boards with another network chip.

So I checked the uboot from git, but this doesn't support Microns NAND Flash anymore. It is just working with ONENAND.

I found a patch which shows the L2 Cache status while kernel boot and implemented it : L2 Cache seems to be already enabled - so this is not the reason.

So any other ideas?

Signature · 6 lines
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw
Quoted 4 lines

There is no newer u-boot from TI available. There is a SDK 02.01.03.11 but it contains the same uboot 2008.10 with the only addition of the second generation of EVM boards with another network chip.

Quoted 8 lines

So I checked the uboot from git, but this doesn't support Microns NAND Flash anymore. It is just working with ONENAND.

I found a patch which shows the L2 Cache status while kernel boot and implemented it : L2 Cache seems to be already enabled - so this is not the reason.

So any other ideas?

Are you confident your memory bus isn't running at 1/2 speed?

I recall there was a couple day window during wtbu kernel upgrades where memory bus speed with pm was running 1/2 speed after kernel started up. This was somewhat a side effect of constraints frame work and a regression in forward porting. It seems unlikely psp kernel would have shipped with this bug but its something to check. This would match your results.

If your memcpy() is neon based then I might be worried about l1neon-caching effects along with factors of (exlcusive-l1-l2-read-allocate cache + pld not being effective on l1 only l2).

Which memcpy test are you using? Something in lmbench or just one you wrote. Generally results are a little hard to interpret with exclusive cache behavior in 3430's r1px core. 3630's r3p2 core gives more traditional results as exclusive feature has been removed by arm.

If you have the ability using Lauterbach + per file will allow internal space dump which will show all critical parameters during test. It's a 1 minute check for someone who has done it before to ensure the few parameters needed are in line. I can send an example off line of how to do capture. I won't have time to expand on all relevant parameters.

Regards,
Richard W.

#6 lore raw

Mem clock is both times 166MHz. I don't know whether are differences in cycle access and timing, but memclock is fine.

Following Siarhei hints of initialize the buffers (around 1.2 MByte each) I get different results in 22kernel for use of
malloc alone
memcpy = 473.764, loop4 = 448.430, loop1 = 102.770, rand = 29.641
calloc alone
memcpy = 405.947, loop4 = 361.550, loop1 = 95.441, rand = 21.853
malloc+memset:
memcpy = 239.294, loop4 = 188.617, loop1 = 80.871, rand = 4.726

In 31kernel all 3 measures are about the same (unfortunatly low) level of malloc+memset in 22.

First of all: What performance can be expected? Does 22 make failures if it is so much faster? Can the later kernels get a boost in memory handling?

I used a standard memcpy (think this is glib and hence not neonbased)? To be neonbased I guess it has to be recompiled?

How can I find out that neon and cache settings are ok? Using a Omap3530 on EVM board

Unfortunatly I don't have a Lauterbach, just a Spectrum Digital which works only until Linux kernel is booting.

Best regards
Steffen

-------- Original-Nachricht --------

Quoted 46 lines

Datum: Wed, 14 Oct 2009 08:59:05 -0500
Von: "Woodruff, Richard" <[email protected]>
An: "[email protected]" <[email protected]>, "Premi, Sanjeev" <[email protected]>, "[email protected]" <[email protected]>
Betreff: RE: RE: Memory performance / Cache problem

Quoted 2 lines

There is no newer u-boot from TI available. There is a SDK 02.01.03.11 but it contains the same uboot 2008.10 with the only addition of the

second

Quoted 3 lines

generation of EVM boards with another network chip.

So I checked the uboot from git, but this doesn't support Microns NAND

Flash

Quoted 4 lines

anymore. It is just working with ONENAND.

I found a patch which shows the L2 Cache status while kernel boot and implemented it : L2 Cache seems to be already enabled - so this is not

the

Quoted 3 lines

reason.

So any other ideas?

Are you confident your memory bus isn't running at 1/2 speed?

I recall there was a couple day window during wtbu kernel upgrades where memory bus speed with pm was running 1/2 speed after kernel started up. This was somewhat a side effect of constraints frame work and a regression in forward porting. It seems unlikely psp kernel would have shipped with this bug but its something to check. This would match your results.

If your memcpy() is neon based then I might be worried about l1neon-caching effects along with factors of (exlcusive-l1-l2-read-allocate cache + pld not being effective on l1 only l2).

Which memcpy test are you using? Something in lmbench or just one you wrote. Generally results are a little hard to interpret with exclusive cache behavior in 3430's r1px core. 3630's r3p2 core gives more traditional results as exclusive feature has been removed by arm.

If you have the ability using Lauterbach + per file will allow internal space dump which will show all critical parameters during test. It's a 1 minute check for someone who has done it before to ensure the few parameters needed are in line. I can send an example off line of how to do capture. I won't have time to expand on all relevant parameters.

Regards,
Richard W.

Signature · 2 lines
Neu: GMX DSL bis 50.000 kBit/s und 200,- Euro Startguthaben!
http://portal.gmx.net/de/go/dsl02
#7 lore raw
Quoted 8 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, October 14, 2009 9:49 AM
To: Woodruff, Richard; [email protected]; Premi, Sanjeev
Subject: Re: RE: RE: Memory performance / Cache problem

Mem clock is both times 166MHz. I don't know whether are differences in cycle access and timing, but memclock is fine.

How did you physically verify this?

Quoted 11 lines

Following Siarhei hints of initialize the buffers (around 1.2 MByte each) I get different results in 22kernel for use of
malloc alone
memcpy = 473.764, loop4 = 448.430, loop1 = 102.770, rand = 29.641
calloc alone
memcpy = 405.947, loop4 = 361.550, loop1 = 95.441, rand = 21.853
malloc+memset:
memcpy = 239.294, loop4 = 188.617, loop1 = 80.871, rand = 4.726

In 31kernel all 3 measures are about the same (unfortunatly low) level of malloc+memset in 22.

Yes aligned buffers can make a difference. But probably more so for small copies. Of course you must touch the memory or mprotect() it so its faulted in, but indications are you have done this.

Quoted 2 lines

I used a standard memcpy (think this is glib and hence not neonbased)? To be neonbased I guess it has to be recompiled?

The version of glibc in use can make a difference. CodeSourcery in 2009 release added PLD's to mem operations. This can give a good benefit. It might be you have optimized library in one case and a non-optimized in another.

Regards,
Richard W.

#8 lore raw
Quoted 5 lines
Quoted 1 line

Mem clock is both times 166MHz. I don't know whether are differences in

cycle

Quoted 1 line

access and timing, but memclock is fine.

How did you physically verify this?

Oszi show 166MHz, also the kernel message about freq are in both kernels the same.

Quoted 13 lines
Quoted 1 line

Following Siarhei hints of initialize the buffers (around 1.2 MByte

each)

Quoted 3 lines

I get different results in 22kernel for use of
malloc alone
memcpy = 473.764, loop4 = 448.430, loop1 = 102.770, rand =

29.641

Quoted 2 lines

calloc alone
memcpy = 405.947, loop4 = 361.550, loop1 = 95.441, rand =

21.853

Quoted 2 lines

malloc+memset:
memcpy = 239.294, loop4 = 188.617, loop1 = 80.871, rand =

4.726

Quoted 1 line
Quoted 7 lines
Quoted 1 line

In 31kernel all 3 measures are about the same (unfortunatly low) level

of

Quoted 1 line

malloc+memset in 22.

Yes aligned buffers can make a difference. But probably more so for small copies. Of course you must touch the memory or mprotect() it so its faulted in, but indications are you have done this.

Mh, alignment (to an address) is done with malloc already. Probably you mean something different. I don't understand the difference. For me is malloc+memset=calloc. I'll send you the benchmark code, if you like.

Quoted 7 lines
Quoted 2 lines

I used a standard memcpy (think this is glib and hence not neonbased)? To be neonbased I guess it has to be recompiled?

The version of glibc in use can make a difference. CodeSourcery in 2009 release added PLD's to mem operations. This can give a good benefit. It might be you have optimized library in one case and a non-optimized in another.

In both kernels I used the same rootfs (via NFS). Indeed I used CS2009q1 and its libs, but we are talking about factor 2..6. This must be something serious.

What is your feeling? Does the 22 something strange or are the newer kernels slower that they have to be.

Would be interesting to see results on other Omap3 boards with both old an new kernels.

Best regards
Steffen

Signature · 6 lines
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#9 lore raw
Quoted 6 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, October 14, 2009 12:23 PM
To: Woodruff, Richard; Premi, Sanjeev; [email protected]
Subject: Re: RE: RE: RE: Memory performance / Cache problem

Quoted 8 lines
Quoted 3 lines

Yes aligned buffers can make a difference. But probably more so for small copies. Of course you must touch the memory or mprotect() it so its faulted in, but indications are you have done this.

Mh, alignment (to an address) is done with malloc already. Probably you mean something different. I don't understand the difference. For me is malloc+memset=calloc.
I'll send you the benchmark code, if you like.

Ok, if it compiles I could try on sdp3430 or sdp3630.

Alignment to a cache line is always best. This is 64 bytes in A8. Better yet, being 4K aligned is a good thing to reduce MMU effects.

Quoted 2 lines

In both kernels I used the same rootfs (via NFS). Indeed I used CS2009q1 and its libs, but we are talking about factor 2..6. This must be something serious.

2009 pay version has optimized thrumb2 and arm mode libraries you download separately. I don't recall if free/lite version integrated this.

Quoted 2 lines

What is your feeling? Does the 22 something strange or are the newer kernels slower that they have to be.

There are a lot of differences between 22 kernel and current ones. First things I'd check would be around MMU settings, special ARM cp15 memory control regs, then omap memory and clock settings. Also some bad device could undercut you. Always good to cat /proc/interrupts and make sure something isn't spewing.

Quoted 2 lines

Would be interesting to see results on other Omap3 boards with both old an new kernels.

I've not noticed anything on sdp on somewhat recent kernels.

Regards,
Richard W.

#10 lore raw
In reply to [email protected]
Quoted 12 lines · On Wednesday 14 October 2009 17:48:39 ext [email protected] wrote:

Mem clock is both times 166MHz. I don't know whether are differences in cycle access and timing, but memclock is fine.

Following Siarhei hints of initialize the buffers (around 1.2 MByte each) I get different results in 22kernel for use of
malloc alone
memcpy = 473.764, loop4 = 448.430, loop1 = 102.770, rand = 29.641
calloc alone
memcpy = 405.947, loop4 = 361.550, loop1 = 95.441, rand = 21.853
malloc+memset:
memcpy = 239.294, loop4 = 188.617, loop1 = 80.871, rand = 4.726

Quoted 6 lines

In 31kernel all 3 measures are about the same (unfortunatly low) level of malloc+memset in 22.

First of all: What performance can be expected? Does 22 make failures if it is so much faster? Can the later kernels get a boost in memory handling?

What you see is just a (fake) performance boost because you have a single physical page shared between all the virtual pages in the source buffer. So you get no cache misses on read operations and everything seems fast.

This is unlikely to happen on real use, and it does not reflect real memory performance. So the benchmark is inadequate.

You can get some basic information here:
http://en.wikipedia.org/wiki/Copy-on-write

Regarding the difference in behavior between .22 and recent kernels. It may be some regression in copy-on-write implementation, or just some change done on purpose. That is assuming that the userspace stuff was identical in both tests.

Signature · 2 lines
Best regards,
Siarhei Siamashka
#11 lore raw
Quoted 11 lines

From: Siarhei Siamashka [mailto:[email protected]]
Sent: Wednesday, October 14, 2009 12:37 PM
To: ext [email protected]

What you see is just a (fake) performance boost because you have a single physical page shared between all the virtual pages in the source buffer. So you get no cache misses on read operations and everything seems fast.

This is unlikely to happen on real use, and it does not reflect real memory performance. So the benchmark is inadequate.

Yep, benchmark is only useful so far. If you control factors it can be useful but it's far from 1-1, to extrapolate it to something meaningful at system level.

You can actually get even 'better' numbers if you take the DDR part geometry into mind and SDRC (sdram-controller) scheduler. Our silicon validation people report out of this world numbers for very specific test cases. These are component tests for the memory controller to make sure it behaves. If you alternate between open banks you can do really fast operations.

A good amount of that test is not practical to count on at HLOS level. It can help explain some anomalies and help in designing a fair test.

Regards,
Richard W.

#12 lore raw
In reply to Siarhei Siamashka
Quoted 39 lines · On Wednesday 14 October 2009 17:48:39 ext [email protected] wrote:
Quoted 4 lines

Mem clock is both times 166MHz. I don't know whether are differences in cycle access and timing, but memclock is fine.

Following Siarhei hints of initialize the buffers (around 1.2 MByte

each)

Quoted 3 lines

I get different results in 22kernel for use of
malloc alone
memcpy = 473.764, loop4 = 448.430, loop1 = 102.770, rand =

29.641

Quoted 2 lines

calloc alone
memcpy = 405.947, loop4 = 361.550, loop1 = 95.441, rand =

21.853

Quoted 2 lines

malloc+memset:
memcpy = 239.294, loop4 = 188.617, loop1 = 80.871, rand =

4.726

Quoted 2 lines

In 31kernel all 3 measures are about the same (unfortunatly low) level

of

Quoted 5 lines

malloc+memset in 22.

First of all: What performance can be expected? Does 22 make failures if it is so much faster? Can the later kernels get a boost in memory handling?

What you see is just a (fake) performance boost because you have a single physical page shared between all the virtual pages in the source buffer. So
you get no cache misses on read operations and everything seems fast.

This is unlikely to happen on real use, and it does not reflect real memory
performance. So the benchmark is inadequate.

You can get some basic information here:
http://en.wikipedia.org/wiki/Copy-on-write

Quoted 7 lines

Regarding the difference in behavior between .22 and recent kernels. It may be
some regression in copy-on-write implementation, or just some change done on
purpose. That is assuming that the userspace stuff was identical in both tests.

Ok, understand the difference if the memory is uninitialised. But why there is the difference in "malloc + memset" and "calloc"? In both cases the memory will be cleared.

Signature · 2 lines
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser