flash read performance

Hi list,

I am currently trying to improve the flash read performance of my platform. It's a gumstix verdex board with a pxa270 running at 400MHz. My flash is a 16MB NOR Intel StrataFlash P30 (128P30T) and it's operating in the _normal_ asynchronous mode. In my opinion the read performance is very poor, only around 1.2 to 1.4 MB/s depending on the blocksize. I think it should be possible to get much higher transfer rates.

In Linux, I ran my tests with dd like this (copy 10MB):
time dd if=/dev/mtd5 of=/dev/null bs=16k count=640 640+0 records in
640+0 records out

real    0m 7.17s
user    0m 0.00s
sys     0m 7.17s

Running top in another console brings up, that the CPU load is very high during copy. I am not sure if the system is doing some sort of busy waiting or something like that? However, it should be possible to do a copy without having such a high load.

Quoted 8 lines · Andre Puschmann

Hi list,

I am currently trying to improve the flash read performance of my platform. It's a gumstix verdex board with a pxa270 running at 400MHz. My flash is a 16MB NOR Intel StrataFlash P30 (128P30T) and it's operating in the _normal_ asynchronous mode. In my opinion the read performance is very poor, only around 1.2 to 1.4 MB/s depending on the blocksize. I think it should be possible to get much higher transfer rates.

Why do you think that?

Quoted 1 line

In my opinion the read performance is very poor, only around 1.2 to 1.4 MB/s depending on the blocksize. I think it should be possible to get much higher transfer rates.

Why do you think that?

Why do you think that? The chip drivers don't do DMA, so all I/O goes through the CPU.

Quoted 4 lines · Josh Boyer

Quoted 2 lines

In my opinion the read performance is very poor, only around 1.2 to 1.4 MB/s depending on the blocksize. I think it should be possible to get much higher transfer rates.

Why do you think that?

I guess there is something wrong with the timing parameters and/or the way the CPU core speaks to the flash controller, which results in long wait-states. But at least for my understanding, these transfer rates have nothign to do with _high speed NOR flashes_ :-)

Quoted 3 lines

Why do you think that? The chip drivers don't do DMA, so all I/O goes through the CPU.

Yes, DMA is not used. However, the CPU should be strong enough to do this transfer faster. On the other hand, my understanding is, DMA brings no speed improvements in all cases. It boosts memory transfers without adding an extra overhead to the CPU. But in this case, copy data is the only task.

Regards,
Andre

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

I was faced with the same wondering in the past : bootloader NOR access was really much faster that Linux one.

Yes, no DMA was used (but the same on bootloader, and anyway that doesn't impact the data rate, only the CPU load), but even worse, Linux code was using memcpy_fromio which a basic byte by byte loop copy in the default ARM implementation.

Hi,

Quoted 2 lines

I was faced with the same wondering in the past : bootloader NOR access was really much faster that Linux one.

About how much faster? It really depends on the access method. I am
using u-boot and if I use the basic cp.b routine its about the same
_slow_ speed. I tried to use the asm-optimised memcpy routine that the kernel has. This is much faster, around 5MB/s.

Quoted 2 lines

Yes, no DMA was used (but the same on bootloader, and anyway that doesn't impact the data rate, only the CPU load), but even worse, Linux code was using memcpy_fromio which a basic byte by byte loop copy in the default ARM implementation.

Yes, memcpy_fromio is quite slow. But using normal memcpy is not suggested, only use writel()/readl() and memcpy_[from|to]io().

I am not sure about the right _fast_ way to to such copies.

Regards,
Andre

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

I was using redboot, configured to use the optimized memcpy (yes, it gives the choice at configuration time)
on kernel side, I just hack memcpy_fromio to add a "weak" attribute, and rewrite it to directly use the linux optimized memcpy (shame on me for this "not suggested" methode, but speed was my goal)

after that, performances are equal between bootloader and linux, and really near the one reached by a DMA access, which is also the performances we can calculate from FLASH time access configuration.

arnaud

Quoted 17 lines · Andre Puschmann

Hi,

Quoted 2 lines

I was faced with the same wondering in the past : bootloader NOR access was really much faster that Linux one.

About how much faster? It really depends on the access method. I am
using u-boot and if I use the basic cp.b routine its about the same
_slow_ speed. I tried to use the asm-optimised memcpy routine that the kernel has. This is much faster, around 5MB/s.

Quoted 2 lines

Yes, no DMA was used (but the same on bootloader, and anyway that doesn't impact the data rate, only the CPU load), but even worse, Linux code was using memcpy_fromio which a basic byte by byte loop copy in the default ARM implementation.

Yes, memcpy_fromio is quite slow. But using normal memcpy is not suggested, only use writel()/readl() and memcpy_[from|to]io().

I am not sure about the right _fast_ way to to such copies.

Quoted 9 lines

Regards,
Andre

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Unfortunately, it uses a legacy driver which only does a ioremap() but no ioremap_nocache().

Hi,

I spent some more time on this issue and investigated some mtd-maps drivers.
The kernel I am using is a 2.6.21 that comes out of the gumstix svn-repo. Unfortunately, it uses a legacy driver which only does a ioremap() but no ioremap_nocache(). Patching the driver with this additional call boosts up transfers up to around 5.5MB/s, which is a fairly improvement.
I will send a patch to the gumstix list. Users of newer kernel might not need this, as they use a newer driver (pxa2xx-flash.c) anyway.

I don't know much about this area, but will _writing_ to the flash work reliably if ioremap_cached() is used?

-- Jamie

Quoted 5 lines · Jamie Lokier

I don't know much about this area, but will _writing_ to the flash work reliably if ioremap_cached() is used?

-- Jamie

Good point. I only was into reading and so I totally forgot writing ;-) I gave it a try, although it was terribly slow (only a few kb/s), it worked. I just did a cp uImage /dev/mtd3. On the other hand, I never tried writing with the old driver. So I don't know if this is faster.

I also did some more testing with my improved flash-timing parameters, which yields to read speeds of up to 18-19MB/s, which is really fast compared
to 1,3MB/s at the beginning :-)

Anyway, with these results, booting the complete system in nearly (or even less than) 2s should be possible.

Quoted 4 lines · Andre Puschman

Quoted 3 lines · Jamie Lokier

I don't know much about this area, but will _writing_ to the flash work reliably if ioremap_cached() is used?

Quoted 4 lines

Good point. I only was into reading and so I totally forgot writing ;-) I gave it a try, although it was terribly slow (only a few kb/s), it worked. I just did a cp uImage /dev/mtd3. On the other hand, I never tried writing with the old driver. So I don't know if this is faster.

I've found that writes do not work with caching enabled. When the CPU writes to the flash and then reads it back, it gets returned what it wrote.

That's not what is supposed to happen. For example, to program flash word 'i' to the value 'val' using the Spansion/AMD method, you do this:

flash[0] = 0xf0f0;
flash[0x555] = 0xaaaa;
flash[0x2aa] = 0x5555;
flash[0x555] = 0xa0a0;
flash[i] = val;
while(flash[i] != val); /* wait for it to finish */

After this flash[0] should be whatever data was there before, not 0xf0f0. Same with flash[0x555] and the rest. Only flash[i] should be modified. But if flash is cached, the cpu will use the cached values and think flash[0] is 0xf0f0 until the cache gets flushed.

Quoted 4 lines

I also did some more testing with my improved flash-timing parameters, which yields to read speeds of up to 18-19MB/s, which is really fast compared
to 1,3MB/s at the beginning :-)

My results, from a mpc8572 (powerpc) with a spansion s96gl064n flash chip on a 100 MHz bus.

Mapping Speed (MB/sec) (MB = 1048576 bytes)
un-cached and guarded 12.30
cached and gaurded 14.24
cached and un-guarded 14.31
un-cached and un-guarded 14.66

I measured by reading flash linearly from beginning to end 32-bits at a time. Since the flash is bigger than the cache, ever read should have come from the flash. If I just read the same 1k over and over that would obviously be much faster if it could come from the cache.

The biggest bootup delay I have now is waiting for the ethernet phy to get online, which takes almost 3 seconds.

Based on Andre's observation, I will soon try enabling cache for my NOR, and see if it makes a difference to cold-cache read performance. I don't expect it, but it's worth a try.

You might also find the write operation to be unreliable, if the caching mode is write-back rather than write-through.

Really, you should use an uncached mapping to write commands to the flash, flush the cached mapping (for reads) when commands are written, and prevent any access during the writes (this is in MTD normally). You could optimise by flushing only the cached read regions which are affected by write and erase commands.

Quoted 6 lines

Mapping Speed (MB/sec) (MB = 1048576 bytes)
un-cached and guarded 12.30
cached and gaurded 14.24
cached and un-guarded 14.31
un-cached and un-guarded 14.66

Quoted 4 lines

I measured by reading flash linearly from beginning to end 32-bits at a time. Since the flash is bigger than the cache, ever read should have come from the flash. If I just read the same 1k over and over that would obviously be much faster if it could come from the cache.

That's nice to see that cache helps cold-read performance too, not just cached reads. Thanks :-)

Quoted 2 lines

The biggest bootup delay I have now is waiting for the ethernet phy to get online, which takes almost 3 seconds.

Do you need to delay everything else for that, or can you parallelise?

Quoted 3 lines · Jamie Lokier

Based on Andre's observation, I will soon try enabling cache for my NOR, and see if it makes a difference to cold-cache read performance. I don't expect it, but it's worth a try.

It possible it could help by efficiently doing the reads back-to-back with no wasted cycles between them. I think it's necessary if you want to benefit from page mode, but that's something I haven't tried yet.

Quoted 2 lines

You might also find the write operation to be unreliable, if the caching mode is write-back rather than write-through.

Proper use of "sync" instructions, or whatever the arch uses to insure that writel() is strictly ordered should fix that.

I've found that writes do not work with caching enabled. When the CPU writes to the flash and then reads it back, it gets returned what it wrote. That's not what is supposed to happen.

I measured by reading flash linearly from beginning to end 32-bits at a time. Since the flash is bigger than the cache, ever read should have come from the flash. If I just read the same 1k over and over that would obviously be much
faster if it could come from the cache.

I'm just using the GPCM mode of the Freescale eLBC, which means I have to use the same timings both for writes and reads. There are parts of the timing I could make faster for reads, but then they would be too short for writes, and vice versa. It also means I can't use the page burst mode, which would speed up reads significantly.

The biggest bootup delay I have now is waiting for the ethernet phy to get online, which takes almost 3 seconds.

Quoted 3 lines · Trent Piepho

I've found that writes do not work with caching enabled. When the CPU writes to the flash and then reads it back, it gets returned what it wrote. That's not what is supposed to happen.

Sure, writes should be uncached and unbuffered. But I thought the mtd layer handles this correctly as there are two different ioremap's in the driver:

map.virt = ioremap(..);
map.cached = ioremap_cached(..);
map.inval_cache = inval_cache_fct();

So, calling inval_cache_fct() just before any write operation and then using the uncached mapping should do the trick, no? On the other hand, I am not sure if the mtd-layer really behaves like that. Can somebody confirm this?

From looking at the code I'd say you're right.

I am not familiar with GPCM and eLBC, but it sounds about the same here. Same timings for writes and reads. But I use burst mode (4 words), but this only applies to reads.

Quoted 2 lines

The biggest bootup delay I have now is waiting for the ethernet phy to get online, which takes almost 3 seconds.

Same here, ethernet phy takes sooo long. Here is what I do: I am using a parallel init. On script is just for loading the ethernet-module which is done next to the other scripts. So 2sec is cheated, cause ethernet isn't really available at this point of time.

Quoted 3 lines

map.virt = ioremap(..);
map.cached = ioremap_cached(..);
map.inval_cache = inval_cache_fct();

It depends on what mapping driver you're using. It looks like only the pxa2xx driver uses map.cached. The physmap or of_physmap drivers that I'm using don't use it.

I've switched from GPCM to UPM, which lets me use different timings for read and write as well as use burst mode.

In non-cached and guarded mode, I now get 13.61 vs 12.30 MB/s. That's just from slightly better timings because I could make them different for read vs write. The big difference is cached and non-guarded reads, which went to 44.79 MB/s from 14.24 MB/s. That boost is from using burst mode.

It might be the case that the ethernet module resets the PHY when it

and/or when the ethernet device is opened. That was a problem I was having. The PHY would almost be done when the dhcp client would run and open eth0, which would start the phy all over again.

Quoted 7 lines · Trent Piepho

Quoted 3 lines

map.virt = ioremap(..);
map.cached = ioremap_cached(..);
map.inval_cache = inval_cache_fct();

It depends on what mapping driver you're using. It looks like only the pxa2xx driver uses map.cached. The physmap or of_physmap drivers that I'm using don't use it.

The flashmap-drive I am using is custom one made for the gumstix-board. However, it is almost identically to pxa2xx-driver from newer kernels.

Quoted 4 lines

In non-cached and guarded mode, I now get 13.61 vs 12.30 MB/s. That's just from slightly better timings because I could make them different for read vs write. The big difference is cached and non-guarded reads, which went to 44.79 MB/s from 14.24 MB/s. That boost is from using burst mode.

Whop, this is great news. Btw. do you drive your flash in asynchronous or in synchronous mode? Do you have an extra flash configuration register that you need to modify in order to use the burst mode? My intel NOR flash has an extra read configuration register (RCR). However, for some reason I am not able to read/modify/read this register successfully.