Skip to contentSkip to conversation

Benchmarking: POP flash vs. MMC?

DISCUSSION [email protected] 7 participants 15 messages Apr 3, 2009 – Apr 7, 2009

lore
Share & tools
lore
#0 lore raw

Has anybody done any benchmarking on POP flash read/write speeds (using JFFS2) vs. MMC/SDHC card access (using ext[234])? Or for that matter, has anybody done any playing around with using the execute in place function of JFFS2 to see if there is any over-all speed advantage (due to reduced pressure on the RAM) vs. loading into RAM?

#1 lore raw
Quoted 6 lines · On Fri, Apr 3, 2009 at 2:24 PM, <[email protected]> wrote:

Has anybody done any benchmarking on POP flash read/write speeds (using JFFS2) vs. MMC/SDHC card access (using ext[234])? Or for that matter, has anybody done any playing around with using the execute in place function of JFFS2 to see if there is any over-all speed advantage (due to reduced pressure on the RAM) vs. loading into RAM?

UBIFS on POP NAND
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
beagleboard    300M  2333  86 19899  45 10420  59  2289  95 17719  93  2060  93
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5310  94  7931  96  4439  87  4734  85 +++++ +++  3082  95

beagleboard,300M,2333,86,19899,45,10420,59,2289,95,17719,93,2060.1,93,16,5310,94,7931,96,4439,87,4734,85,+++++,+++,3082,95

ext3 (noatime,async,data=ordered) on MMC (4GB class 6)
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
beagleboard    300M  2026  81  4553  11  4281   6  2502  92 16079  14  1672  23
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  7135  95 +++++ +++ 12173  95  5040  67 +++++ +++ 11766  96

beagleboard,300M,2026,81,4553,11,4281,6,2502,92,16079,14,1672.3,23,16,7135,95,+++++,+++,12173,95,5040,67,+++++,+++,11766,96

ext3 (noatime,async,data=writeback) on MMC (4GB class 6)
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
beagleboard    300M  2343  85 19370  40 10937  61  2312  95 19176  94  3122  93
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5194  93  7863  95  4511  87  4970  89 +++++ +++  3076  95

beagleboard,300M,2343,85,19370,40,10937,61,2312,95,19176,94,3122.3,93,16,5194,93,7863,95,4511,87,4970,89,+++++,+++,3076,95

#2 lore raw

Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.

#3 lore raw
Quoted 1 line · David Hagood

Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.

The POP flash may not have a DMA ability. Also, the POP flash contents are compressed and ECC corrected, so its a lot of extra work for the CPU.

#4 lore raw
Quoted 6 lines · Russ Dill
Quoted 3 lines · David Hagood

Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.

The POP flash may not have a DMA ability.

ISTR it does, but it's not used. There are other speedups possible; the NAND driver in the OMAP tree is pretty simplistic. It doesn't use the "Prefetch and Write-Posting Engine" (we don't know if that really helps though); or the more aggressive ECC support (4-bit and 8-bit modes both work).

Of course using DMA in the MTD stack can be a bit problematic in general. It assumes that DMA is not used ... to the extent of not guaranteeing to provide DMA-mappable I/O buffers.

That said, it's hard to achieve speedups through DMA for such small blocks of memory ... an issue which is not unique to NAND.

Quoted 3 lines

Also, the POP flash contents
are compressed and ECC corrected, so its a lot of extra work for the CPU.

ECC is done in hardware for the native NAND stack on OMAP; but you're right about compression.

Another point is that "managed NAND" (like MMC and eMMC, and various SSD flavors) is there to offload some complicated and error-prone tasks like wear-leveling.

#5 lore raw
In reply to David Hagood
Quoted 3 lines · David Hagood

Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.

1. As Russ and David said, OneNAND driver does not really use DMA, because the I/O is done in 2K chunks, and this is just too small piece of data for DMA.

2. UBIFS also compresses data on-the-flight. You may try disabling it and see what changes, but probably not too much, because of the way the driver writes (no DMA). Try mounting with 'compre=none' option, see here:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_mountopts

BTW, some compression testing results may be found here:

http://www.linux-mtd.infradead.org/misc/misc.html#L_ubifs_compr

although they are not 100% relevant for this case.

3. UBIFS provides you greater data reliability. E.g., it CRCs all data (see here
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_checksumming)
OneNAND was very reliable last time we tested it, and we disable _some_ CRC checking for it. Try to use the 'no_chk_data_crc' and get better read speed.

4. UBIFS has 'bulk read' feature which works well on OneNAND, (see here: http://www.linux-mtd.infradead.org/doc/ubifs.html#L_readahead)
Try to enable it as well. You should end up with faster
read speed.

5. Last but not least, UBIFS+OneNAND provides just another level of reliability, comparing to SD. In general, SDs are not very good for storing system libraries, etc. I tried to summarize this at some point here:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_raw_vs_ftl

HTH.

Signature · 6 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#6 lore raw
Quoted 9 lines · Artem Bityutskiy
Quoted 4 lines · David Hagood

Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.

1. As Russ and David said, OneNAND driver does not really use DMA, because the I/O is done in 2K chunks, and this is just too small piece of data for DMA.

All very relevant, but just to avoid confusion; the tests were performed on a Beagleboard with 256MB of Micron NAND.

http://www.micron.com/products/partdetail?part=MT29C2G24MAKLAJG-6 IT

Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.

Quoted 30 lines

2. UBIFS also compresses data on-the-flight. You may try disabling it and see what changes, but probably not too much, because of the way the driver writes (no DMA). Try mounting with 'compre=none' option, see here:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_mountopts

BTW, some compression testing results may be found here:

http://www.linux-mtd.infradead.org/misc/misc.html#L_ubifs_compr

although they are not 100% relevant for this case.

3. UBIFS provides you greater data reliability. E.g., it CRCs all data (see here
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_checksumming)
OneNAND was very reliable last time we tested it, and we disable _some_ CRC checking for it. Try to use the 'no_chk_data_crc' and get better read speed.

4. UBIFS has 'bulk read' feature which works well on OneNAND, (see here: http://www.linux-mtd.infradead.org/doc/ubifs.html#L_readahead)
Try to enable it as well. You should end up with faster
read speed.

5. Last but not least, UBIFS+OneNAND provides just another level of reliability, comparing to SD. In general, SDs are not very good for storing system libraries, etc. I tried to summarize this at some point here:

Quoted 8 lines

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_raw_vs_ftl

HTH.

--
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 11 lines · Russ Dill
Quoted 7 lines · Artem Bityutskiy
Quoted 3 lines · David Hagood

Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.

1. As Russ and David said, OneNAND driver does not really use DMA, because the I/O is done in 2K chunks, and this is just too small piece of data for DMA.

All very relevant, but just to avoid confusion; the tests were performed on a Beagleboard with 256MB of Micron NAND.

Ok, sorry. The CRC part should be relevant. The bulk_read part may also be relevant if your HW is able to read multiple NAND pages faster than reading them one-by-one. The compression links should be helpful for compressor selection.

Signature · 6 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#8 lore raw
In reply to Russ Dill
Quoted 2 lines · Russ Dill

Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.

We had asked for some benchmark data -- anything! -- to get a handle on that, and the prefetch/etc engine; nothing forthcoming, so far.

#9 lore raw

Hi David

Quoted 7 lines · David Brownell
Quoted 2 lines · Russ Dill

Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.

We had asked for some benchmark data -- anything! -- to get a handle on that, and the prefetch/etc engine; nothing forthcoming, so far.

We'd also have to make sure that the comparison is between the linux-omap kernel and the OMAPZoo kernel, rather than o-z PIO vs. o-z DMA. The OMAPZoom kernel doesn't post any device register writes. That should cause any driver using PIO to drag, compared to the l-o kernel.

- Paul

#10 lore raw
Quoted 8 lines · Paul Walmsley

Hi David

Quoted 4 lines · David Brownell
Quoted 2 lines · Russ Dill

Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.

Quoted 8 lines
Quoted 3 lines

We had asked for some benchmark data -- anything! -- to get a handle on that, and the prefetch/etc engine; nothing forthcoming, so far.

We'd also have to make sure that the comparison is between the linux-omap kernel and the OMAPZoo kernel, rather than o-z PIO vs. o-z DMA. The OMAPZoom kernel doesn't post any device register writes. That should cause any driver using PIO to drag, compared to the l-o kernel.

I'd think the first thing to check is two linux-omap PIO flavors:
with vs without the prefetch/postwrite engine. Benchmarking DMA would be a separate issue. Ditto comparing kernels with/without write posting.

I've benchmarked small DMA transfers and it's rather hard to get them faster than io{read,write}32_rep(). Overhead of DMA mapping and just DMA setup/teardown/complete is annoyingly high. While bus overheads aren't large.

Puzzle: get a dma_copypage() to work faster than copy_page(). Or a dma_clear_page() faster than clear_page(). Not easy...

- Dave

#11 lore raw
Quoted 4 lines · David Brownell

I'd think the first thing to check is two linux-omap PIO flavors:
with vs without the prefetch/postwrite engine. Benchmarking DMA would be a separate issue. Ditto comparing kernels with/without write posting.

Makes sense to me. FIFO-less GPMC PIO should be full of fail.

Quoted 2 lines

Puzzle: get a dma_copypage() to work faster than copy_page(). Or a dma_clear_page() faster than clear_page(). Not easy...

Doing it via the DMA engine may save power, since MPU can sleep.

- Paul

#12 lore raw
Quoted 4 lines · Paul Walmsley
Quoted 2 lines

Puzzle:  get a dma_copypage() to work faster than copy_page(). Or a dma_clear_page() faster than clear_page().  Not easy...

Doing it via the DMA engine may save power, since MPU can sleep.

But the CPU overhead of calling the DMA engine can exceed that of the memcpy()/memset() ... ;)

Another concern is cache impact. In some cases, having the dirty data in dcache is a big win. With DMA, the cache will have been purged.

It'd be nice to see DMA versions of this stuff winning; all I'm saying is that such wins are hard to achieve.

- Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw
Quoted 9 lines · David Brownell
Quoted 4 lines · Paul Walmsley
Quoted 2 lines

Puzzle:  get a dma_copypage() to work faster than copy_page(). Or a dma_clear_page() faster than clear_page().  Not easy...

Doing it via the DMA engine may save power, since MPU can sleep.

But the CPU overhead of calling the DMA engine can exceed that of the memcpy()/memset() ... ;)

Quoted 6 lines

Another concern is cache impact. In some cases, having the dirty data in dcache is a big win. With DMA, the cache will have been purged.

It'd be nice to see DMA versions of this stuff winning; all I'm saying is that such wins are hard to achieve.

If the DMA transfer can loop over the small FIFO by and transfer larger chunks of data, the setup overhead is small. Currently at least tusb and ASoC loop the DMA over the FIFO.

This way you can let the hardware trigger huge reads or writes from the small FIFO. To loop over the FIFO, set dst_fi or src_fi to a _negative_ number.

Maybe onenand could use this too? If you can tell onenand to read or write multiple FIFOs, and there is a hardware DMA request line, the DMA request line can tell onenand to reload the FIFO.

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw
In reply to Paul Walmsley
Quoted 4 lines

We'd also have to make sure that the comparison is between the linux-omap kernel and the OMAPZoo kernel, rather than o-z PIO vs. o-z DMA. The OMAPZoom kernel doesn't post any device register writes. That should cause any driver using PIO to drag, compared to the l-o kernel.

There effetely are a few levels of posting. The patch stops _ARM_ PIO to peripheral control registers from hanging out in the interconnect. Other levels still are there. The big winner of interconnect from ARM side is DDR operations which is still gets posted at all levels. The size of buffering in interconnect is like a couple cache lines so it will fill up pretty fast. DMA and other initiators are not impacted as they don't use arm-mmu attributes.

Really, if you think about it "big" block writes may not be impacted either as you will back up at the slower device's speed. Cache is a familiar example. If you write out 100M quickly you will get to a point where you are bottlenecked on main memory speed very quickly (every write is a miss or cast out at some point). The fact you have a cache in the way doesn't matter.

It can make some difference if you're intermixing some small PIOs with other work. In general benchmarks I've yet to see the hit on system against all the bigger noises. Probably I can construct a case where ~10% is lost.

I recall some tests which have dma + prefetch working on nand. I'll see if I can dig them up.

I've was at several meetings a year back where different memory vendors come in and showed with some tweak they can get 2x l-o on flash. Then if you also take their device optimized file system you will get like 5x. Hopefully at least the in tree 2x is gone now that it's a year later.

Regards,
Richard W.