Benchmarking: POP flash vs. MMC?
Has anybody done any benchmarking on POP flash read/write speeds (using JFFS2) vs. MMC/SDHC card access (using ext[234])? Or for that matter, has anybody done any playing around with using the execute in place function of JFFS2 to see if there is any over-all speed advantage (due to reduced pressure on the RAM) vs. loading into RAM?
Quoted 6 lines · On Fri, Apr 3, 2009 at 2:24 PM, <[email protected]> wrote:
Has anybody done any benchmarking on POP flash read/write speeds (using JFFS2) vs. MMC/SDHC card access (using ext[234])? Or for that matter, has anybody done any playing around with using the execute in place function of JFFS2 to see if there is any over-all speed advantage (due to reduced pressure on the RAM) vs. loading into RAM?
UBIFS on POP NAND
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
beagleboard 300M 2333 86 19899 45 10420 59 2289 95 17719 93 2060 93
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 5310 94 7931 96 4439 87 4734 85 +++++ +++ 3082 95beagleboard,300M,2333,86,19899,45,10420,59,2289,95,17719,93,2060.1,93,16,5310,94,7931,96,4439,87,4734,85,+++++,+++,3082,95
ext3 (noatime,async,data=ordered) on MMC (4GB class 6)
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
beagleboard 300M 2026 81 4553 11 4281 6 2502 92 16079 14 1672 23
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 7135 95 +++++ +++ 12173 95 5040 67 +++++ +++ 11766 96beagleboard,300M,2026,81,4553,11,4281,6,2502,92,16079,14,1672.3,23,16,7135,95,+++++,+++,12173,95,5040,67,+++++,+++,11766,96
ext3 (noatime,async,data=writeback) on MMC (4GB class 6)
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
beagleboard 300M 2343 85 19370 40 10937 61 2312 95 19176 94 3122 93
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 5194 93 7863 95 4511 87 4970 89 +++++ +++ 3076 95beagleboard,300M,2343,85,19370,40,10937,61,2312,95,19176,94,3122.3,93,16,5194,93,7863,95,4511,87,4970,89,+++++,+++,3076,95
Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.
Quoted 1 line · David Hagood
Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.
The POP flash may not have a DMA ability. Also, the POP flash contents are compressed and ECC corrected, so its a lot of extra work for the CPU.
Quoted 6 lines · Russ Dill
Quoted 3 lines · David Hagood
Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.
The POP flash may not have a DMA ability.
ISTR it does, but it's not used. There are other speedups possible; the NAND driver in the OMAP tree is pretty simplistic. It doesn't use the "Prefetch and Write-Posting Engine" (we don't know if that really helps though); or the more aggressive ECC support (4-bit and 8-bit modes both work).
Of course using DMA in the MTD stack can be a bit problematic in general. It assumes that DMA is not used ... to the extent of not guaranteeing to provide DMA-mappable I/O buffers.
That said, it's hard to achieve speedups through DMA for such small blocks of memory ... an issue which is not unique to NAND.
Quoted 3 lines
Also, the POP flash contents
are compressed and ECC corrected, so its a lot of extra work for the CPU.
ECC is done in hardware for the native NAND stack on OMAP; but you're right about compression.
Another point is that "managed NAND" (like MMC and eMMC, and various SSD flavors) is there to offload some complicated and error-prone tasks like wear-leveling.
Quoted 3 lines · David Hagood
Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.
1. As Russ and David said, OneNAND driver does not really use DMA, because the I/O is done in 2K chunks, and this is just too small piece of data for DMA.
2. UBIFS also compresses data on-the-flight. You may try disabling it and see what changes, but probably not too much, because of the way the driver writes (no DMA). Try mounting with 'compre=none' option, see here:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_mountopts
BTW, some compression testing results may be found here:
http://www.linux-mtd.infradead.org/misc/misc.html#L_ubifs_compr
although they are not 100% relevant for this case.
3. UBIFS provides you greater data reliability. E.g., it CRCs all data (see here
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_checksumming)
OneNAND was very reliable last time we tested it, and we disable _some_ CRC checking for it. Try to use the 'no_chk_data_crc' and get better read speed.
4. UBIFS has 'bulk read' feature which works well on OneNAND, (see here: http://www.linux-mtd.infradead.org/doc/ubifs.html#L_readahead)
Try to enable it as well. You should end up with faster
read speed.
5. Last but not least, UBIFS+OneNAND provides just another level of reliability, comparing to SD. In general, SDs are not very good for storing system libraries, etc. I tried to summarize this at some point here:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_raw_vs_ftl
HTH.
Signature · 6 lines
Best Regards, Artem Bityutskiy (Артём Битюцкий) -- To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Artem Bityutskiy
Quoted 4 lines · David Hagood
Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.
1. As Russ and David said, OneNAND driver does not really use DMA, because the I/O is done in 2K chunks, and this is just too small piece of data for DMA.
All very relevant, but just to avoid confusion; the tests were performed on a Beagleboard with 256MB of Micron NAND.
http://www.micron.com/products/partdetail?part=MT29C2G24MAKLAJG-6 IT
Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.
Quoted 30 lines
2. UBIFS also compresses data on-the-flight. You may try disabling it and see what changes, but probably not too much, because of the way the driver writes (no DMA). Try mounting with 'compre=none' option, see here:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_mountopts
BTW, some compression testing results may be found here:
http://www.linux-mtd.infradead.org/misc/misc.html#L_ubifs_compr
although they are not 100% relevant for this case.
3. UBIFS provides you greater data reliability. E.g., it CRCs all data (see here
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_checksumming)
OneNAND was very reliable last time we tested it, and we disable _some_ CRC checking for it. Try to use the 'no_chk_data_crc' and get better read speed.4. UBIFS has 'bulk read' feature which works well on OneNAND, (see here: http://www.linux-mtd.infradead.org/doc/ubifs.html#L_readahead)
Try to enable it as well. You should end up with faster
read speed.5. Last but not least, UBIFS+OneNAND provides just another level of reliability, comparing to SD. In general, SDs are not very good for storing system libraries, etc. I tried to summarize this at some point here:
Quoted 8 lines
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_raw_vs_ftl
HTH.
--
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Russ Dill
Quoted 7 lines · Artem Bityutskiy
Quoted 3 lines · David Hagood
Well, that's not what I would have expected - I would have thought reads on POP would have been faster than that, and cheaper - the SD being the same speed but less CPU is surprising.
1. As Russ and David said, OneNAND driver does not really use DMA, because the I/O is done in 2K chunks, and this is just too small piece of data for DMA.
All very relevant, but just to avoid confusion; the tests were performed on a Beagleboard with 256MB of Micron NAND.
Ok, sorry. The CRC part should be relevant. The bulk_read part may also be relevant if your HW is able to read multiple NAND pages faster than reading them one-by-one. The compression links should be helpful for compressor selection.
Signature · 6 lines
Best Regards, Artem Bityutskiy (Артём Битюцкий) -- To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Russ Dill
Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.
We had asked for some benchmark data -- anything! -- to get a handle on that, and the prefetch/etc engine; nothing forthcoming, so far.
Hi David
Quoted 7 lines · David Brownell
Quoted 2 lines · Russ Dill
Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.
We had asked for some benchmark data -- anything! -- to get a handle on that, and the prefetch/etc engine; nothing forthcoming, so far.
We'd also have to make sure that the comparison is between the linux-omap kernel and the OMAPZoo kernel, rather than o-z PIO vs. o-z DMA. The OMAPZoom kernel doesn't post any device register writes. That should cause any driver using PIO to drag, compared to the l-o kernel.
- Paul
Quoted 8 lines · Paul Walmsley
Hi David
Quoted 4 lines · David Brownell
Quoted 2 lines · Russ Dill
Also, it appears from looking an the openzoom git, there are some patches to add DMA support in, but I'm not sure what effect they have.
Quoted 8 lines
Quoted 3 lines
We had asked for some benchmark data -- anything! -- to get a handle on that, and the prefetch/etc engine; nothing forthcoming, so far.
We'd also have to make sure that the comparison is between the linux-omap kernel and the OMAPZoo kernel, rather than o-z PIO vs. o-z DMA. The OMAPZoom kernel doesn't post any device register writes. That should cause any driver using PIO to drag, compared to the l-o kernel.
I'd think the first thing to check is two linux-omap PIO flavors:
with vs without the prefetch/postwrite engine. Benchmarking DMA would be a separate issue. Ditto comparing kernels with/without write posting.
I've benchmarked small DMA transfers and it's rather hard to get them faster than io{read,write}32_rep(). Overhead of DMA mapping and just DMA setup/teardown/complete is annoyingly high. While bus overheads aren't large.
Puzzle: get a dma_copypage() to work faster than copy_page(). Or a dma_clear_page() faster than clear_page(). Not easy...
- Dave
Quoted 4 lines · David Brownell
I'd think the first thing to check is two linux-omap PIO flavors:
with vs without the prefetch/postwrite engine. Benchmarking DMA would be a separate issue. Ditto comparing kernels with/without write posting.
Makes sense to me. FIFO-less GPMC PIO should be full of fail.
Quoted 2 lines
Puzzle: get a
dma_copypage()to work faster thancopy_page(). Or adma_clear_page()faster thanclear_page(). Not easy...
Doing it via the DMA engine may save power, since MPU can sleep.
- Paul
Quoted 4 lines · Paul Walmsley
Quoted 2 lines
Puzzle: get a
dma_copypage()to work faster thancopy_page(). Or adma_clear_page()faster thanclear_page(). Not easy...Doing it via the DMA engine may save power, since MPU can sleep.
But the CPU overhead of calling the DMA engine can exceed that of the memcpy()/memset() ... ;)
Another concern is cache impact. In some cases, having the dirty data in dcache is a big win. With DMA, the cache will have been purged.
It'd be nice to see DMA versions of this stuff winning; all I'm saying is that such wins are hard to achieve.
- Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · David Brownell
Quoted 4 lines · Paul Walmsley
Quoted 2 lines
Puzzle: get a
dma_copypage()to work faster thancopy_page(). Or adma_clear_page()faster thanclear_page(). Not easy...Doing it via the DMA engine may save power, since MPU can sleep.
But the CPU overhead of calling the DMA engine can exceed that of the memcpy()/memset() ... ;)
Quoted 6 lines
Another concern is cache impact. In some cases, having the dirty data in dcache is a big win. With DMA, the cache will have been purged.
It'd be nice to see DMA versions of this stuff winning; all I'm saying is that such wins are hard to achieve.
If the DMA transfer can loop over the small FIFO by and transfer larger chunks of data, the setup overhead is small. Currently at least tusb and ASoC loop the DMA over the FIFO.
This way you can let the hardware trigger huge reads or writes from the small FIFO. To loop over the FIFO, set dst_fi or src_fi to a _negative_ number.
Maybe onenand could use this too? If you can tell onenand to read or write multiple FIFOs, and there is a hardware DMA request line, the DMA request line can tell onenand to reload the FIFO.
Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
We'd also have to make sure that the comparison is between the linux-omap kernel and the OMAPZoo kernel, rather than o-z PIO vs. o-z DMA. The OMAPZoom kernel doesn't post any device register writes. That should cause any driver using PIO to drag, compared to the l-o kernel.
There effetely are a few levels of posting. The patch stops _ARM_ PIO to peripheral control registers from hanging out in the interconnect. Other levels still are there. The big winner of interconnect from ARM side is DDR operations which is still gets posted at all levels. The size of buffering in interconnect is like a couple cache lines so it will fill up pretty fast. DMA and other initiators are not impacted as they don't use arm-mmu attributes.
Really, if you think about it "big" block writes may not be impacted either as you will back up at the slower device's speed. Cache is a familiar example. If you write out 100M quickly you will get to a point where you are bottlenecked on main memory speed very quickly (every write is a miss or cast out at some point). The fact you have a cache in the way doesn't matter.
It can make some difference if you're intermixing some small PIOs with other work. In general benchmarks I've yet to see the hit on system against all the bigger noises. Probably I can construct a case where ~10% is lost.
I recall some tests which have dma + prefetch working on nand. I'll see if I can dig them up.
I've was at several meetings a year back where different memory vendors come in and showed with some tweak they can get 2x l-o on flash. Then if you also take their device optimized file system you will get like 5x. Hopefully at least the in tree 2x is gone now that it's a year later.
Regards,
Richard W.