Great jffs2 speedup
Hi,
We started using new CVS code that provides EBS and fragtree method. Below are some measurements of mount times for old and new setup.
NAND partitions size tested were:
flash4 - 4Mb
flash5 - 12Mb
flash6 - 16Mb
OLD setup uses ~April 2005 MTD code, 2.6.11 kernel, no EBS and produces following:
Mounting flash4 clean partition [1]
real 0m 0.83s
user 0m 0.01s
sys 0m 0.83s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition [2]
real 0m 8.37s
user 0m 0.01s
sys 0m 8.35s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 2.70s
user 0m 0.02s
sys 0m 2.38s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 26.42s
user 0m 0.01s
sys 0m 26.40s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.29s
user 0m 0.02s
sys 0m 3.16s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 35.81s
user 0m 0.01s
sys 0m 35.77s
-- -- -- -- -- -- -- -- -- --
On new setup - using yesterdays MTD CVS and 2.6.12 kernel with EBS enabled:
Mounting flash4 clean partition [1]
real 0m 3.75s user 0m 0.02s sys 0m 1.17s -- -- -- -- -- -- -- -- -- -- Mounting flash4 dirty partition [2] real 0m 0.75s user 0m 0.02s sys 0m 0.73s -- -- -- -- -- -- -- -- -- -- Mounting flash5 clean partition real 0m 3.55s user 0m 0.02s sys 0m 3.47s -- -- -- -- -- -- -- -- -- -- Mounting flash5 dirty partition real 0m 2.12s user 0m 0.03s sys 0m 2.08s -- -- -- -- -- -- -- -- -- -- Mounting flash6 clean partition real 0m 4.82s user 0m 0.01s sys 0m 4.63s -- -- -- -- -- -- -- -- -- -- Mounting flash6 dirty partition real 0m 2.82s user 0m 0.01s sys 0m 2.80s
-- -- -- -- -- -- -- -- -- --
Compared to old setup this is GREAAAAAT improvment (if my mesurements are correct:). Great job!
This holds true /only/ if partition is erased beforehand and then mounted, filled with files, umounted and mounted again to mesure mount time for dirty partition. Mounting jffs2 partition filled on old setup shows no performance improvement.
We found sumtool utility in tools dir and if my understanding is correct it is used to convert jffs2 image with no EBS to jffs2 image with EBS.
Is this the only way to upgrade 'old' jffs2 partitions already in place (on the nand flash) to use EBS? Thing is, we have several system up and running in the wild and would like to upgrade jffs2 partitions as-painless-as-possible, preferably without complete reflash of the systems.
I've just ran sumtool on old raw 4Mb mtd char device (containing jffs2 fs) and it produced 4Mb image on the output. I figure, if I copy this image back to flash and test it with EBS supported kernel it should recognize it as new jffs2 image and mount it super-fast. I still need to erase the nand partition and gather enough ram for temporary EBS image on the system and so this could be a problem in our setup.
regards,
hinko
[1] Mounting clean partition means that we erased the partition with flash_eraseall and then mounted it. mount time was measured with 'time'.
[2] After cleaning up the partition, then creating, appending, erasing several small files on it, we umounted the partition and mounted it back. mount time was measured eith 'time'.
All partitions contained 150 - 700 few kbytes sizes files and were ~80 % full when dirty.
Quoted 11 lines · On Wed, 28 September 2005 11:36:13 +0200, [email protected] wrote:
On new setup - using yesterdays MTD CVS and 2.6.12 kernel with EBS enabled:
Mounting flash4 clean partition [1]real 0m 3.75s user 0m 0.02s sys 0m 1.17s -- -- -- -- -- -- -- -- -- -- Mounting flash4 dirty partition [2] real 0m 0.75s user 0m 0.02s sys 0m 0.73s
I assume above numbers are for a dirty partition and below for a clean one?
Jörn
Signature · 3 lines
Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Perlis's Programming Proverb #58, SIGPLAN Notices, Sept. 1982
Quoted 14 lines · Jörn Engel
Quoted 10 lines · On Wed, 28 September 2005 11:36:13 +0200, [email protected] wrote:
On new setup - using yesterdays MTD CVS and 2.6.12 kernel with EBS enabled:
Mounting flash4 clean partition [1]real 0m 3.75s user 0m 0.02s sys 0m 1.17s -- -- -- -- -- -- -- -- -- -- Mounting flash4 dirty partition [2] real 0m 0.75s user 0m 0.02s sys 0m 0.73s
Quoted 2 lines
I assume above numbers are for a dirty partition and below for a clean one?
No.
I've used the same script - It seemed little weird, but ... I'll do more tests in days to come, hopefully I'll be able solve this too.
regrds,
hinko
Quoted 16 lines · On Wed, 28 September 2005 17:26:05 +0200, [email protected] wrote:
Quoted 15 lines · Jörn Engel
Quoted 11 lines · On Wed, 28 September 2005 11:36:13 +0200, [email protected] wrote:
On new setup - using yesterdays MTD CVS and 2.6.12 kernel with EBS enabled:
Mounting flash4 clean partition [1]real 0m 3.75s user 0m 0.02s sys 0m 1.17s -- -- -- -- -- -- -- -- -- -- Mounting flash4 dirty partition [2] real 0m 0.75s user 0m 0.02s sys 0m 0.73s
Quoted 7 lines
Quoted 2 lines
I assume above numbers are for a dirty partition and below for a clean one?
No.
I've used the same script - It seemed little weird, but ... I'll do more tests in days to come, hopefully I'll be able solve this too.
Then your results don't make sense to me. Either you made a stupid mistake when measuring this or you have found something quite interesting. Remember, most inventions don't start with the word "Heureka!" but rather with "This is odd!".
Can anyone verify/falsify?
Jörn
Signature · 4 lines
To announce that there must be no criticism of the President, or that we are to stand by the President, right or wrong, is not only unpatriotic and servile, but is morally treasonable to the American public. -- Theodore Roosevelt, Kansas City Star, 1918
Quoted 13 lines · Jörn Engel
Quoted 10 lines · On Wed, 28 September 2005 11:36:13 +0200, [email protected] wrote:
On new setup - using yesterdays MTD CVS and 2.6.12 kernel with EBS enabled:
Mounting flash4 clean partition [1]real 0m 3.75s user 0m 0.02s sys 0m 1.17s -- -- -- -- -- -- -- -- -- -- Mounting flash4 dirty partition [2] real 0m 0.75s user 0m 0.02s sys 0m 0.73s
Here are more tests from our system. Setup is the same as before:
system1: 2.6.11, April MTD, no EBS
system2: 2.6.12, Latest CVS, EBS
I haven't looked at the system2 clean/dirty mount times much closer, but
It appears that dirty partition is mounted faster evrytime - hence
several iterarions of the test.
I also added my testing script...
regards,
hinko
FLASH test v1.0 START @ Thu Sep 29 18:21:59 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 2.20s
user 0m 0.03s
sys 0m 0.55s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.28s
user 0m 0.01s
sys 0m 5.24s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.70s
user 0m 0.02s
sys 0m 1.62s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.40s
user 0m 0.01s
sys 0m 17.34s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.24s
user 0m 0.02s
sys 0m 2.17s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.29s
user 0m 0.03s
sys 0m 23.15s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 18:53:14 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 18:53:15 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.02s
sys 0m 0.56s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.49s
user 0m 0.02s
sys 0m 5.44s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.75s
user 0m 0.02s
sys 0m 1.63s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.52s
user 0m 0.01s
sys 0m 17.45s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.20s
user 0m 0.03s
sys 0m 2.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.13s
user 0m 0.01s
sys 0m 23.02s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 19:24:11 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 19:24:11 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.59s
user 0m 0.01s
sys 0m 0.57s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.37s
user 0m 0.03s
sys 0m 5.32s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.73s
user 0m 0.01s
sys 0m 1.63s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.47s
user 0m 0.02s
sys 0m 17.37s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.24s
user 0m 0.03s
sys 0m 2.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.25s
user 0m 0.03s
sys 0m 23.13s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 19:54:47 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 19:54:48 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.01s
sys 0m 0.57s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.52s
user 0m 0.01s
sys 0m 5.49s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.77s
user 0m 0.01s
sys 0m 1.63s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.29s
user 0m 0.02s
sys 0m 17.20s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.13s
user 0m 0.02s
sys 0m 2.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.14s
user 0m 0.01s
sys 0m 23.03s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 20:25:24 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 20:25:25 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.03s
sys 0m 0.54s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.19s
user 0m 0.03s
sys 0m 5.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.69s
user 0m 0.04s
sys 0m 1.61s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.42s
user 0m 0.01s
sys 0m 17.34s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.98s
user 0m 0.01s
sys 0m 2.16s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.13s
user 0m 0.01s
sys 0m 23.04s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 20:55:58 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 20:55:58 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.02s
sys 0m 0.56s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.41s
user 0m 0.03s
sys 0m 5.36s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.71s
user 0m 0.02s
sys 0m 1.63s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.43s
user 0m 0.01s
sys 0m 17.36s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.22s
user 0m 0.02s
sys 0m 2.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.17s
user 0m 0.02s
sys 0m 23.05s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 21:26:39 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 21:26:39 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.01s
sys 0m 0.57s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.12s
user 0m 0.03s
sys 0m 5.08s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.71s
user 0m 0.01s
sys 0m 1.63s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.33s
user 0m 0.02s
sys 0m 17.22s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.21s
user 0m 0.02s
sys 0m 2.17s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.18s
user 0m 0.03s
sys 0m 23.08s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 21:57:22 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 21:57:23 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.02s
sys 0m 0.56s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.26s
user 0m 0.02s
sys 0m 5.21s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 3.26s
user 0m 0.02s
sys 0m 1.62s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.35s
user 0m 0.01s
sys 0m 17.26s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.23s
user 0m 0.02s
sys 0m 2.17s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.21s
user 0m 0.03s
sys 0m 23.07s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 22:28:18 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 22:28:19 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.58s
user 0m 0.01s
sys 0m 0.57s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.25s
user 0m 0.01s
sys 0m 5.22s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.74s
user 0m 0.01s
sys 0m 1.66s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.21s
user 0m 0.02s
sys 0m 17.13s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.24s
user 0m 0.01s
sys 0m 2.18s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.13s
user 0m 0.01s
sys 0m 23.00s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 22:59:27 UTC 2005
FLASH test v1.0 START @ Thu Sep 29 22:59:27 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 2.20s
user 0m 0.02s
sys 0m 0.56s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 5.44s
user 0m 0.03s
sys 0m 5.38s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 1.76s
user 0m 0.02s
sys 0m 1.65s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 17.41s
user 0m 0.03s
sys 0m 17.28s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.19s
user 0m 0.01s
sys 0m 2.17s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 23.15s
user 0m 0.01s
sys 0m 23.04s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Thu Sep 29 23:30:24 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 01:23:39 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.75s
user 0m 0.02s
sys 0m 0.73s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.49s
user 0m 0.02s
sys 0m 0.48s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 2.23s
user 0m 0.02s
sys 0m 2.16s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.44s
user 0m 0.01s
sys 0m 1.42s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.25s
user 0m 0.02s
sys 0m 2.85s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.90s
user 0m 0.03s
sys 0m 1.86s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 01:53:43 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 01:53:43 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.77s
user 0m 0.03s
sys 0m 0.74s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.48s
user 0m 0.02s
sys 0m 0.47s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 2.22s
user 0m 0.01s
sys 0m 2.16s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.50s
user 0m 0.02s
sys 0m 1.47s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.89s
user 0m 0.02s
sys 0m 2.86s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.85s
user 0m 0.01s
sys 0m 1.85s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 02:24:17 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 02:24:18 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 2.15s
user 0m 0.02s
sys 0m 0.73s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.49s
user 0m 0.02s
sys 0m 0.47s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 3.18s
user 0m 0.02s
sys 0m 2.13s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.45s
user 0m 0.02s
sys 0m 1.42s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.18s
user 0m 0.02s
sys 0m 2.86s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.85s
user 0m 0.02s
sys 0m 1.82s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 02:55:11 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 02:55:11 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 2.15s
user 0m 0.02s
sys 0m 0.73s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.50s
user 0m 0.01s
sys 0m 0.49s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 3.32s
user 0m 0.02s
sys 0m 2.14s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.52s
user 0m 0.01s
sys 0m 1.50s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.92s
user 0m 0.01s
sys 0m 2.84s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 2.02s
user 0m 0.02s
sys 0m 2.00s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 03:25:28 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 03:25:29 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.75s
user 0m 0.01s
sys 0m 0.74s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.51s
user 0m 0.02s
sys 0m 0.49s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 2.21s
user 0m 0.01s
sys 0m 2.14s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.43s
user 0m 0.01s
sys 0m 1.42s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.22s
user 0m 0.03s
sys 0m 2.84s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.87s
user 0m 0.02s
sys 0m 1.83s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 03:55:30 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 03:55:31 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.76s
user 0m 0.02s
sys 0m 0.74s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.48s
user 0m 0.02s
sys 0m 0.46s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 3.33s
user 0m 0.02s
sys 0m 2.14s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.44s
user 0m 0.02s
sys 0m 1.41s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 2.92s
user 0m 0.01s
sys 0m 2.85s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.88s
user 0m 0.03s
sys 0m 1.84s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 04:27:05 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 04:27:06 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.76s
user 0m 0.01s
sys 0m 0.74s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.49s
user 0m 0.02s
sys 0m 0.48s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 3.29s
user 0m 0.04s
sys 0m 2.12s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.43s
user 0m 0.01s
sys 0m 1.42s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.27s
user 0m 0.02s
sys 0m 2.87s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.89s
user 0m 0.01s
sys 0m 1.88s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 04:57:26 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 04:57:26 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.75s
user 0m 0.03s
sys 0m 0.72s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.49s
user 0m 0.01s
sys 0m 0.48s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 3.20s
user 0m 0.01s
sys 0m 2.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.44s
user 0m 0.02s
sys 0m 1.40s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.22s
user 0m 0.03s
sys 0m 2.84s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.91s
user 0m 0.02s
sys 0m 1.87s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 05:27:40 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 05:27:40 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.76s
user 0m 0.02s
sys 0m 0.73s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.49s
user 0m 0.02s
sys 0m 0.47s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 2.21s
user 0m 0.02s
sys 0m 2.14s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.42s
user 0m 0.02s
sys 0m 1.39s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.16s
user 0m 0.02s
sys 0m 2.85s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.88s
user 0m 0.02s
sys 0m 1.84s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 05:58:35 UTC 2005
FLASH test v1.0 START @ Fri Sep 30 05:58:36 UTC 2005
Nakljucno dodajanje datotek, brisanje datotek Mounting flash4 clean partition
real 0m 0.75s
user 0m 0.02s
sys 0m 0.74s
-- -- -- -- -- -- -- -- -- --
Mounting flash4 dirty partition
real 0m 0.52s
user 0m 0.01s
sys 0m 0.51s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 clean partition
real 0m 2.19s
user 0m 0.01s
sys 0m 2.15s
-- -- -- -- -- -- -- -- -- --
Mounting flash5 dirty partition
real 0m 1.45s
user 0m 0.02s
sys 0m 1.41s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 clean partition
real 0m 3.25s
user 0m 0.02s
sys 0m 2.87s
-- -- -- -- -- -- -- -- -- --
Mounting flash6 dirty partition
real 0m 1.95s
user 0m 0.02s
sys 0m 1.92s
-- -- -- -- -- -- -- -- -- --
FLASH test END @ Fri Sep 30 06:29:16 UTC 2005
Hi Hinko,
Quoted 2 lines · [email protected] wrote:
Compared to old setup this is GREAAAAAT improvment (if my mesurements are correct:). Great job!
Thanks. :)
Quoted 9 lines
We found sumtool utility in tools dir and if my understanding is correct it is used to convert jffs2 image with no EBS to jffs2 image with EBS.
Is this the only way to upgrade 'old' jffs2 partitions already in place (on the nand flash) to use EBS? Thing is, we have several system up and running in the wild and would like to upgrade jffs2 partitions as-painless-as-possible, preferably without complete reflash of the systems.
At this moment: yes, that is the only way. If more people need it we can implement a special extension for EBS. (but it takes time) This extension would make possible if user set a special mount option the system will convert all the non-summarizied erase blocks to summarized. It would take a lot of time (and need some percent free space), but only at that mount.
Or you may try our other technique: CS (Centralized Summary). That technique does not requie user space tool, and cause very fast mount after any clean umount. (if the umount was not clean the speed will be same as before). It is not the part of the official JFFS2 now, we would like to little bit rewrite before bringing on it. But it works already - use the variant which stores the reference in the first erase block. You can download it from our webpage ( http://www.inf.u-szeged.hu/jffs2/ ).
Bye,
Ferenc
Quoted 7 lines · Ferenc Havasi
Quoted 6 lines
Is this the only way to upgrade 'old' jffs2 partitions already in place (on the nand flash) to use EBS? Thing is, we have several system up and running in the wild and would like to upgrade jffs2 partitions as-painless-as-possible, preferably without complete reflash of the systems.
Quoted 6 lines
At this moment: yes, that is the only way. If more people need it we can implement a special extension for EBS. (but it takes time) This extension would make possible if user set a special mount option the system will convert all the non-summarizied erase blocks to summarized. It would take a lot of time (and need some percent free space), but only at that mount.
Just fyi - such a patch should not get merged into cvs. Long-term, the demand for it will approximate zero, so there is no good justification for extra code.
Quoted 7 lines
Or you may try our other technique: CS (Centralized Summary). That technique does not requie user space tool, and cause very fast mount after any clean umount. (if the umount was not clean the speed will be same as before). It is not the part of the official JFFS2 now, we would like to little bit rewrite before bringing on it. But it works already - use the variant which stores the reference in the first erase block. You can download it from our webpage ( http://www.inf.u-szeged.hu/jffs2/ ).
How is the CS information found during mount? Special location?
Jörn
Signature · 2 lines
"Security vulnerabilities are here to stay." -- Scott Culp, Manager of the Microsoft Security Response Center, 2001
Quoted 11 lines · Jörn Engel
Quoted 6 lines · Ferenc Havasi
At this moment: yes, that is the only way. If more people need it we can implement a special extension for EBS. (but it takes time) This extension would make possible if user set a special mount option the system will convert all the non-summarizied erase blocks to summarized. It would take a lot of time (and need some percent free space), but only at that mount.
Just fyi - such a patch should not get merged into cvs. Long-term, the demand for it will approximate zero, so there is no good justification for extra code.
IMO, an external tool would be a good compromise.
Quoted 10 lines
Quoted 8 lines
Or you may try our other technique: CS (Centralized Summary). That technique does not requie user space tool, and cause very fast mount after any clean umount. (if the umount was not clean the speed will be same as before). It is not the part of the official JFFS2 now, we would like to little bit rewrite before bringing on it. But it works already - use the variant which stores the reference in the first erase block. You can download it from our webpage ( http://www.inf.u-szeged.hu/jffs2/ ).
How is the CS information found during mount? Special location?
You may try the approach that is described in the JFFS3 design. That would be cute if one implement it and check how it works :-))
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 9 lines · Jörn Engel
Quoted 8 lines
At this moment: yes, that is the only way. If more people need it we can implement a special extension for EBS. (but it takes time) This extension would make possible if user set a special mount option the system will convert all the non-summarizied erase blocks to summarized. It would take a lot of time (and need some percent free space), but only at that mount.
Quoted 5 lines
Just fyi - such a patch should not get merged into cvs. Long-term, the demand for it will approximate zero, so there is no good justification for extra code.
Yes, I agree.
Quoted 10 lines
Quoted 9 lines
Or you may try our other technique: CS (Centralized Summary). That technique does not requie user space tool, and cause very fast mount after any clean umount. (if the umount was not clean the speed will be same as before). It is not the part of the official JFFS2 now, we would like to little bit rewrite before bringing on it. But it works already - use the variant which stores the reference in the first erase block. You can download it from our webpage ( http://www.inf.u-szeged.hu/jffs2/ ).
Quoted 3 lines
How is the CS information found during mount? Special location?
There is two way (can be selected via kernel config):
The first way stores a reference in the first (usable) erase block. (If it is not free, it moves the data to somewhere else, and reserve). It stores only a reference to it at umount, and a "mount-log" at mount (just to mark that it is invalid in case of unclean umount). It means one page write at mount, one page write at umount. If the erase block consist 16 pages than it is necesarry to erase the first erase block after every 8 mounts. So the typical timelife of it is about 8*100.000 mounting, I think it is big enough.
The other way is to specify the cenratlized summary information with mount option (not to use the first erase block to store this pointer). In this case some other system have to store it (typically on an other file system), and the new location after umount can be read using sysfs.
Ferenc
Quoted 14 lines · Ferenc Havasi
Quoted 2 lines
How is the CS information found during mount? Special location?
There is two way (can be selected via kernel config):
The first way stores a reference in the first (usable) erase block. (If it is not free, it moves the data to somewhere else, and reserve). It stores only a reference to it at umount, and a "mount-log" at mount (just to mark that it is invalid in case of unclean umount). It means one page write at mount, one page write at umount. If the erase block consist 16 pages than it is necesarry to erase the first erase block after every 8 mounts. So the typical timelife of it is about 8*100.000 mounting, I think it is big enough.
Quoted 4 lines
The other way is to specify the cenratlized summary information with mount option (not to use the first erase block to store this pointer). In this case some other system have to store it (typically on an other file system), and the new location after umount can be read using sysfs.
Sounds sane. 100k clean unmounts should be fine for most people. I know a couple of testers that would immediatly create a simple while true; do mount mtd0 /mnt -t jffs2; umount /mnt; done testcase just to prove you wrong, but apart from that...
Jörn
Signature · 4 lines
To recognize individual spam features you have to try to get into the mind of the spammer, and frankly I want to spend as little time inside the minds of spammers as possible. -- Paul Graham
Quoted 8 lines · Ferenc Havasi
The first way stores a reference in the first (usable) erase block. (If it is not free, it moves the data to somewhere else, and reserve). It stores only a reference to it at umount, and a "mount-log" at mount (just to mark that it is invalid in case of unclean umount). It means one page write at mount, one page write at umount. If the erase block consist 16 pages than it is necesarry to erase the first erase block after every 8 mounts. So the typical timelife of it is about 8*100.000 mounting, I think it is big enough.
This spoils the overall wear-leveling, which is no good. But should work.
Quoted 4 lines
The other way is to specify the cenratlized summary information with mount option (not to use the first erase block to store this pointer). In this case some other system have to store it (typically on an other file system), and the new location after umount can be read using sysfs.
This also spoils wear-levelling...
And I don't like CS in general as it is not tolerant to unclean reboots... Albeit, if this does not matter to a specific system - this will work.
Nevertheless, I don't believe CS will make JFFS2 usable on 256MB flashes, due to memory consumprion issues. Nor it will not (IMO) make it scalable, which is sad :-(
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 10 lines · Artem B. Bityutskiy
Quoted 8 lines · Ferenc Havasi
The first way stores a reference in the first (usable) erase block. (If it is not free, it moves the data to somewhere else, and reserve). It stores only a reference to it at umount, and a "mount-log" at mount (just to mark that it is invalid in case of unclean umount). It means one page write at mount, one page write at umount. If the erase block consist 16 pages than it is necesarry to erase the first erase block after every 8 mounts. So the typical timelife of it is about 8*100.000 mounting, I think it is big enough.
This spoils the overall wear-leveling, which is no good. But should work.
Dummy argument. Wear levelling is just a means to avoid the real problem - early wear-out of specific blocks. Spending one erase block for cs is pretty sane, as long as this block doesn't wear-out well before the rest.
Quoted 5 lines
Quoted 4 lines
The other way is to specify the cenratlized summary information with mount option (not to use the first erase block to store this pointer). In this case some other system have to store it (typically on an other file system), and the new location after umount can be read using sysfs.
This also spoils wear-levelling...
How?
Quoted 7 lines
And I don't like CS in general as it is not tolerant to unclean reboots... Albeit, if this does not matter to a specific system - this will work.
Nevertheless, I don't believe CS will make JFFS2 usable on 256MB flashes, due to memory consumprion issues. Nor it will not (IMO) make it scalable, which is sad :-(
CS definitely doesn't solve all problems and it trades the mount time for additional writes - plus the special first erase block. Still, it may have its place.
Jörn
Signature · 2 lines
When in doubt, use brute force. -- Ken Thompson
Quoted 10 lines · Jörn Engel
Quoted 9 lines · Artem B. Bityutskiy
Quoted 8 lines
The first way stores a reference in the first (usable) erase block. (If it is not free, it moves the data to somewhere else, and reserve). It stores only a reference to it at umount, and a "mount-log" at mount (just to mark that it is invalid in case of unclean umount). It means one page write at mount, one page write at umount. If the erase block consist 16 pages than it is necesarry to erase the first erase block after every 8 mounts. So the typical timelife of it is about 8*100.000 mounting, I think it is big enough.
Quoted 7 lines
Quoted 1 line
This spoils the overall wear-leveling, which is no good. But should work.
Dummy argument. Wear levelling is just a means to avoid the real problem - early wear-out of specific blocks. Spending one erase block for cs is pretty sane, as long as this block doesn't wear-out well before the rest.
"As long as" is the essential part. If it is true - nice. In general this can't be true. And, anticipating your remark, I agree that it still may have its place. :-)
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
The other way is to specify the cenratlized summary information with mount option (not to use the first erase block to store this pointer). In this case some other system have to store it (typically on an other file system), and the new location after umount can be read using sysfs.
This also spoils wear-levelling...
How?
The same way as above unless you evenly rotete the CS eraseblock. And if ("as long as" == TRUE) then all is ok.
Quoted 3 lines
CS definitely doesn't solve all problems and it trades the mount time for additional writes - plus the special first erase block. Still, it may have its place.
Thanks, I have no doubts. :-)
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 9 lines · Artem B. Bityutskiy
Quoted 5 lines · Jörn Engel
Dummy argument. Wear levelling is just a means to avoid the real problem - early wear-out of specific blocks. Spending one erase block for cs is pretty sane, as long as this block doesn't wear-out well before the rest.
"As long as" is the essential part. If it is true - nice. In general this can't be true. And, anticipating your remark, I agree that it still may have its place. :-)
Agreed. In general, this patch opens an option for a DoS attack.
Jörn
Signature · 4 lines
Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. -- Doug MacIlroy
Quoted 12 lines · Jörn Engel
Quoted 9 lines · Artem B. Bityutskiy
Quoted 6 lines · Jörn Engel
Dummy argument. Wear levelling is just a means to avoid the real problem - early wear-out of specific blocks. Spending one erase block for cs is pretty sane, as long as this block doesn't wear-out well before the rest.
Quoted 9 lines
Quoted 5 lines
"As long as" is the essential part. If it is true - nice. In general this can't be true. And, anticipating your remark, I agree that it still may have its place. :-)
Agreed. In general, this patch opens an option for a DoS attack.
Maybe I don't understand something. How makes CS possible DoS attacks?
Ferenc
Quoted 2 lines · Ferenc Havasi
Quoted 1 line
Maybe I don't understand something. How makes CS possible DoS attacks?
while true; do mount mtd0 /mnt -t jffs2; umount /mnt; done
Run this for about a day, and the first block will be exhausted. I'm not saying this is easy for a malicious attacker, but certainly possible.
Jörn
Signature · 4 lines
Public Domain - Free as in Beer General Public - Free as in Speech BSD License - Free as in Enterprise Shared Source - Free as in "Work will make you..."
Quoted 7 lines · Jörn Engel
Quoted 3 lines · Ferenc Havasi
Maybe I don't understand something. How makes CS possible DoS attacks?
Quoted 7 lines
while true; do mount mtd0 /mnt -t jffs2; umount /mnt; done
Run this for about a day, and the first block will be exhausted. I'm not saying this is easy for a malicious attacker, but certainly possible.
Yes, it is true. But if an attacker already has the permission to do it, I think he has many simpler and faster way to crash the system. :)
Ferenc
Quoted 3 lines · Ferenc Havasi
Quoted 1 line
Yes, it is true. But if an attacker already has the permission to do it, I think he has many simpler and faster way to crash the system. :)
Doesn't matter. If there is a good way to avoid this, we should do it.
Problem is: Maybe there isn't.
Jörn
Signature · 2 lines
Ninety percent of everything is crap. -- Sturgeon's Law
Quoted 9 lines · Jörn Engel
Quoted 3 lines · Ferenc Havasi
Quoted 1 line
Yes, it is true. But if an attacker already has the permission to do it, I think he has many simpler and faster way to crash the system. :)
Doesn't matter. If there is a good way to avoid this, we should do it.
Problem is: Maybe there isn't.
Depending on what your definition of DoS is, JFFS2 _without_ EBS or CS could be considered that already. If mounts take extremely long times on any decent sized flash, that could be perceived as a DoS.
Think along the lines of "new ogg vorbis player with 4 GiB NAND flash using JFFS2 as the filesystem". You think people are going to want to wait for anything over 2 seconds for their music to be available?
I know, I know... far fetched perhaps. But not any more than your mount/unmount loop :).
josh
Quoted 11 lines · Artem B. Bityutskiy
Quoted 8 lines · Ferenc Havasi
The first way stores a reference in the first (usable) erase block. (If it is not free, it moves the data to somewhere else, and reserve). It stores only a reference to it at umount, and a "mount-log" at mount (just to mark that it is invalid in case of unclean umount). It means one page write at mount, one page write at umount. If the erase block consist 16 pages than it is necesarry to erase the first erase block after every 8 mounts. So the typical timelife of it is about 8*100.000 mounting, I think it is big enough.
Quoted 1 line
This spoils the overall wear-leveling, which is no good. But should work.
It concept of goodness is subjective. IMHO if someting has such a design what works well and fit to the concept of the real thing (the flash) that is good.
Quoted 6 lines
Quoted 4 lines
The other way is to specify the cenratlized summary information with mount option (not to use the first erase block to store this pointer). In this case some other system have to store it (typically on an other file system), and the new location after umount can be read using sysfs.
This also spoils wear-levelling...
It think it is the method which is really does not spoil the wear-leveling. The selection of the storing place uses exactly the same function what JFFS2 uses to store any other data.
Quoted 3 lines
And I don't like CS in general as it is not tolerant to unclean reboots... Albeit, if this does not matter to a specific system - this will work.
Yes, CS was designed for clean umounts. And it can be combinated with EBS, too.
Quoted 3 lines
Nevertheless, I don't believe CS will make JFFS2 usable on 256MB flashes, due to memory consumprion issues. Nor it will not (IMO) make it scalable, which is sad :-(
Yes, it does not solve the memory problems of JFFS2. It is yet another mount time speed up.
Bye,
Ferenc
Quoted 7 lines · Ferenc Havasi
Quoted 2 lines · Artem B. Bityutskiy
This also spoils wear-levelling...
It think it is the method which is really does not spoil the wear-leveling. The selection of the storing place uses exactly the same function what JFFS2 uses to store any other data.
Fair enough. If not on its own, this should work in combination with EBH.
Another problem introduced by this patch, btw, is an increased umount time. If you have to erase some blocks first, that could be substantial.
Jörn
Signature · 2 lines
The strong give up and move away, while the weak give up and stay. -- unknown
Quoted 3 lines · Ferenc Havasi
It think it is the method which is really does not spoil the wear-leveling. The selection of the storing place uses exactly the same function what JFFS2 uses to store any other data.
Then I just don't undesstand how it works. A user selects a bolck X where CS ought to be kept. JFFS2 crereates CS in X. Then, on each mount user specifies X in mount options. And if there are too frequent mounts, A becomes worn-out earlier. To avoid this, user should specify 2 options - where to find CS and where to store it next time? And evenly rotate the CS eraseblock?
Or you mean JFFS2 is itself picking the needed EB? How to find it on next mount then?
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 14 lines · Artem B. Bityutskiy
Quoted 3 lines · Ferenc Havasi
It think it is the method which is really does not spoil the wear-leveling. The selection of the storing place uses exactly the same function what JFFS2 uses to store any other data.
Then I just don't undesstand how it works. A user selects a bolck X where CS ought to be kept. JFFS2 crereates CS in X. Then, on each mount user specifies X in mount options. And if there are too frequent mounts, A becomes worn-out earlier. To avoid this, user should specify 2 options - where to find CS and where to store it next time? And evenly rotate the CS eraseblock?
Quoted 2 lines
Or you mean JFFS2 is itself picking the needed EB? How to find it on next mount then?
Using this (second way) when the user umounts the fs, CS stores the centralized summary with the same function as JFFS2 stores nodes (using nextblock, etc) The starting location of it can be read from sysfs after umount. The user has to store it somewhere else, and specify it as a mount option at next mount. So nothing is stored in fixed place, so I think it does not spoil wear-leveling.
This (second) way can be usefull if there is a small root partition with EBS and big home/usr/... with EBS+CS, and the location of the CS is stored on the root fs. It was /a requisition of our insturial partner.
Ferenc
/
Quoted 7 lines
Using this (second way) when the user umounts the fs, CS stores the centralized summary with the same function as JFFS2 stores nodes (using nextblock, etc) The starting location of it can be read from sysfs after umount. The user has to store it somewhere else, and specify it as a mount option at next mount. So nothing is stored in fixed place, so I think it does not spoil wear-leveling.
Well, are you effectively talking about the situation where there is an additional persistent storage available, and the CS position is kept there. Fair enough, providing it is available. But it is often not :-(
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 3 lines · Artem B. Bityutskiy
Nevertheless, I don't believe CS will make JFFS2 usable on 256MB flashes, due to memory consumprion issues. Nor it will not (IMO) make it scalable, which is sad :-(
CS does, in fact, go a long way toward making 256MB flash parts usable. The devices I am building use 256MB, 512MB, and 1024MB nand flash parts. We have been using Ferenc's CS patch for some time now and we are able to get sub 1 second mount times on 512+ MB filesystems. A big part of this performance is that I have a nand flash controller that interfaces with the nand part on my board. But nonetheless, jffs2 can be used successfully on very large nand parts. Ferenc's EBS and CS work are essential, IMHO, for getting there.
Pete
Hi,
Quoted 1 line
CS does, in fact, go a long way toward making 256MB flash parts usable.
Plese define usable. Yes, it does make flash available for read operations fast but the first time you try to write to it you will be blocked for, I'd say, anywhere from 5 minutes to a few hours depending on how the files were written and assuming full 256 MByte filesystem.
Quoted 3 lines
The devices I am building use 256MB, 512MB, and 1024MB nand flash parts. We have been using Ferenc's CS patch for some time now and we are able to get sub 1 second mount times on 512+ MB filesystems.
Could you please provide time to first write for 256, 512 and 1024 MByte full filesystems? I presume MB stands for MByte, not Mbit ;-)
Quoted 3 lines
A big part of
this performance is that I have a nand flash controller that interfaces with the nand part on my board.
Good point.. Can you estimate speedup due to the hw controller?
Quoted 3 lines
But nonetheless, jffs2 can be used
successfully on very large nand parts. Ferenc's EBS and CS work are essential, IMHO, for getting there.
Most embedded systems cannot rely on proper shutdown. So, IMHO CS is of limited use in this respect.
Sergei Sharonov
Quoted 4 lines · Sergei Sharonov
Plese define usable. Yes, it does make flash available for read operations fast but the first time you try to write to it you will be blocked for, I'd say, anywhere from 5 minutes to a few hours depending on how the files were written and assuming full 256 MByte filesystem.
This has not been my experience. By usable I mean that a 500 MB filesystem with a couple hundred megabytes in use mounts and is usable for reads and writes in a few seconds.
Quoted 2 lines
Could you please provide time to first write for 256, 512 and 1024 MByte full filesystems? I presume MB stands for MByte, not Mbit ;-)
MB is megabyte.
This is the time to mount a freshly erased partition:
# time mount /mnt/mtdb2 real 0m 3.48s user 0m 0.00s sys 0m 1.78s
I then exploded a kernel source tarball into the partition (this took about 2.5 minutes, but that accounts for the time to stream the tarball to my device via ethernet over USB.
# df -h Filesystem Size Used Available Use% Mounted on /dev/mtdblock2 1020.0M 228.6M 791.4M 22% /mnt/mtdb2
# time umount /mnt/mtdb2 real 0m 0.73s user 0m 0.00s sys 0m 0.49s
I was a little unsure about what you meant by "time to first write". How's this?:
# time mount /mnt/mtdb2 && \ time echo "Hello world!" > /mnt/mtdb2/hello.txt real 0m 1.10s user 0m 0.00s sys 0m 1.03s real 0m 0.01s user 0m 0.00s sys 0m 0.00s
# time umount /mnt/mtdb2 real 0m 0.56s user 0m 0.00s sys 0m 0.29s
There is not any difference writing to an existing file:
# time mount /mnt/mtdb2 && \ time echo "# some text" >> /mnt/mtdb2/linux-2.6.13.1/Makefile real 0m 1.10s user 0m 0.00s sys 0m 1.03s real 0m 0.01s user 0m 0.00s sys 0m 0.00s
I then doubled the amount of data in the filesystem.
# time cp -r /mnt/mtdb2/linux-2.6.13.1 /mnt/mtdb2/linux-2.6.13.1-2 real 2m 0.83s user 0m 5.17s sys 1m 7.80s
# df -h Filesystem Size Used Available Use% Mounted on /dev/mtdblock2 1020.0M 431.2M 588.8M 42% /mnt/mtdb2
This yields a linear increase in unmount time, but in absolute terms it is still small.
# time umount /mnt/mtdb2 real 0m 1.20s user 0m 0.00s sys 0m 0.78s
Time to first write again. Mount time is worse:
# time mount /mnt/mtdb2 && \ time echo "# some text" >> /mnt/mtdb2/linux-2.6.13.1/Makefile real 0m 3.60s user 0m 0.00s sys 0m 3.49s real 0m 0.01s user 0m 0.00s sys 0m 0.00s
Let's double the amount of data in the filesystem again. You can see that the read and write times stay consistent:
# time cp -r /mnt/mtdb2/linux-2.6.13.1 /mnt/mtdb2/linux-2.6.13.1-3 && \ time cp -r /mnt/mtdb2/linux-2.6.13.1 /mnt/mtdb2/linux-2.6.13.1-4 real 1m 59.50s user 0m 4.88s sys 1m 7.98s real 2m 2.69s user 0m 5.08s sys 1m 10.57s
# df -h Filesystem Size Used Available Use% Mounted on /dev/mtdblock2 1020.0M 836.2M 183.8M 82% /mnt/mtdb2
Unmount time has stabilized. I've noticed that the garbage collector seems to add some variance to the unmount times.
# time umount /mnt/mtdb2 real 0m 1.94s user 0m 0.00s sys 0m 1.21s
TTFW take 3. Mount times get worse still:
# time mount /mnt/mtdb2 && \ time echo "# some text" >> /mnt/mtdb2/linux-2.6.13.1/Makefile real 0m 16.74s user 0m 0.00s sys 0m 16.53s real 0m 0.01s user 0m 0.00s sys 0m 0.00s
Hope that covers what you wanted to see. For the record, I am using the head of mtd cvs with the centralized summary patch applied. EBS and CS are both enabled.
Quoted 1 line
Good point.. Can you estimate speedup due to the hw controller?
We can get about 15 MB/s for raw reads and about 6 MB/s for raw writes. By raw, I mean reading and writing via an mtd character device to/from a raw partition. I do not know how these numbers compare to other flash access methods; but bit-banging a flash part directly would have to be considerably slower and probably very dependent on the CPU speed. My device has a 384 MHz PowerPC 405 CPU, by the way.
Another advantage to this controller is that it does ECC in hardware and is capable of doing partial page reads. This means that even thought the nand flash part I am using has a 2048 byte page size, I can read as little as 256 bytes (ECC chunk size) at a time. One of the things that hurts jffs2 performance is that it very often reads small chunks that are only a fraction of the nand page size. This is why the nand_base driver caches pages in RAM. I implement a similar chunk caching scheme, but keep the chunks cached in my controller's buffers; this way I avoid extra copying of data.
Quoted 2 lines
Most embedded systems cannot rely on proper shutdown. So, IMHO CS is of limited use in this respect.
My device is usb powered, but has a small battery so that when the device looses bus power, it can shutdown cleanly. However, the thing to remember about centralized summary is that in the unclean shutdown cases, the mount time is no worse than jffs2 without CS. So even if you only expect the filesystem to be cleanly unmounted sometimes, CS can be worth it.
Pete
Quoted 2 lines · Peter Grayson
I was a little unsure about what you meant by "time to first write". How's this?:
Could you please create a 200MB file 'dedekind' (or larger, the larger - the better), mount your FS, run 'time stat dedekind' and report how much time did that command stat took.
Also, just after mount of a fully filled FS (again, the larger is your FS, the better), run top, and look how much time your cpu is busy by executing the GC thread. Basically, you can start writing only after the GC thread has stopped working.
I am really curious about the results.
And another fun is to create a directory with thousands of small (or empty) files, remount the FS and run stat on the directory or enter it. But this is not that fundamental problem and may be fixed.
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 3 lines · Artem B. Bityutskiy
And another fun is to create a directory with thousands of small (or empty) files, remount the FS and run stat on the directory or enter it. But this is not that fundamental problem and may be fixed.
Oh, I'm actually lying, it is fundamental, but it may be relaxed greatly if we switch to balanced trees instead of lists in the internal directory representation.
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 3 lines · Artem B. Bityutskiy
Could you please create a 200MB file 'dedekind' (or larger, the larger - the better), mount your FS, run 'time stat dedekind' and report how much time did that command stat took.
# time dd if=/dev/zero of=/mnt/mtdb2/dedekind bs=1M count=300 300+0 records in 300+0 records out real 1m 37.08s user 0m 0.01s sys 0m 40.64s
# time umount /mnt/mtdb2 real 0m 0.45s user 0m 0.00s sys 0m 0.33s
# time mount /mnt/mtdb2 && time stat /mnt/mtdb2/dedekind real 0m 0.18s user 0m 0.00s sys 0m 0.14s File: `/mnt/mtdb2/dedekind' Size: 314572800 Blocks: 614400 IO Block: 4096 regular file Device: 1f02h/7938d Inode: 2 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 1969-12-31 17:01:29.000000000 -0700 Modify: 1969-12-31 17:03:06.000000000 -0700 Change: 1969-12-31 17:03:06.000000000 -0700 real 0m 9.50s user 0m 0.00s sys 0m 4.47s
And again with a bigger file:
# time dd if=/dev/zero of=/mnt/mtdb2/dedekind bs=1M count=600 600+0 records in 600+0 records out real 3m 6.93s user 0m 0.02s sys 1m 19.27s
# time umount /mnt/mtdb2 real 0m 0.81s user 0m 0.00s sys 0m 0.58s
# time mount /mnt/mtdb2 && time stat /mnt/mtdb2/dedekind real 0m 0.32s user 0m 0.00s sys 0m 0.26s File: `/mnt/mtdb2/dedekind' Size: 629145600 Blocks: 1228800 IO Block: 4096 regular file Device: 1f02h/7938d Inode: 3 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 1969-12-31 17:19:56.000000000 -0700 Modify: 1969-12-31 17:19:56.000000000 -0700 Change: 1969-12-31 17:19:56.000000000 -0700 real 0m 18.72s user 0m 0.00s sys 0m 8.40s
Yet bigger:
# time dd if=/dev/zero of=/mnt/mtdb2/dedekind bs=1M count=900 900+0 records in 900+0 records out real 5m 28.55s user 0m 0.02s sys 1m 55.58s
# time umount /mnt/mtdb2 real 0m 1.17s user 0m 0.00s sys 0m 0.83s
# time mount /mnt/mtdb2 && time stat /mnt/mtdb2/dedekind real 0m 0.45s user 0m 0.00s sys 0m 0.36s File: `/mnt/mtdb2/dedekind' Size: 943718400 Blocks: 1843200 IO Block: 4096 regular file Device: 1f02h/7938d Inode: 4 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 1969-12-31 17:30:46.000000000 -0700 Modify: 1969-12-31 17:30:46.000000000 -0700 Change: 1969-12-31 17:30:46.000000000 -0700 real 0m 28.31s user 0m 0.00s sys 0m 12.93s
So, there seems to be a linear relationship between the time-to-stat and the file size.
Quoted 4 lines
Also, just after mount of a fully filled FS (again, the larger is your FS, the better), run top, and look how much time your cpu is busy by executing the GC thread. Basically, you can start writing only after the GC thread has stopped working.
I do not have the time to do this test right now.
Quoted 1 line
I am really curious about the results.
Me too!
Quoted 3 lines
And another fun is to create a directory with thousands of small (or empty) files, remount the FS and run stat on the directory or enter it. But this is not that fundamental problem and may be fixed.
I'll try to make some time to do these tests next week.
Pete
Quoted 1 line · Peter Grayson
# time dd if=/dev/zero of=/mnt/mtdb2/dedekind bs=1M count=300
Hmm, not bad with zero files, and now with if=/dev/urandom ?
Quoted 2 lines
So, there seems to be a linear relationship between the time-to-stat and the file size.
Neh, that's known fact and this is why we call it unscalable.
Quoted 1 line
I do not have the time to do this test right now.
That's not compulsory :-) That's for you, it may be interesting for you.
Quoted 1 line
I'll try to make some time to do these tests next week.
Yhank you!
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Peter,
thanks for taking time to do the tests.
Quoted 3 lines
Hope that covers what you wanted to see. For the record, I am using the head of mtd cvs with the centralized summary patch applied. EBS and CS are both enabled.
Aha!!! Of course, CS reduced your time to first write down to seconds. Could you repeat the test after the un-clean unmount (e.g. power off without sync)? I bet the numbers will be very different. Probably you also need to be writing to flash during the power off to force full scan on startup.
Quoted 3 lines
Quoted 1 line
Good point.. Can you estimate speedup due to the hw controller?
We can get about 15 MB/s for raw reads and about 6 MB/s for raw writes.
Hmm.. I was getting a read throughput jffs2 -> samba network fs of about 1.1 MB/s (180 MHz ARM9, linux 2.6.10, mtd 3/28/2005). I believe jffs2/mtd was the bottleneck since the throughput was 7.25x higher when files were comming from ext2/RAM. Looks like your hw controller is doing a good job. Not sure how overhead is split between mtd and jffs2.
Quoted 5 lines
By raw, I mean reading and writing via an mtd character device to/from a raw partition. I do not know how these numbers compare to other flash access methods; but bit-banging a flash part directly would have to be considerably slower and probably very dependent on the CPU speed. My device has a 384 MHz PowerPC 405 CPU, by the way.
That helps too ;-)
Quoted 5 lines
Quoted 2 lines
Most embedded systems cannot rely on proper shutdown. So, IMHO CS is of limited use in this respect.
My device is usb powered, but has a small battery so that when the device looses bus power, it can shutdown cleanly.
Tricky, tricky, tricky ;-)
Not everybody is that lucky to have a battery. Reminds me of a conversation with one "high reliability RTOS" vendor, you guess who.
- Is your file system power fail safe?
- Yes, of course. We do require that you use UPS though.
Quoted 5 lines
However, the thing to
remember about centralized summary is that in the unclean shutdown cases, the mount time is no worse than jffs2 without CS. So even if you only expect the filesystem to be cleanly unmounted sometimes, CS can be worth it.
As long as specification allows for an "occasional" 30 minutes startup time.
Your hw controller does sound very interesting.
What is the hardware implementation : asic, fpga? Is it available as a separate product (chip)?
Sergei
Quoted 4 lines · Sergei Sharonov
Aha!!! Of course, CS reduced your time to first write down to seconds. Could you repeat the test after the un-clean unmount (e.g. power off without sync)? I bet the numbers will be very different. Probably you also need to be writing to flash during the power off to force full scan on startup.
I'll add this to my list of "things to try for the list".
Quoted 5 lines
Hmm.. I was getting a read throughput jffs2 -> samba network fs of about 1.1 MB/s (180 MHz ARM9, linux 2.6.10, mtd 3/28/2005). I believe jffs2/mtd was the bottleneck since the throughput was 7.25x higher when files were comming from ext2/RAM. Looks like your hw controller is doing a good job. Not sure how overhead is split between mtd and jffs2.
The mtd layer does not really introduce any overhead; it accounts for the time it takes to get bytes in and out of the hardware. Jffs2, on the other hand, has a number of aspects that use RAM and cpu cycles. Also, the choices jffs2 makes affects how often it has to access flash.
Quoted 6 lines
Tricky, tricky, tricky ;-)
Not everybody is that lucky to have a battery. Reminds me of a conversation with one "high reliability RTOS" vendor, you guess who.
- Is your file system power fail safe?
- Yes, of course. We do require that you use UPS though.
Quoted 7 lines
Quoted 5 lines
However, the thing to
remember about centralized summary is that in the unclean shutdown cases, the mount time is no worse than jffs2 without CS. So even if you only expect the filesystem to be cleanly unmounted sometimes, CS can be worth it.As long as specification allows for an "occasional" 30 minutes startup time.
Touche.
Quoted 3 lines
Your hw controller does sound very interesting.
What is the hardware implementation : asic, fpga? Is it available as a separate product (chip)?
This is a soft core in an fpga. My company may be willing to license its use.
Quoted 7 lines · Peter Grayson
Quoted 4 lines · Sergei Sharonov
Aha!!! Of course, CS reduced your time to first write down to seconds. Could you repeat the test after the un-clean unmount (e.g. power off without sync)? I bet the numbers will be very different. Probably you also need to be writing to flash during the power off to force full scan on startup.
I'll add this to my list of "things to try for the list".
Mount should be anyway fast enough with EBS but w/o CS. But unfortunately:
1. your CPU will be busy for quite long time afterwords;
2. you will in general unable to write during that time - writers will be blocked (sometimes they may write, though).
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 4 lines · Ferenc Havasi
Quoted 4 lines
We found sumtool utility in tools dir and if my understanding is correct it is used to convert jffs2 image with no EBS to jffs2 image with EBS.
Quoted 8 lines
Quoted 5 lines
Is this the only way to upgrade 'old' jffs2 partitions already in place (on the nand flash) to use EBS? Thing is, we have several system up and running in the wild and would like to upgrade jffs2 partitions as-painless-as-possible, preferably without complete reflash of the systems.
At this moment: yes, that is the only way.
Ok, just to be on the right track. I guess we will have to come up with the solution using sumtool.
Quoted 4 lines
Or you may try our other technique: CS (Centralized Summary). That technique does not requie user space tool, and cause very fast mount after any clean umount.
All of our umounts are unclean - or said better application is not taking care of umount at all. System may be powered off anytime. Which raises another question: Will this impact mount time in long run? Eg. after more and more unclean (u)mounts should we expect significant performance degradation?
regards,
hinko
Quoted 5 lines · [email protected] wrote:
All of our umounts are unclean - or said better application is not taking care of umount at all. System may be powered off anytime. Which raises another question: Will this impact mount time in long run? Eg. after more and more unclean (u)mounts should we expect significant performance degradation?
Using EBS there should not be significant performance degradation even if there is no umount at all.
Only CS needs clean umount.
Ferenc
Quoted 6 lines · On Thu, 29 September 2005 12:15:49 +0200, [email protected] wrote:
All of our umounts are unclean - or said better application is not taking care of umount at all. System may be powered off anytime. Which raises another question: Will this impact mount time in long run? Eg. after more and more unclean (u)mounts should we expect significant performance degradation?
Only if your applications write to the fs in small chunks. Known problem, noone cared enough to actually fix it yet.
If most writes are in the range of 4k, you're safe.
Jörn
Signature · 4 lines
My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. -- Edsger W. Dijkstra