jffs2_get_inode_nodes() very very slow
Hello,
On a 64 MiB NAND Flash I created a jffs2 file system. After several tests especially power fail tests resulting in a lot off CRC and Data CRC errors. This appears not to be a problem except it takes a very long time to check the file system.
The initial check (scanning for erased blocks) is done in aproximatly 10 seconds, but after that the jffs2_get_inode_nodes check is running. This process takes up to 8 minutes.
During this time the file-system is inaccessible which in oure case a watchdog decides to reboot the system, making it even worse.
It appears that the check is done with a stepsize of 16 bytes. On an 64MiB NAND flash it does this check +/- 4 million times.
Is there a way to reduce the time to check the file-system and/or how can I recover from these errors.
Kind Regards,
Rudi.
Signature · 4 lines
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. Benjamin Franklin (1706-1790), Letter to Josiah Quincy, Sept. 11, 1773. GnuPG Key fingerprint = 706C E2AC 7AE2 BCEE 04EB A962 0A75 7F9B 07A1 83E8
Quoted 4 lines · Rudi Engelbertink
Hello,
On a 64 MiB NAND Flash I created a jffs2 file system. After several tests especially power fail tests resulting in a lot off CRC and Data CRC errors.
How do you do your power-fail tests?
Quoted 5 lines
This appears not to be a problem except it takes a very long time to check the file system.
The initial check (scanning for erased blocks) is done in aproximatly 10 seconds, but after that thejffs2_get_inode_nodescheck is running. This process takes up to 8 minutes.
Do you mean this happens every time or in case of "powerfails"?
Quoted 4 lines
During this time the file-system is inaccessible which in oure case a watchdog decides to reboot the system, making it even worse. It appears that the check is done with a stepsize of 16 bytes. On an 64MiB NAND flash it does this check +/- 4 million times.
That's strange. JFFS2 should share processor's cycles.
Quoted 16 lines
Is there a way to reduce the time to check the file-system and/or how can I recover from these errors.
Kind Regards,
Rudi.Signature · 8 lines
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. Benjamin Franklin (1706-1790), Letter to Josiah Quincy, Sept. 11, 1773. GnuPG Key fingerprint = 706C E2AC 7AE2 BCEE 04EB A962 0A75 7F9B 07A1 83E8 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.
On Tue, 1 Feb 2005 16:03:31 +0000 (GMT), Artem B. Bityuckiy wrote
Quoted 7 lines · Rudi Engelbertink
Quoted 5 lines
Hello,
On a 64 MiB NAND Flash I created a jffs2 file system. After several tests especially power fail tests resulting in a lot off CRC and Data CRC errors.
How do you do your power-fail tests?
The powerfail tests are done by:
A. a clock. Just turn off and on the power every 15 minutes and start the an application which logs 2 40-60 byte events every second. B. an internal (hardware) watchdog which reboots the system when the 'application' appears to be dead for 10 minutes.
Quoted 7 lines
Quoted 6 lines
This appears not to be a problem except it takes a very long time to check the file system.
The initial check (scanning for erased blocks) is done in aproximatly 10 seconds, but after that thejffs2_get_inode_nodescheck is running. This process takes up to 8 minutes.Do you mean this happens every time or in case of "powerfails"?
Yes it happens every time the system reboots or when the (nand) filesystem is mounted.
Quoted 6 lines
Quoted 5 lines
During this time the file-system is inaccessible which in oure case a watchdog decides to reboot the system, making it even worse. It appears that the check is done with a stepsize of 16 bytes. On an 64MiB NAND flash it does this check +/- 4 million times.
That's strange. JFFS2 should share processor's cycles.
Yes, the root is accessable but the directory where the logging is stored, is unavailable for several minutes.
Quoted 18 lines
Quoted 17 lines
Is there a way to reduce the time to check the file-system and/or how can I recover from these errors.
Kind Regards,
Rudi.Signature · 8 lines
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. Benjamin Franklin (1706-1790), Letter to Josiah Quincy, Sept. 11, 1773. GnuPG Key fingerprint = 706C E2AC 7AE2 BCEE 04EB A962 0A75 7F9B 07A1 83E8 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
Quoted 4 lines
--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.
RGDS Rudi.
Signature · 4 lines
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. Benjamin Franklin (1706-1790), Letter to Josiah Quincy, Sept. 11, 1773. GnuPG Key fingerprint = 706C E2AC 7AE2 BCEE 04EB A962 0A75 7F9B 07A1 83E8
Quoted 6 lines · Rudi Engelbertink
The powerfail tests are done by:
A. a clock. Just turn off and on the power every 15 minutes and start the an application which logs 2 40-60 byte events every second.
...
Yes, the root is accessable but the directory where the logging is stored, is unavailable for several minutes.
You hit the worst case for JFFS2.
Your event logging creates tons of small nodes for your logfiles. Actually are about 96000 very small nodes on the chip, so the mount time is not surprising. This also will use a quite big amount of memory.
We have no real cure for this at the moment. We have this scenario in our design list for JFFS3. I remember that somebody else came up with this issue sometime ago. IIRC changing the log method did help a bit.
while true
do
log_event
let cnt=cnt+1
if [ $cnt -ge $LIMIT ]
then
closelog
cat log.small >>log.big
rm log.small
fi
done
This converts the small nodes to bigger nodes when the data are appended to log.big. I guess garbage collection should kick in quite fast and clean up the small nodes. It might not totally go away, but it should be much better than now. This will also give you more capacity on your partition as the small nodes consist mostly of node overhead.
You may also try YAFFS for the logging partition. It should deal with this situation a bit better.
tglx
Quoted 6 lines · Thomas Gleixner
This converts the small nodes to bigger nodes when the data are appended to log.big. I guess garbage collection should kick in quite fast and clean up the small nodes. It might not totally go away, but it should be much better than now. This will also give you more capacity on your partition as the small nodes consist mostly of node overhead.
We could possibly do this for ourselves quite easily. When writing to a page which already has more than N fragments, obsolete them all and write the whole page out again.
Or maybe, when writing the _last_ byte in a page which already has N fragments, rewrite the whole page?
Signature · 1 line
dwmw2
Quoted 11 lines · David Woodhouse
Quoted 6 lines · Thomas Gleixner
This converts the small nodes to bigger nodes when the data are appended to log.big. I guess garbage collection should kick in quite fast and clean up the small nodes. It might not totally go away, but it should be much better than now. This will also give you more capacity on your partition as the small nodes consist mostly of node overhead.
We could possibly do this for ourselves quite easily. When writing to a page which already has more than N fragments, obsolete them all and write the whole page out again.
IMO, perfectly good idea! I think the reasonable thrashhold is about 3-6 fragments.
Quoted 13 lines
Or maybe, when writing the _last_ byte in a page which already has N fragments, rewrite the whole page?
Signature · 7 lines
dwmw2 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.
On Wed, 02 Feb 2005 11:26:51 +0100, Thomas Gleixner wrote
Quoted 30 lines · Rudi Engelbertink
Quoted 6 lines
The powerfail tests are done by:
A. a clock. Just turn off and on the power every 15 minutes and start the an application which logs 2 40-60 byte events every second.
...
Yes, the root is accessable but the directory where the logging is stored, is unavailable for several minutes.You hit the worst case for JFFS2.
Your event logging creates tons of small nodes for your logfiles. Actually are about 96000 very small nodes on the chip, so the mount time is not surprising. This also will use a quite big amount of memory.
We have no real cure for this at the moment. We have this scenario in our design list for JFFS3. I remember that somebody else came up with this issue sometime ago. IIRC changing the log method did help a bit.
while true
do
log_event
let cnt=cnt+1
if [ $cnt -ge $LIMIT ]
then
closelog
cat log.small >>log.big
rm log.small
fi
done
Quoted 6 lines
This converts the small nodes to bigger nodes when the data are appended to log.big. I guess garbage collection should kick in quite fast and clean up the small nodes. It might not totally go away, but it should be much better than now. This will also give you more capacity on your partition as the small nodes consist mostly of node overhead.
Digging through the logfile I noticed that several inodes had a lot of versions. Especialy the inode #737 which had 60000+ versions. This is a bit strange because this is a 16 byte pointer file which will be opened once and never closed. This should not result in 60000+ versions. A copy move action (cp -p file file.cp && mv file.cp file) indeed makes a big performance improvement.
Will a regular close open of this file make some improvement regarding the stored version information of this inode. During normal operations this file, and some other files, never gets closed. So there will be never a 'commit'. So after a powerfail, all the inode version information must be checked.
Quoted 5 lines
You may also try YAFFS for the logging partition. It should deal with this situation a bit better.
tglx
RGDS Rudi.
Signature · 4 lines
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. Benjamin Franklin (1706-1790), Letter to Josiah Quincy, Sept. 11, 1773. GnuPG Key fingerprint = 706C E2AC 7AE2 BCEE 04EB A962 0A75 7F9B 07A1 83E8
Quoted 11 lines · Rudi Engelbertink
Quoted 6 lines
This converts the small nodes to bigger nodes when the data are appended to log.big. I guess garbage collection should kick in quite fast and clean up the small nodes. It might not totally go away, but it should be much better than now. This will also give you more capacity on your partition as the small nodes consist mostly of node overhead.
Digging through the logfile I noticed that several inodes had a lot of versions. Especialy the inode #737 which had 60000+ versions. This is a bit strange because this is a 16 byte pointer file which will be opened once and never closed. This should not result in 60000+ versions.
It's because JFFS2 does synchronous writes. That means, every write() you do, is written to the FLASH immidiately. So every small log write goes into FLASH and builds a node.
The version count is just incremented and the old nodes are declared obsolete. I'm not sure, why the blocks are not garbage collected.
Quoted 5 lines
A copy move action (cp -p file file.cp && mv file.cp file) indeed makes a big performance improvement.
Will a regular close open of this file make some improvement regarding the stored version information of this inode. During normal operations this file, and some other files, never gets closed. So there will be never a 'commit'.
close / open actually does not help. Without changes to JFFS2 code only the copy to a different file will help you.
tglx
Quoted 13 lines · Rudi Engelbertink
Hello,
On a 64 MiB NAND Flash I created a jffs2 file system. After several tests especially power fail tests resulting in a lot off CRC and Data CRC errors. This appears not to be a problem except it takes a very long time to check the file system.
The initial check (scanning for erased blocks) is done in aproximatly 10 seconds, but after that thejffs2_get_inode_nodescheck is running. This process takes up to 8 minutes.
During this time the file-system is inaccessible which in oure case a watchdog decides to reboot the system, making it even worse.
It appears that the check is done with a stepsize of 16 bytes. On an 64MiB NAND flash it does this check +/- 4 million times.
Hmm, which kernel version ?
Can you switch on JFFS2 debugging (debuglevel 1), log the output on a serial line and send me the log, please ?
tglx