JFFS2: truncated files after power loss scenario

Hi mtd list,

What is the expected behavior for files beeing written on a jffs2 (on
nand) during a power loss? Would an incomplete file exist, or would the journaling do a roll back because of an incomplete transaction?
Is there a difference with new files created, or existing files beeing modified?

My tests show that on next bootup the file exists, but is truncated. I have not seen any journaling related roll backs.

I took a look at this, because we have seen a truncated configuration file once. I'm not sure what caused this. Perhaps a power loss during write.

What do you think about the following jffs2dump?:
( dwnld.conf was created once by extracting it from an archive,
it might have been modified afterwards. (fopen "w+"; fprintf; close), the file existed afterwards, but was truncated in to 4k, then during error analysis the file was renamed "mv dwnld.conf
dwnld.conf.org" and a new file dwnld.conf was created there)

Dirent     node at 0x00346f2c, totlen 0x00000032, #pino    41, version    23, #ino        42, nsize       10, name dwnld.conf
Dirent     node at 0x003e63e8, totlen 0x00000032, #pino    41, version     5, #ino        42, nsize       10, name dwnld.conf
Dirent     node at 0x009c19e0, totlen 0x00000032, #pino    41, version    28, #ino      2484, nsize       10, name dwnld.conf
Dirent     node at 0x009fc398, totlen 0x00000036, #pino    41, version    27, #ino        42, nsize       14, name dwnld.conf.org
Inode      node at 0x003e63a4, totlen 0x00000044, #ino     42, version     1, isize        0, csize        0, dsize        0, offset 0
Inode      node at 0x003e641c, totlen 0x000005d8, #ino     42, version     2, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x003e69f4, totlen 0x000006f5, #ino     42, version     3, isize     8192, csize     1713, dsize     4096, offset 4096
Inode      node at 0x003e70ec, totlen 0x0000028f, #ino     42, version     4, isize     9445, csize      587, dsize     1253, offset 8192
Inode      node at 0x003e737c, totlen 0x00000044, #ino     42, version     5, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x003e73c0, totlen 0x00000044, #ino     42, version     6, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x003e7404, totlen 0x00000044, #ino     42, version     7, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x009bcba8, totlen 0x000005d8, #ino     42, version     9, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x00e51ea0, totlen 0x000005d8, #ino     42, version     9, isize     4096, csize     1428, dsize     4096, offset 0

I reproduced a good-case of the actions on the file with a fresh filesystem:

Dirent     node at 0x00364e3c, totlen 0x00000032, #pino    41, version    11, #ino        42, nsize       10, name dwnld.conf
Dirent     node at 0x00553000, totlen 0x00000036, #pino    41, version    12, #ino        42, nsize       14, name dwnld.conf.org
Dirent     node at 0x00553038, totlen 0x00000032, #pino    41, version    13, #ino         0, nsize       10, name dwnld.conf
Dirent     node at 0x00555844, totlen 0x00000032, #pino    41, version    14, #ino      1171, nsize       10, name dwnld.conf
Inode      node at 0x00364e70, totlen 0x00000044, #ino     42, version    10, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x00449cb4, totlen 0x000005d8, #ino     42, version     2, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x0044a2c8, totlen 0x000006f5, #ino     42, version     3, isize     8192, csize     1713, dsize     4096, offset 4096
Inode      node at 0x0044accc, totlen 0x0000028f, #ino     42, version     4, isize     9445, csize      587, dsize     1253, offset 8192
Inode      node at 0x0096e1c4, totlen 0x00000044, #ino     42, version    11, isize        0, csize        0, dsize        0, offset 0
Inode      node at 0x0096e208, totlen 0x000005d8, #ino     42, version    12, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x0096e7e0, totlen 0x000006f5, #ino     42, version    13, isize     8192, csize     1713, dsize     4096, offset 4096
Inode      node at 0x0096eed8, totlen 0x0000028f, #ino     42, version    14, isize     9445, csize      587, dsize     1253, offset 8192

I wonder about the Inode #42 Version 5 to 9 and about why two Dirent exist(ed) for #ino 42. Do you think this was caused by a power loss scenario?

The truncated file error happened on a 2.6.18 based kernel.
I have done power loss tests on 2.6.34, which showed truncated files as well.

regards, Sven Jaborek

How big was this file?

No, this is how JFFS2 works, this is related to the design. In short, only v9 is valid, other versions are invalid, you can ignore them.

Please, give more information:

1. Your flash type, if NAND - page size.
2. What do you do to the file in your test - how exactly you change it.
3. How big is the file?

Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Hi

9445 bytes.

Need to check this. Page size is 128k i think.

I did two tests.
First one is a bash script running an endless loop. It creates new files over and over by copying the 9445 bytes size file every time with a new filename.
A power loss with this script running always produced the truncated last file. It had the size of 4k or 8k then.

The second test reflects more our simple use case. A C-program which uses fopen with "w+", and then uses GLib GKeyFile operations to modify a key and then store the file again. An fclose follows of course.

9445 bytes

Please, could you say something general about what the journaling in jffs2+nand should be able to do about power loss scenarios? I could not find specs about that.

br, Sven

That's eraseblock size, PAGE size must be 2KiB in your case.

Then it is expected I guess. Your I bet your script opens the file, truncates it, then writes to it. So when you have a power cut, you can find your file truncated to 0, 4, or 8KiB, of 9445 bytes. In your case JFFS2 writes in 4KiB pieces (maximum it can do), this is why you see 4 and 8 KiB.

I believe your library is truncating the file as well. You can check this with strace.

If you write a program which just overrides the file without truncation, you won't see truncations. But of course, yous will see inconsistent file contents containing, say, 4KiB of new data, and the rest is old data.

The bottom line is that it is difficult to fix broken applications by changing JFFS2. The right way to update configuration files is described here:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change

Oh, no, but JFFS2 is very simple FS. It is almost 100% synchronous, only last 2KiB are cached in the write-buffer. Se basically, if you cut power at any point, the maximum you can lose is the last written 2KiB. The rest is on the flash and cannot be lost.

But any app which relies on this is bad and not portable.

Ask specific questions please.

Please, read this UBIFS piece of doc about proper apps behavior and expectations about the file-systems:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback

There are many docs in the Internet as well.

Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Am 06.02.11 17:03 schrieb(en) Artem Bityutskiy:

Dumb question: isn't it necessary to also run sync() after the fsync() - close() - rename() sequence, as to ensure that the directory entries are actually flushed to the flash? Or does a call to

	rename("configfile.tmp", "configfile");

*ensure* that after a crash during this call either the entry of the old file (configfile) or of the newly written contents (configfile.tmp) is accessible, independent of the moment when the crash happens?

Best, Albrecht.

Well, there was a huge debate about this. Rename does not ensure sync in many file-systems, and posix does not require this. But user-spaces relies on this and many people believe this should obviously be guaranteed. So ext4 and may be some other FSes adjusted accordingly. I wrote about this in UBIFS FAQ:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_exceptions

You'll find all the relevant links there.

Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Am 06.02.11 18:39 schrieb(en) Artem Bityutskiy:

Thanks a lot for the pointers!

Unfortunately, they are more more about ubi & ext4, but I am in particular interested in JFFS2's behaviour, so please excuse my questions, but it's a really critical point... Currently, I use the following sequence to write my configs to jffs2 (both NOR flash and mtd-ram):

open(scratch, O_RDWR | O_CREAT | O_TRUNC, perms);
write();
fsync();
close();
sync(); // #1
rename(scratch, realname);
sync(); // #2

As far as I understand the information you cited, this should be safe under all circumstances for jffs2. Is that assumption correct?

As calling sync() might be a lot more expensive than fsync() if other file systems are used simultaneously (I have a vfat fs on a CF card open), my question is now if I can safely omit either sync() #1 or #2 or both?

When reading, I first try realname, and if that fails scratch, as to cover the case that rename() was killed in mid-air so realname has already been removed from the directory, but scratch is still there. Can this case happen at all for jffs2?

Thanks in advance,
Albrecht.

Hi,

sorry for delay, I was very busy.

OK, but you do realize that proper portable applications should not rely on JFFS2-specific behavior.

Also, if this is crucial for you, you should not generally rely on the answer of a random mailing list guy like me, of course :-)

OK, if scratch existed, at this point you truncate it. The change goes to the JFFS2 write buffer. And the write-buffer is flushed by time-out, or if some other app wrote anything else to the file-system, of if someone issued sync.

So if power cut happens, you may either end up with intact "scratch" or with "scratch" truncated to zero.

If you do not want it, you have to first copy "scratch" to "scratch_saved" and 'fsync()' "scratch_saved".

OK, you write to "scratch". What is your flash? NAND, NOR? In case of typical 2KiB per page NAND, you can lose the last 0-8KiB of written data in the worst case. Not sure how much, depends, and I do not remember the details well enough.

OK, if you get a power cut now, you are safe, you'll have modified "scratch".

Not necessary to do, but does not hurt. 'fsync()' was enough.

The same, not needed. You fsync()'ed "scratch" already.

Now if you have a power cut, you'll either have "scratch" or "realname" with correct data.

You can just do "fsync(realname)" instead. In case of JFFS2 there is not big difference, but if you ever migrate to another FS, there will be big difference.

Not sure what is your definition of "safe". Step 1 does truncation, I've explained possible outcome of this above.

You do not need 'sync()' #1 at all. And 'sync()' #2 may be turned into 'fsync(realname)'.

I think this can happen, AFAIR, rename will just do the change and put it to the write-buffer, or you can have a power cut while doing re-name.

But you are not safe at the open(O_TRUNC) and write() stages.

I explained the steps of the reliable file (e.g., config file) update here:

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change

You can read about UBIFS write-buffer here (JFFS2 write-buffer is almost the same, but there is only one write-buffer in JFFS2):

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writebuffer

I mean, the concept is the same, we borrowed it from JFFS2.

Best Regards,
Artem Bityutskiy (Артём Битюцкий)