JFFS2 as transactional FS (in other words: how to be sure that data have been writtent correctly from userspace)

Hi all MTD developers,

we are currently using an MTD partition on a NAND device, of course with JFFS2 on it ;-) , for transaction logging purpose.
This transacion is mission critical and we cannot afford to lose data (or, even worse, have corrupted data!)

For this reason we also use a battery-backed SRAM as temporary storage for the transaction state machine. After the transacion has been completed we flush the content of the SRAM to a file and (after the written is completed) we can overwrite the temporary storage with new data. Of course the machine can be interrupted in any moment without notice (e.g. watchdog, power failure). Only the content of the SRAM is guaranteed to be valid at any time.

The "main" problem, of course, is to know "when" we can say "ok the data has been _completely_ written to the final storage".

By reading back on this mailing list, "goooogling" on internet and reading JFFS2 FAQ
(http://www.linux-mtd.infradead.org/faq/jffs2.html#L_writewell) I think I have found some kind of solution (I'm currently running some test on it) depending on the storage medium (NOR vs NAND):

- on *NOR*: in our understanding, we can just use a simple fwrite() followed by fsync() or sync(). After the sync() return the control to the user's program, we can be sure that the data has been written on the device. So

file = fopen(file_on_jffs2_nor)

while(isneeded) {
	while (space_available(SRAM)) {
		fill(SRAM);
	}

	fread(buffer, SRAM);
	fwrite(buffer, file);
	fsync(file);
	invalidate_SDRAM();
}

fclose(file)

(Of course I have intetionally omitted the code for resuming from a warm reset.)

QUESTION: Is this pseudo code correct? Is fsync() needed? (O_SYNC is not supported by JFFS2, AFAIK) or data has been _completely_ written right before the fwrite() return (so no sync() required)?

- on *NAND*: things are a bit tricky ;-). Even if you call fsync() data may not have been written to storage, due the fact that "it's better to fill a NAND page before commit"
For this reason only after "a while" the (dirty) page is written to storage even if it's not full. In the FAQ you say that this "a while" is controlled by the standard kernel vm functions by setting /proc/sys/vm/dirty_writeback_centisecs.

By reading this I think about use this code:

at system startup:
echo smallvalue > /proc/sys/vm/dirty_writeback_centisecs

file = fopen(file_on_jffs2_nand)

while(isneeded) {
	while (space_available(SRAM)) {
		fill(SRAM);
	}

	fread(buffer, SRAM);
	fwrite(buffer, file);
	fsync(file);
	sleep(smallvalue+anothervalue)
	invalidate_SDRAM();
}

fclose(file)

'smallvalue' should be something less that the standard 5 secs but something that will not waste to much CPU or NAND storage (by using not-completely-filled pages, correct me if I'm wrong about this point). I was thinking about 500 millis.

'anothervalue' should be something '>>smallvalue' and it should be used (IMHO) because Linux is not an RTOS, so timing are not tightly guaranteed.

Is this approach correct or the something better that can be done??

Of course you can still flush buffers and dirty pages but umounting the partition but.. this is too long for our needs

BTW I have seen, in my current test, that, without the sleep(), sometime my "last" data is not written correctly.

Hope this (long ;-) ) email can lead to a useful discussion about this problem! :-)

Best Regards,

Andrea

...

On NOR you don't need the sync(). At least, if you're using write() you don't need the sync. I make no claims about what glibc does with fwrite(), but I believe fsync() ought to be perfectly sufficient.

JFFS2 doesn't support O_SYNC because it's _already_ synchronous.

If you call fsync() an the data for the given file isn't actually written to the NAND before the system call returns, that's a very serious bug. We went to great lengths to ensure that fsync() works as it should. If you think this is misbehaving, please show JFFS2 debugging output demonstrating the error.

Your proposal of using 'sleep()' really ought to fill you with dread. Adding an extra sleep is almost _never_ the way to achieve reliable operation. I hope you did that only to draw attention to the problem and weren't _honestly_ considering it in production :)

dwmw2

It used to be. And then came wbuf.c. With write buffer enabled, JFFS2 is still serialized, but no longer synchronous (in the meaning of having an implicit sync() after each operations).

Last time I checked, JFFS2 got one case wrong. Don't remember which one, but there are not that many to check:
- mount -o sync
- open(..., O_SYNC);
- fsync()
- sync()

[ Side note - NOR flash can still have a write buffer and require explicit sync. ]

Jörn

Prosperity makes friends, adversity tries them.
-- Publilius Syrus

I don't think we've implemented those two at all.

These two should work.

True. For 'NOR' read 'flash without wbuf' and for 'NAND' read 'flash with wbuf'.

dwmw2

Then you have your bug report. Ignoring those was ok in 2001, but with wbuf I would at least expect JFFS2 to return an error.

One option for implementing this would be to do the write, call schedule to let other processes write something, then check whether the buffer has been flushed out by other processes. Or do some GC to fill the buffer with useful data. Anything but padding it.

Jörn

The strong give up and move away, while the weak give up and stay.
-- unknown

Not quite, because that's not what he was trying. He said he was using fsync().

I would hope that JFFS2 doesn't _have_ to return an error -- the VFS should do so if the filesystem doesn't support a given option. That doesn't seem to be the case though, so I agree that we should make JFFS2 check for those two flags and refuse the operation (mount/open) for write-buffered flash.

Better just to say 'no', I think :)

dwmw2

Wait... why? Rejecting mount -o sync seems sane, but why can't O_SYNC support be handled? If users want to open their files like that and it causes JFFS2 to flush a bunch of padding out, well they asked for it.

josh

Users ask for shared writable mmap too. That isn't a good idea either.

Contemplate what Linus always says about using 'volatile' vs. proper barriers. Now consider it for O_SYNC vs. fsync(). And factor in the fact that O_SYNC is going to be massively suboptimal if it causes syncs when they otherwise didn't need to happen.

dwmw2

Ok, good point. You've convinced me.

josh

I can imagine cases when O_SYNC makes sense. Logfiles, basically. When debugging it is useful to have as much information in them as possible, especially right before the crash. O_SYNC is a useful relaxation here on system where fsync() is identical to sync() - O_SYNC will not flush every other file in the system.

Whether any applications actually do this is anyone's guess, though.

Jörn

Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface.
-- Doug MacIlroy