JFFS2 is broken

For all practical purposes, JFFS2, in its present form, IMHO, is broken.

I've been doing a lot of "jitter" or "blocking" time testing for various tasks running on a system where there is JFFS2 activity going on (info for those that have not been following my posts).

Here are the results:

Task interacting with JFFS2 fs directly. JFFS2 compression enabled. (the latest code in CVS):

Worst case jitter on a POSIX real time task interacting with JFFS2~>30*seconds*

POSIX RT Tast NOT directly interacting with JFFS2. JFFS2 compression enabled, but another task reading/writing to JFFS2 system.

Worst case jitter on *task NOT interacting with JFFS2* ~>30 seconds! (same for task interacting with JFFS2).

Ok, so I turned compression off (hacked the code. There is no option to do this).

Worst case jitter on task interacting with JFFS2, ~>4 seconds! Quite am improvement!

Worst case jitter on task NOT interacting with JFFS2, ~>4seconds! :(

So, in other words, if you use JFFS2 in your embedded system, you cannot expect a guranteed response to anything in less than 30 seconds if you use the stock code.
If you turn compression off, that time is ~4 seconds.

Note that these times are HIGHLY system speed dependent. My test system is a AMD SC520 (486 DX4 w/16MB L1 cache) @133MHz w/ 64MB 66MHz SDRAM. (~61 VAX MIPS). 8MB of AMD flash connected 32 bits wide.

The problem is that JFFS2 tries to be a good guy and tries its hand at GC'ing dirty flash, _from within a write() system call_

Now, I don't know if this can be made schedulable or not, but at this time, *all other* activity in the system stops.
When the GC is complete, life resumes as before, but more than 30-40 seconds may have elapsed.

To test my hypothesis, I hacked the code, to refuse to try to GC from within a write() to the JFFS2 fs. all GC is now done by the gc thread (as it should).
In the compression turned off case, my block times for the task not interacting with JFFS2 WENT DOWN TO 49.9 *ms* worst case, with the test going
from an empty JFFS2 to a completely full JFFS2 fs (as in all cases above).

Unfortunately, there is a problem with this approach. If write() cannot find space and now we refuse to GC inside the write and return with -ENOSPC, a lot of stock programs may break. I am returning -ENSPC because I just didn't take the time to figure out how to return 0, which

IMHO is the right thing to do.

Under POSIX write() can return 0, and it not be an error. The system is not ready for the write yet- exactly as in our case. However, I think stock programs will break with this too.

The only solution, that I think will work, is to find a way to block the write() to JFFS2 but allow kernel schedduling to go on. I really don't know
if this is possible under Linux as it exists today, maybe someone else can answer this question.

Comments welcome

Vipin

This is completely wrong. There is no excuse for the compression code to monopolize the CPU that way. This, of course, might be solved by the patch that makes the kernel preemptive. You could try the patch from ftp://ftp.mvista.com/pub/Area51/preemptible_kernel/ and see the difference. Of course the compression will still take a significant amount of CPU time, but the rest of the system won't be starved. Without the preemptive kernel patch, the code executing in kernel mode is following the cooperative model i.e. you must give up the CPU volontarily after a while. So the alternative to the preemptive kernel would be something like inserting this construct within the inner loop in the compression code:

	if (current->need_resched) schedule();

This should solve the problem with all other activities stalling while compression is going on.

In fact, if you're not using aio's, you can't expect any predictable delay for a write operation. Even on floppies a write may take several seconds to complete. If you really want your process to keep running even while the write is going on then dispatch the write operation to a separate thread. HOWEVER the fact that all system activities are stopped while compression is going on is actually a bug and should be solved by the introduction of the if(...) schedule() above into the compression loop at strategic places.

Not only it's possible, but mandatory for long operations like I said above. You can try inserting the magic line in the compression code. For instance it shouldn't hurt even if you add too much or if you don't put it at the best places, as long as schedule() isn't called from interrupt handlers or bottom halves which the JFFS2 compression code doesn't have anyway. This will certainly give you a different behavior.

Nicolas

That in itself is probably fine, write should only hold the inode lock for that file

One thing you can do here is to wake the gc thread and sleep politely on it

Well there are two things there.

1. You could wake the GC and sleep on it using sleep/wakeup or
semaphores

2. Profile the kernel and find out where it is tight looping. I can't
see any reason for tight loops except for the compression itself so it suggests a code bug.

Finally within the compression loops you can check current->need_resched and if it is set call schedule() to allow the compression to switch to other tasks at the point it has used its time slice.

Alan

If David is already looking at this, maybe I'll wait for a few days before bumbling through the code myself ;) David? You've been awfully quite!

Vipin

Just as a follow up to this last email, I just confirmed the results with my "I_refuse_to_do_a_GC_from_within_a_write()" hack test *with* compression enabled.

I get the same results: namely, max jitter on a task NOT directly interacting with the JFFS2 fs is ~50ms worst case, with the JFFS2 going from
empty to full in the background (another task is filling it up) (vs. >40 secs w/o the hack).

So ,this confirms that the excessive blocking time is somewhere inside the
function: "jffs2_garbage_collect_pass(c)"

Here is the trivial hack that I used to "refuse_to_gc_from_within_a_write()"
(Note: This is against the patched nodemgmt.c with the patch that David sent me. Not against the code in CVS).

Vipin

@@ -116,6 +116,17 @@
                        int ret;
 
                        up(&c->alloc_sem);
+
+                       /* Try to see what happens if we refuse to do GC

when we have been
+ requested to do just a simple write().
+ This is to test if our blocking times on "other"
tasks (that
+ are not interacting with the fs) are improved.
-Vipin 06/28/2001
+ */
+ printk("jffs2_reserve_space(): Refusing to GC! ret
-ENOSPC\n");
+
+ spin_unlock_bh(&c->erase_completion_lock);
+ return -ENOSPC;
+
if (c->dirty_size < c->sector_size) {
D1(printk(KERN_DEBUG "Short on space, but
total dirty size 0x%08x < sector size 0x%08x, so -ENOSPC\n", c->dirty_size,

c->sector_size));
                                spin_unlock_bh(&c->erase_completion_lock);

I've tried to keep the GC thread optional. That's not set in stone - but we ought to be able to fix this - there's no real reason why the GC in the write() path should behave like this.

Thinks... is the BKL still held in write()? Should we be releasing it?

AFAIR the compression routines aren't turning up on the profiles. But then we're getting 71 timer ticks in a 30-second period, according to the profiles I saw.

We already do this between each node we move, in the garbage collection loop. Doing it inside the compression loops would be possible - although it'd confuse mkfs.jffs2, which just imports those files directly too.

Better still to avoid decompressing and recompressing data when we move nodes intact.

I'll poke at this on Monday.

--
dwmw2

Is there any more word on this problem? I'm planning on porting jffs2 to the Agenda VR3, but have been holding off for a resolution of this issue... By the way, has anybody else noticed that there has been no traffic on this list in at least a week?

-Snipped, but boils down to kernel blocks for up to 30 seconds-

Craig West         Ph: (416) 213-0300	|  It's not a bug,
[email protected]          	|  It's a feature...

Yes,

As it stands right now, a task (not reading/writing to JFFS2) marked POSIX real time (via the set_sched(2) calls), is blocked for ~50ms on my 133MHz 486 DX4 while another _non posix_ task is writing to JFFS2.

The blocking time is highly processor performance dependent.

However, any task writing to JFFS2 _can_ be blocked upto 10's of seconds (in my test case >40 seconds).
This happens when there is no more free space left on the JFFS2 fs and garbage collect needs to be done to make space for the write.

If a POSIX task is writing to JFFS2, then that AND ALL OTHER tasks will get blocked as the POSIX task has priority and the GC is done in the context of the writing task.

So the moral of the story is that at the very least do not write to JFFS2 from a POSIX real time task under Linux.

I really haven't done any tests re. blocking times of a _regular_ task not writing to JFFS2, when there is another _regular_ task writing to JFFS2.

That should be easy to setup and run with my jitter test program. At this time I haven't released that program under GPL, but I have permission from my company to release it, so if anyone wants it just ask and I'll send it to you/put it in th mtd CVS rep.

Re. why it takes that long for GC to happen on a full JFFS2 fs? David had some thoughts that a particular singly linked list should be doubly linked- which would help quite a lot. David?

Vipin
P.S. There's been lots of traffic on the MTD list. Are you referring to the JFFS list?