ext3-2.4-0.9.0
An update of the ext3 journalling filesystem for 2.4 kernels is available at
http://www.uow.edu.au/~andrewm/linux/ext3/
Patches are against 2.4.6-ac1 and 2.4.6.
Changes since 0.0.8 include:
- Multiplied the version numbering by ten to cater for bugfix releases against the 0.9.0 stream.
- The main thrust has been the removal of a number of changes in the core kernel which were required for to support the journalling of data. This has caused some duplication of core code within ext3, but it's not too bad.
- A number of cleanups and resyncs with latest ext2. (Thanks, Al).
- Reorganised and optimised ext3_write_inode() and the handling of files which were opened O_SYNC.
- Move quota operations outside lock_super() - fixes last known source of quota deadlocks in -ac kernels.
- Deleted large chunks of debug/development support code.
- Improved handling of corner-case errors.
- Improved robustness in out-of-memory situations.
The last change is probably the most significant - it prevents possible crashes and fs corruption under extreme workloads.
-
Quoted 19 lines · Neil Brown
Quoted 6 lines · On Saturday July 7, [email protected] wrote:
An update of the ext3 journalling filesystem for 2.4 kernels is available at
http://www.uow.edu.au/~andrewm/linux/ext3/Patches are against 2.4.6-ac1 and 2.4.6.
I thought it was time to try out ext3 between nfsd and raid5, so I built 2.4.6 plus this patch, and an ext3 filesystem on a largish raid5 volume, exported it (with the "sync" flag), mounted it from another machines with NFSv2, and ran "dbench 4".
This produces a live-lock (I think that it the right term). Throughput would drop to zero (determined by watching the counts in
/proc/nfs/rpc/nfsd), but could be coaxed along by generating other filesystem activity.
Quoted 7 lines
I tried nfs over ext3 on a plain ide disc and it worked fine. I tried dbench directly on ext3/raid5 and it worked fine. I tried dbench/nfs/ext2/raid5 and it worked fine.
So I think it is some interaction between ext3fs and raid5 triggered by the high rate of "fsync" calls made by nfsd. Naturally I blame ext3 because I know more about raid5 and nfsd :-)
fsync will cause ext3 to commit the current transaction once all handles against it close - so that will produce rapid bursts of small numbers of writes.
Quoted 6 lines
One particular aspect of raid5 that *could* be related is that it is very reticent to schedule write requests. It tries to hang on the them as long as possible in the hope of getting more write requests in the same stripe. My guess as to what is happening is that as write request is submitted and then waited-for without an intervening
run_task_queue(&tq_disk);
Could well be. ext3 will happily feed 2,000 buffers into submit_bh() prior to running tq_disk. Everything else is happy with this, so I blame nfsd and raid5 :) Rapid fsyncs will break this up, however.
Does this patch help?
jbd/commit.c +2 -0
@@ -202,6 +202,7 @@spin_unlock (&journal_datalist_lock);unlock_journal (journal);ll_rw_block (WRITE , bufs, wbuf); +run_task_queue (&tq_disk);lock_journal (journal);journal_brelse_array (wbuf, bufs); goto write_out_data;
@@ -410,6 +411,7 @@ bh->b_end_io = end_buffer_io_sync;submit_bh (WRITE , bh); } +run_task_queue (&tq_disk);lock_journal (journal);/* Force a new descriptor to be generated next
Quoted 4 lines
When the system is livelocked, all I can tell at the moment (I am at home and the console is at work so I cannot use alt-sysrq) is that kjournal is waiting in
wait_on_bufferand an nfsd thread is waiting on the journal.
That sounds like Something Wierd is going on. wait_on_buffer will unplug and the disks should be going hell-for-leather.
Quoted 4 lines
I will try to explore it more deeply next time I am at work, but if there are any suggestions as to what it might be, or how I might more easily find out what is going on, I am all ears.
I'll see if I can get it to happen here. Thanks.
-