Journal guided resync and support
Subsystems (3)
cover Journal guided resync and support
This is an updated implementation of journal guided resync, intended to be suitable for production systems. This feature addresses the problem with RAID arrays that take too long to resync - similar to the existing MD write-intent bitmap feature, we resync only the stripes that were undergoing writes at the time of the crash. Unlike write-intent bitmaps, our testing shows very little performance degredation as a result of the feature - around 3-5% vs around 30% for bitmaps.
This feature is based on work described in this paper:
http://www.usenix.org/events/fast05/tech/denehy.html
As a summary, we introduce a new data write mode known as declared mode. This is based on ordered mode except that a list of blocks to be written during the current transaction is added to the journal before the blocks themselves are written to the disk. Then, if the system crashes, we can resync only those blocks during journal replay and skip the rest of the resync of the RAID array.
The changes consist of patches to ext3, jbd, MD, and the raid456 personality. These patches are currently against the RHEL 5 kernel 2.6.18-128.7.1. Porting to ext4/jbd2 and a more modern kernel is a TODO item.
Changes since the previous set of patches: I have addressed all review comments received. Noteable is a design change based on Neil Brown's suggestions: the filesystem now sets a buffer flag (fs_raidsync) to inform MD that the filesystem is taking responsibility for resyncing parity on this stripe in the event of a system crash. For RAID 4/5/6, setting this flag causes the write intent bitmap NOT to be updated for the write in question. There is also a buffer flag (syncraid) used by jbd to resync parity. Together these eliminate most of the need for ioctls, though one is still needed for e2fsck.
Unfortunately, we have determined that these patches are NOT useful to Lustre. Therefore I will not be doing any more work on them. I am sending them now in case they are useful as a starting point for someone else's work.
Cheers,
Jody
PATCH 1/5 Add fs_raidsync buffer & bio flags
Adds the BIO_FS_RAIDSYNC bio flag and the fs_raidsync buffer flag. These are used to inform md that the filesystem takes responsibility for resynchronizing parity information after a system crash.
When this flag is set on a raid4/5/6 device, the write will not be recorded in the bitmap.
drivers/md/raid5.c +26 -9
@@ -1465,7 +1465,8 @@ static int add_stripe_bio(struct stripe_ (unsigned long long )bi->bi_sector, (unsigned long long)sh->sector, dd_idx); -if (conf->mddev->bitmap && firstwrite) { +if (conf->mddev->bitmap && firstwrite && + !bio_flagged (bi,BIO_FS_RAIDSYNC )) {bitmap_startwrite (conf->mddev->bitmap, sh->sector,STRIPE_SECTORS ,0 ); sh->bm_seq = conf->seq_flush+1 ;
@@ -1643,7 +1644,11 @@ static void handle_stripe5(struct stripe/* fail all writes first */ bi = sh->dev[i].towrite; sh->dev[i].towrite =NULL ; -if (bi) { to_write--; bitmap_end =1 ; } +if (bi) { + to_write--; +if (!bio_flagged (bi,BIO_FS_RAIDSYNC )) + bitmap_end =1 ; + }if (test_and_clear_bit (R5_Overlap, &sh->dev[i].flags))wake_up (&conf->wait_for_overlap);
@@ -1661,7 +1666,8 @@ static void handle_stripe5(struct stripe/* and fail all 'written' */ bi = sh->dev[i].written; sh->dev[i].written =NULL ; -if (bi) bitmap_end =1 ; +if (bi && !bio_flagged (bi,BIO_FS_RAIDSYNC )) + bitmap_end =1 ;while (bi && bi->bi_sector < sh->dev[i].sector +STRIPE_SECTORS ) {struct bio *bi2 =r5_next_bio (bi, sh->dev[i].sector);clear_bit (BIO_UPTODATE , &bi->bi_flags);
@@ -1736,11 +1742,14 @@ static void handle_stripe5(struct stripemd_write_end (conf->mddev); wbi->bi_next = return_bi; return_bi = wbi; +if (!bio_flagged (wbi, +BIO_FS_RAIDSYNC )) + bitmap_end =1 ; } wbi = wbi2; } -if (dev->towrite ==NULL ) - bitmap_end =1 ; +if (dev->towrite !=NULL ) + bitmap_end =0 ;spin_unlock_irq (&conf->device_lock);if (bitmap_end) bitmap_endwrite(conf->mddev->bitmap, sh->sector,
@@ -2220,7 +2229,11 @@ static void handle_stripe6(struct stripe/* fail all writes first */ bi = sh->dev[i].towrite; sh->dev[i].towrite =NULL ; -if (bi) { to_write--; bitmap_end =1 ; } +if (bi) { + to_write--; +if (!bio_flagged (bi,BIO_FS_RAIDSYNC )) + bitmap_end =1 ; + }if (test_and_clear_bit (R5_Overlap, &sh->dev[i].flags))wake_up (&conf->wait_for_overlap);
@@ -2238,7 +2251,8 @@ static void handle_stripe6(struct stripe/* and fail all 'written' */ bi = sh->dev[i].written; sh->dev[i].written =NULL ; -if (bi) bitmap_end =1 ; +if (bi && !bio_flagged (bi,BIO_FS_RAIDSYNC )) + bitmap_end =1 ;while (bi && bi->bi_sector < sh->dev[i].sector +STRIPE_SECTORS ) {struct bio *bi2 =r5_next_bio (bi, sh->dev[i].sector);clear_bit (BIO_UPTODATE , &bi->bi_flags);
@@ -2324,11 +2338,14 @@ static void handle_stripe6(struct stripemd_write_end (conf->mddev); wbi->bi_next = return_bi; return_bi = wbi; +if (!bio_flagged (wbi, +BIO_FS_RAIDSYNC )) + bitmap_end =1 ; } wbi = wbi2; } -if (dev->towrite ==NULL ) - bitmap_end =1 ; +if (dev->towrite !=NULL ) + bitmap_end =0 ;spin_unlock_irq (&conf->device_lock);if (bitmap_end) bitmap_endwrite(conf->mddev->bitmap, sh->sector,
fs/buffer.c +3 -0
@@ -2859,6 +2859,9 @@ int submit_bh(int rw, struct buffer_head bio->bi_end_io = end_bio_bh_io_sync; bio->bi_private = bh; +if (buffer_fs_raidsync (bh)) +set_bit (BIO_FS_RAIDSYNC , &bio->bi_flags); +bio_get (bio);submit_bio (rw, bio);
include/linux/bio.h +1 -0
@@ -124,6 +124,7 @@ struct bio {#define BIO_BOUNCED 5/* bio is a bounce bio */ #define BIO_USER_MAPPED 6/* contains user pages */ #define BIO_EOPNOTSUPP 7/* not supported */ +#define BIO_FS_RAIDSYNC 8/* fs is responsible for RAID parity resync */ #define bio_flagged (bio, flag) ((bio)->bi_flags & (1 << (flag)))/*
include/linux/buffer_head.h +2 -0
@@ -32,6 +32,7 @@ enum bh_state_bits { BH_Write_EIO,/* I/O error on write */ BH_Ordered,/* ordered write */ BH_Eopnotsupp,/* operation not supported (barrier) */ + BH_FS_Raidsync,/* FS is responsible for RAID parity resyncs */ BH_PrivateStart,/* not a state bit, but the first bit available* for private allocation by other entities */
@@ -124,6 +125,7 @@ BUFFER_FNS(Write_EIO, write_io_error) BUFFER_FNS(Ordered, ordered) BUFFER_FNS(Eopnotsupp, eopnotsupp) BUFFER_FNS(Unwritten, unwritten) +BUFFER_FNS(FS_Raidsync, fs_raidsync)#define bh_offset (bh) ((unsigned long)(bh)->b_data & ~PAGE_MASK)#define touch_buffer (bh) mark_page_accessed(bh->b_page)
PATCH 2/5 Add syncraid buffer & bio flags
In reply to [email protected]Adds the BIO_SYNCRAID bio flag and the syncraid buffer flag. These are used to request a resync of the stripe containing the data being read or written.
drivers/md/raid5.c +25 -3
@@ -1704,7 +1704,8 @@ static void handle_stripe5(struct stripe } }if (failed >1 && syncing) { -md_done_sync (conf->mddev,STRIPE_SECTORS ,0 ); +if (!test_bit (STRIPE_RESYNC_REQ , &sh->state)) +md_done_sync (conf->mddev,STRIPE_SECTORS ,0 );clear_bit (STRIPE_SYNCING , &sh->state); syncing =0 ; }
@@ -1941,7 +1942,8 @@ static void handle_stripe5(struct stripe } }if (syncing && locked ==0 &&test_bit (STRIPE_INSYNC , &sh->state)) { -md_done_sync (conf->mddev,STRIPE_SECTORS ,1 ); +if (!test_bit (STRIPE_RESYNC_REQ , &sh->state)) +md_done_sync (conf->mddev,STRIPE_SECTORS ,1 );clear_bit (STRIPE_SYNCING , &sh->state); }
@@ -2289,7 +2291,8 @@ static void handle_stripe6(struct stripe } }if (failed >2 && syncing) { -md_done_sync (conf->mddev,STRIPE_SECTORS ,0 ); +if (!test_bit (STRIPE_RESYNC_REQ , &sh->state)) +md_done_sync (conf->mddev,STRIPE_SECTORS ,0 );clear_bit (STRIPE_SYNCING , &sh->state); syncing =0 ; }
@@ -2589,6 +2592,8 @@ static void handle_stripe6(struct stripeif (syncing && locked ==0 &&test_bit (STRIPE_INSYNC , &sh->state)) {md_done_sync (conf->mddev,STRIPE_SECTORS ,1 ); +if (!test_bit (STRIPE_RESYNC_REQ , &sh->state)) +md_done_sync (conf->mddev,STRIPE_SECTORS ,1 );clear_bit (STRIPE_SYNCING , &sh->state); }
@@ -2880,6 +2885,9 @@ static int make_request(request_queue_telse atomic_inc (&conf->reads_in); +PRINTK ("make_request on %llu, %s\n" , bi->bi_sector, + ((rw ==WRITE ) ?"write" :"read" )); + logical_sector = bi->bi_sector & ~((sector_t )STRIPE_SECTORS -1 ); last_sector = bi->bi_sector + (bi->bi_size>>9 ); bi->bi_next =NULL ;
@@ -3006,6 +3014,19 @@ retry: r_sector += sectors_per_chunk; }if (sh) { +if (bio_flagged (bi,BIO_SYNCRAID )) { +if (test_bit (STRIPE_RESYNC_REQ , &sh->state)) { +PRINTK ("SYNCRAID on %llu, already " +"resynced\n" , bi->bi_sector); + }else { +PRINTK ("SYNCRAID on %llu, " +"resyncing\n" , bi->bi_sector); +set_bit (STRIPE_SYNCING , &sh->state); +set_bit (STRIPE_RESYNC_REQ , &sh->state); +clear_bit (STRIPE_INSYNC , &sh->state); + } + } +handle_stripe (sh,NULL , bios);release_stripe (sh); sh =NULL ;
@@ -3234,6 +3255,7 @@ static inline sector_t sync_request(mddespin_lock (&sh->lock);set_bit (STRIPE_SYNCING , &sh->state);clear_bit (STRIPE_INSYNC , &sh->state); +clear_bit (STRIPE_RESYNC_REQ , &sh->state);spin_unlock (&sh->lock);handle_stripe (sh,NULL ,NULL );
fs/buffer.c +2 -0
@@ -2861,6 +2861,8 @@ int submit_bh(int rw, struct buffer_headif (buffer_fs_raidsync (bh))set_bit (BIO_FS_RAIDSYNC , &bio->bi_flags); +if (buffer_syncraid (bh)) +set_bit (BIO_SYNCRAID , &bio->bi_flags);bio_get (bio);submit_bio (rw, bio);
include/linux/bio.h +1 -0
@@ -125,6 +125,7 @@ struct bio {#define BIO_USER_MAPPED 6/* contains user pages */ #define BIO_EOPNOTSUPP 7/* not supported */ #define BIO_FS_RAIDSYNC 8/* fs is responsible for RAID parity resync */ +#define BIO_SYNCRAID 9/* perform RAID parity resync now */ #define bio_flagged (bio, flag) ((bio)->bi_flags & (1 << (flag)))/*
include/linux/buffer_head.h +2 -0
@@ -33,6 +33,7 @@ enum bh_state_bits { BH_Ordered,/* ordered write */ BH_Eopnotsupp,/* operation not supported (barrier) */ BH_FS_Raidsync,/* FS is responsible for RAID parity resyncs */ + BH_SyncRAID,/* Perform RAID parity resync now */ BH_PrivateStart,/* not a state bit, but the first bit available* for private allocation by other entities */
@@ -126,6 +127,7 @@ BUFFER_FNS(Ordered, ordered) BUFFER_FNS(Eopnotsupp, eopnotsupp) BUFFER_FNS(Unwritten, unwritten) BUFFER_FNS(FS_Raidsync, fs_raidsync) +BUFFER_FNS(SyncRAID, syncraid)#define bh_offset (bh) ((unsigned long)(bh)->b_data & ~PAGE_MASK)#define touch_buffer (bh) mark_page_accessed(bh->b_page)
include/linux/raid/raid5.h +2 -0
@@ -180,6 +180,8 @@ struct stripe_head {#define STRIPE_EXPANDING 9#define STRIPE_EXPAND_SOURCE 10#define STRIPE_EXPAND_READY 11 +#define STRIPE_RESYNC_REQ 12 +/* * Plugging: *
PATCH 3/5 Add support for journal guided resync.
In reply to [email protected]Adds support for declare blocks, used by ext3's journal guided resync (declared mode.) A declare block is added to the journal to list blocks to be written during the current transaction. During journal replay, we perform a RAID resync of only these blocks and skip the rest of the resync.
We also set the fs_raidsync flag on buffers being submitted when declared mode is active to inform MD that the filesystem is responsible for resyncing the stripe parity in the event of a system crash.
fs/jbd/checkpoint.c +2 -0
@@ -712,6 +712,8 @@ void __journal_drop_transaction(journal_J_ASSERT (transaction->t_state ==T_FINISHED );J_ASSERT (transaction->t_buffers ==NULL ); +J_ASSERT (transaction->t_declare_root.rnode ==NULL ); +J_ASSERT (transaction->t_declare_done_root.rnode ==NULL );J_ASSERT (transaction->t_sync_datalist ==NULL );J_ASSERT (transaction->t_forget ==NULL );J_ASSERT (transaction->t_iobuf_list ==NULL );
fs/jbd/commit.c +285 -52
389 lines not on this page · raw message
fs/jbd/journal.c +24 -0
@@ -156,6 +156,16 @@ loop:journal_commit_transaction (journal);spin_lock (&journal->j_state_lock); goto loop; + }else if (journal->j_flags &JFS_DECLARE && + (transaction == journal->j_running_transaction) && + transaction->t_declare_request) { +jbd_debug (2 ,"early declare\n" ); +spin_unlock (&journal->j_state_lock); +journal_write_declare_blocks (journal, transaction,0 ); +spin_lock (&journal->j_state_lock); + +wake_up (&journal->j_wait_declare); + goto loop; }wake_up (&journal->j_wait_done_commit);
@@ -299,6 +309,8 @@ int journal_write_metadata_buffer(transa*/ J_ASSERT_BH (bh_in,buffer_jbddirty (bh_in)); +set_buffer_fs_raidsync (bh_in); + new_bh =alloc_buffer_head (GFP_NOFS |__GFP_NOFAIL);/*
@@ -376,6 +388,9 @@ repeat: new_bh->b_blocknr = blocknr;set_buffer_mapped (new_bh);set_buffer_dirty (new_bh); +if (transaction->t_journal->j_flags &JFS_DECLARE && + transaction->t_journal->j_fs_dev == transaction->t_journal->j_dev) +set_buffer_fs_raidsync (new_bh); *jh_out = new_jh;
@@ -635,6 +650,9 @@ struct journal_head *journal_get_descriplock_buffer (bh);memset (bh->b_data,0 , journal->j_blocksize);set_buffer_uptodate (bh); +if (journal->j_flags &JFS_DECLARE && + journal->j_fs_dev == journal->j_dev) +set_buffer_fs_raidsync (bh);unlock_buffer (bh);BUFFER_TRACE (bh,"return this buffer" );return journal_add_journal_head (bh);
@@ -959,6 +977,7 @@ static journal_t * journal_init_common (init_waitqueue_head (&journal->j_wait_checkpoint);init_waitqueue_head (&journal->j_wait_commit);init_waitqueue_head (&journal->j_wait_updates); +init_waitqueue_head (&journal->j_wait_declare);mutex_init (&journal->j_barrier);mutex_init (&journal->j_checkpoint_mutex);spin_lock_init (&journal->j_revoke_lock);
@@ -1259,6 +1278,9 @@ void journal_update_superblock(journal_tBUFFER_TRACE (bh,"marking dirty" );mark_buffer_dirty (bh); +if (journal->j_flags &JFS_DECLARE && + journal->j_fs_dev == journal->j_dev) +set_buffer_fs_raidsync (bh);if (wait)sync_dirty_buffer (bh);else
@@ -1292,6 +1314,8 @@ static int journal_get_superblock(journaJ_ASSERT (bh !=NULL );if (!buffer_uptodate (bh)) { +set_buffer_syncraid (bh); +ll_rw_block (READ ,1 , &bh);wait_on_buffer (bh);if (!buffer_uptodate (bh)) {
fs/jbd/recovery.c +150 -10
274 lines not on this page · raw message
fs/jbd/transaction.c +66 -7
151 lines not on this page · raw message
include/linux/jbd.h +35 -2
@@ -26,6 +26,7 @@#include <linux/types.h> #include <linux/buffer_head.h> #include <linux/journal-head.h> +#include <linux/radix-tree.h> #include <linux/stddef.h> #include <linux/bit_spinlock.h> #include <linux/mutex.h>
@@ -137,6 +138,7 @@ typedef struct journal_s journal_t; /* J#define JFS_SUPERBLOCK_V1 3#define JFS_SUPERBLOCK_V2 4#define JFS_REVOKE_BLOCK 5 +#define JFS_DECLARE_BLOCK 6/* * Standard header for all descriptor blocks:
@@ -261,12 +263,14 @@ typedef struct journal_superblock_s#define JFS_FEATURE_INCOMPAT_REVOKE 0x00000001#define JFS_FEATURE_INCOMPAT_ASYNC_COMMIT 0x00000004 +#define JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS 0x00000008/* Features known to this kernel version: */ #define JFS_KNOWN_COMPAT_FEATURES JFS_FEATURE_COMPAT_CHECKSUM#define JFS_KNOWN_ROCOMPAT_FEATURES 0#define JFS_KNOWN_INCOMPAT_FEATURES (JFS_FEATURE_INCOMPAT_REVOKE | \ - JFS_FEATURE_INCOMPAT_ASYNC_COMMIT) + JFS_FEATURE_INCOMPAT_ASYNC_COMMIT | \ + JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS)#ifdef __KERNEL__
@@ -559,6 +563,15 @@ struct transaction_sstruct journal_head *t_sync_datalist;/* +* Radix tree of all data buffers that must be declared before being +* written, declare mode counters [j_list_lock] +*/ +struct radix_tree_root t_declare_root; +struct radix_tree_root t_declare_done_root; +unsigned int t_declare_count; +unsigned int t_declare_request; + +/* * Doubly-linked circular list of all forget buffers (superseded * buffers which we can un-checkpoint once this transaction commits) * [j_list_lock]
@@ -730,6 +743,7 @@ jbd_time_diff(unsigned int start, unsign* @j_wait_checkpoint: Wait queue to trigger checkpointing * @j_wait_commit: Wait queue to trigger commit * @j_wait_updates: Wait queue to wait for updates to complete +* @j_wait_declare: Wait queue to wait for declarations to complete * @j_checkpoint_mutex: Mutex for locking against concurrent checkpoints * @j_head: Journal head - identifies the first unused block in the journal * @j_tail: Journal tail - identifies the oldest still-used block in the
@@ -768,6 +782,8 @@ jbd_time_diff(unsigned int start, unsign* @j_wbufsize: maximum number of buffer_heads allowed in j_wbuf, the * number that will fit in j_blocksize * @j_last_sync_writer: most recent pid which did a synchronous write +* @j_declare_jhs: array of journal_heads for journal_write_declare_blocks +* @j_declare_bhs: array of buffer_heads for journal_write_declare_blocks * @j_private: An opaque pointer to fs-private information. */
@@ -841,6 +857,9 @@ struct journal_s/* Wait queue to wait for updates to complete */ wait_queue_head_t j_wait_updates; +/* Wait queue to wait for declarations to complete */ +wait_queue_head_t j_wait_declare; +/* Semaphore for locking against concurrent checkpoints */ struct mutex j_checkpoint_mutex;
@@ -970,6 +989,13 @@ struct journal_sstruct transaction_stats_s j_stats;/* +* Arrays of jhs and bhs for journal_write_declare_blocks, to avoid +* having to allocate them each time. +*/ +void *j_declare_jhs[64 ]; +struct buffer_head *j_declare_bhs[64 ]; + +/* * An opaque pointer to fs-private information. ext3 puts its * superblock pointer here */
@@ -985,6 +1011,7 @@ struct journal_s#define JFS_FLUSHED 0x008/* The journal superblock has been flushed */ #define JFS_LOADED 0x010/* The journal superblock has been loaded */ #define JFS_BARRIER 0x020/* Use IDE barriers */ +#define JFS_DECLARE 0x040/* Declare data blocks before writing */ /* * Function declarations for the journaling transaction and buffer
@@ -1008,6 +1035,9 @@ int journal_next_log_block(journal_t *,/* Commit management */ extern void journal_commit_transaction (journal_t *); +extern void journal_write_declare_blocks (journal_t *journal, +transaction_t *commit_transaction, +int committing);/* Checkpoint list management */ int __journal_clean_checkpoint_list (journal_t *journal);
@@ -1100,6 +1130,7 @@ extern void journal_ack_err (journextern int journal_clear_err (journal_t *);extern int journal_bmap (journal_t *,unsigned long ,unsigned long *);extern int journal_force_commit (journal_t *); +extern int journal_write_declare (journal_t *);/* * journal_head management
@@ -1244,7 +1275,9 @@ static inline int jbd_space_needed(journ#define BJ_LogCtl 6/* Buffer contains log descriptors */ #define BJ_Reserved 7/* Buffer is reserved for access by journal */ #define BJ_Locked 8/* Locked for I/O during commit */ -#define BJ_Types 9 +#define BJ_Declare 9/* Needs to be declared first */ +#define BJ_DeclareDone 10/* Has been declared */ +#define BJ_Types 11extern int jbd_blocks_per_page (struct inode *inode);
PATCH 4/5 Add journal guided resync (data=declared mode)
In reply to [email protected]We introduce a new data write mode known as declared mode. This is based on ordered mode except that a list of blocks to be written during the current transaction is added to the journal before the blocks themselves are written to the disk. Then, if the system crashes, we can resync only those blocks during journal replay and skip the rest of the resync of the RAID array.
fs/ext3/file.c +2 -1
@@ -78,7 +78,8 @@ ext3_file_write(struct kiocb *iocb, cons* Open question --- do we care about flushing timestamps too * if the inode is IS_SYNC? */ -if (!ext3_should_journal_data (inode)) +if (!ext3_should_journal_data (inode) && + !ext3_should_declare_data (inode))return ret; goto force_commit;
fs/ext3/fsync.c +6 -1
@@ -66,8 +66,13 @@ int ext3_sync_file(struct file * file, s* filemap_fdatawait() will encounter a ton of newly-dirtied pages * (they were dirtied by commit). But that's OK - the blocks are * safe in-journal, which is all fsync() needs to ensure. +* +* data=declared: +* Declare blocks are written before data blocks, then the +* sync proceeds as for data=ordered. */ -if (ext3_should_journal_data (inode)) { +if (ext3_should_journal_data (inode) || +ext3_should_declare_data (inode)) { ret =ext3_force_commit (inode->i_sb); goto out; }
fs/ext3/inode.c +145 -3
204 lines not on this page · raw message
fs/ext3/super.c +29 -5
@@ -391,6 +391,9 @@ static void ext3_put_super (struct superint i, err;ext3_xattr_put_super (sb); +journal_clear_features (sbi->s_journal,0 ,0 , +JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS ); +journal_update_superblock (sbi->s_journal,1 ); err =journal_destroy (sbi->s_journal); sbi->s_journal =NULL ;if (err <0 )
@@ -553,6 +556,8 @@ static int ext3_show_options(struct seq_seq_puts (seq,",data=ordered" ); else if (test_opt(sb, DATA_FLAGS) == EXT3_MOUNT_WRITEBACK_DATA) seq_puts(seq, ",data=writeback"); + else if (test_opt(sb, DATA_FLAGS) == EXT3_MOUNT_DECLARED_DATA) +seq_puts (seq,",data=declared" );ext3_show_quota_options (seq, sb);
@@ -682,7 +687,7 @@ enum { Opt_usrjquota, Opt_grpjquota, Opt_offusrjquota, Opt_offgrpjquota, Opt_jqfmt_vfsold, Opt_jqfmt_vfsv0, Opt_quota, Opt_noquota, Opt_ignore, Opt_barrier, Opt_err, Opt_resize, Opt_usrquota, - Opt_grpquota + Opt_grpquota, Opt_data_declared }; static match_table_t tokens = {
@@ -721,6 +726,7 @@ static match_table_t tokens = { {Opt_data_journal, "data=journal"}, {Opt_data_ordered, "data=ordered"}, {Opt_data_writeback, "data=writeback"}, + {Opt_data_declared, "data=declared"}, {Opt_offusrjquota, "usrjquota="}, {Opt_usrjquota, "usrjquota=%s"}, {Opt_offgrpjquota, "grpjquota="},
@@ -922,6 +928,9 @@ static int parse_options (char *options, goto datacheck;case Opt_data_writeback: data_opt =EXT3_MOUNT_WRITEBACK_DATA ; + goto datacheck; +case Opt_data_declared: + data_opt =EXT3_MOUNT_DECLARED_DATA ; datacheck:if (is_remount) { if ((sbi->s_mount_opt & EXT3_MOUNT_DATA_FLAGS)
@@ -1740,7 +1749,21 @@ static int ext3_fill_super (struct superelse set_opt (sbi->s_mount_opt,JOURNAL_DATA );break ; - +case EXT3_MOUNT_DECLARED_DATA : +if (!journal_check_available_features (sbi->s_journal,0 ,0 , +JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS )) { +printk (KERN_ERR "EXT3-fs: Journal does not support " +"declared data journaling mode\n" ); + goto failed_mount4; + } +spin_lock (&sbi->s_journal->j_state_lock); + sbi->s_journal->j_flags |=JFS_DECLARE ; +spin_unlock (&sbi->s_journal->j_state_lock); +if (!journal_set_features (sbi->s_journal,0 ,0 , +JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS )) { +printk (KERN_ERR "EXT3-fs: Cannot set declared mode.\n" ); + goto failed_mount4; + }case EXT3_MOUNT_ORDERED_DATA :case EXT3_MOUNT_WRITEBACK_DATA : if (!journal_check_available_features
@@ -1795,9 +1818,10 @@ static int ext3_fill_super (struct superprintk (KERN_INFO "EXT3-fs: recovery complete.\n" );ext3_mark_recovery_complete (sb, es);printk (KERN_INFO "EXT3-fs: mounted filesystem with %s data mode.\n" , -test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_JOURNAL_DATA ?"journal" : -test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_ORDERED_DATA ?"ordered" : -"writeback" ); +test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_JOURNAL_DATA ?"journal" : +test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_ORDERED_DATA ?"ordered" : +test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_DECLARED_DATA ? +"declared" :"writeback" );lock_kernel ();return 0 ;
include/linux/ext3_fs.h +3 -2
@@ -357,11 +357,11 @@ struct ext3_inode {#define EXT3_MOUNT_MINIX_DF 0x00080/* Mimics the Minix statfs */ #define EXT3_MOUNT_NOLOAD 0x00100/* Don't use existing journal*/ #define EXT3_MOUNT_ABORT 0x00200/* Fatal error detected */ -#define EXT3_MOUNT_DATA_FLAGS 0x00C00/* Mode for data writes: */ +#define EXT3_MOUNT_DATA_FLAGS 0x01C00/* Mode for data writes: */ #define EXT3_MOUNT_JOURNAL_DATA 0x00400/* Write data to journal */ #define EXT3_MOUNT_ORDERED_DATA 0x00800/* Flush data before commit */ #define EXT3_MOUNT_WRITEBACK_DATA 0x00C00/* No data ordering */ -#define EXT3_MOUNT_UPDATE_JOURNAL 0x01000/* Update the journal format */ +#define EXT3_MOUNT_DECLARED_DATA 0x01000/* Declare data blocks before writing */ #define EXT3_MOUNT_NO_UID32 0x02000/* Disable 32-bit UIDs */ #define EXT3_MOUNT_XATTR_USER 0x04000/* Extended user attributes */ #define EXT3_MOUNT_POSIX_ACL 0x08000/* POSIX Access Control Lists */
@@ -383,6 +383,7 @@ struct ext3_inode {#define EXT2_MOUNT_ABORT EXT3_MOUNT_ABORT#define EXT2_MOUNT_DATA_FLAGS EXT3_MOUNT_DATA_FLAGS#endif +#define EXT3_MOUNT_UPDATE_JOURNAL 0x40000000/* Update the journal format */ #define ext3_set_bit ext2_set_bit#define ext3_set_bit_atomic ext2_set_bit_atomic
include/linux/ext3_jbd.h +11 -0
@@ -265,4 +265,15 @@ static inline int ext3_should_writeback_ return 0; } +static inline int ext3_should_declare_data (struct inode *inode) +{ +if (!S_ISREG (inode->i_mode)) +return 0 ; +if (EXT3_I (inode)->i_flags &EXT3_JOURNAL_DATA_FL ) +return 0 ; +if (test_opt (inode->i_sb,DATA_FLAGS ) ==EXT3_MOUNT_DECLARED_DATA ) +return 1 ; +return 0 ; +} +#endif /* _LINUX_EXT3_JBD_H */
PATCH 5/5 Add SET_RESYNC_ALL and CLEAR_RESYNC_ALL ioctls.
In reply to [email protected]This patch adds SET_RESYNC_ALL and CLEAR_RESYNC_ALL ioctls, needed for fsck of a filesystem using declared mode.
SET_RESYNC_ALL instructs an MD block device to perform a resync of all stripes read or written until CLEAR_RESYNC_ALL is set. The intended user is e2fsck, which can SET_RESYNC_ALL before journal replay (which includes reading from all blocks declared for write prior to a crash) and then CLEAR_RESYNC_ALL afterwards.
drivers/md/md.c +18 -0
@@ -4484,6 +4484,24 @@ static int md_ioctl(struct inode *inode, err =set_bitmap_file (mddev, (int )arg); goto done_unlock; +case SET_RESYNC_ALL : +if (mddev->pers->set_resync_all ==NULL ) { + err = -EINVAL ; + goto abort_unlock; + } + + err = mddev->pers->set_resync_all (mddev,1 ); + goto done_unlock; + +case CLEAR_RESYNC_ALL : +if (mddev->pers->set_resync_all ==NULL ) { + err = -EINVAL ; + goto abort_unlock; + } + + err = mddev->pers->set_resync_all (mddev,0 ); + goto done_unlock; +default : err = -EINVAL ; goto abort_unlock;
drivers/md/raid5.c +13 -1
@@ -3013,7 +3013,7 @@ retry: r_sector += sectors_per_chunk; }if (sh) { -if (bio_flagged (bi,BIO_SYNCRAID )) { +if (bio_flagged (bi,BIO_SYNCRAID ) || conf->resync_all) {if (test_bit (STRIPE_RESYNC_REQ , &sh->state)) {PRINTK ("SYNCRAID on %llu, already resynced\n" , bi->bi_sector); }else {
@@ -4049,6 +4049,15 @@ static void raid5_quiesce(mddev_t *mddev } } +static int raid5_set_resync_all (mddev_t *mddev,int state) +{ +raid5_conf_t *conf =mddev_to_conf (mddev); + + conf->resync_all = state; + +return 0 ; +} +static struct mdk_personality raid6_personality = { .name ="raid6" ,
@@ -4065,6 +4074,7 @@ static struct mdk_personality raid6_pers .sync_request = sync_request, .resize = raid5_resize, .quiesce = raid5_quiesce, + .set_resync_all = raid5_set_resync_all, };static struct mdk_personality raid5_personality = {
@@ -4086,6 +4096,7 @@ static struct mdk_personality raid5_pers .start_reshape = raid5_start_reshape, #endif .quiesce = raid5_quiesce, + .set_resync_all = raid5_set_resync_all, }; static struct mdk_personality raid4_personality =
@@ -4104,6 +4115,7 @@ static struct mdk_personality raid4_pers .sync_request = sync_request, .resize = raid5_resize, .quiesce = raid5_quiesce, + .set_resync_all = raid5_set_resync_all, };static int __init raid5_init (void )
include/linux/raid/md_u.h +2 -0
@@ -45,6 +45,8 @@#define STOP_ARRAY _IO (MD_MAJOR, 0x32)#define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33)#define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34) +#define SET_RESYNC_ALL _IO (MD_MAJOR, 0x35) +#define CLEAR_RESYNC_ALL _IO (MD_MAJOR, 0x36) typedef struct mdu_version_s { int major;
include/linux/raid/raid5.h +1 -0
@@ -241,6 +241,7 @@ struct raid5_private_data {int seq_flush, seq_write;int quiesce; +int resync_all;int fullsync;/* set to 1 if a full sync is needed, * (fresh device added).
include/linux/raid/md_k.h +1 -0
@@ -283,6 +283,7 @@ struct mdk_personality* others - reserved */ void (*quiesce) (mddev_t *mddev,int state); +int (*set_resync_all) (mddev_t *mddev,int state); };
Hi!
Quoted 7 lines
This is an updated implementation of journal guided resync, intended to be suitable for production systems. This feature addresses the problem with RAID arrays that take too long to resync - similar to the existing MD write-intent bitmap feature, we resync only the stripes that were undergoing writes at the time of the crash. Unlike write-intent bitmaps, our testing shows very little performance degredation as a result of the feature - around 3-5% vs around 30% for bitmaps.
Good. Now when fs know about raid and wise versa... perhaps it is time to journal surrounding data on stripe so that power fails do not destroy data on degraded raid5?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 8 lines · Pavel Machek
Quoted 4 lines
This is an updated implementation of journal guided resync, intended to be suitable for production systems. This feature addresses the problem with RAID arrays that take too long to resync
- similar to the existing MD write-intent bitmap feature, we resync
only the stripes that were undergoing writes at the time of the crash. Unlike write-intent bitmaps, our testing shows very little performance degredation as a result of the feature - around 3-5% vs around 30% for bitmaps.Good. Now when fs know about raid and wise versa... perhaps it is time to journal surrounding data on stripe so that power fails do not destroy data on degraded raid5?
That's an interesting idea. I suspect this could be done more efficiently by only journaling the parity block update, but there is no way for the journal to address this block.
That said, unfortunately Jody has no more time to work on these patches (due to data ordering requirements Lustre can't use them), and while they are functionally complete for RHEL5 + ext3, they need to be ported to 2.6.current and ext4 by someone or they will die a silent death.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.