Journal guided resync and support
cover Journal guided resync and support
This is an updated implementation of journal guided resync, intended to be suitable for production systems. This feature addresses the problem with RAID arrays that take too long to resync - similar to the existing MD write-intent bitmap feature, we resync only the stripes that were undergoing writes at the time of the crash. Unlike write-intent bitmaps, our testing shows very little performance degredation as a result of the feature - around 3-5% vs around 30% for bitmaps.
This feature is based on work described in this paper:
http://www.usenix.org/events/fast05/tech/denehy.html
As a summary, we introduce a new data write mode known as declared mode. This is based on ordered mode except that a list of blocks to be written during the current transaction is added to the journal before the blocks themselves are written to the disk. Then, if the system crashes, we can resync only those blocks during journal replay and skip the rest of the resync of the RAID array.
The changes consist of patches to ext3, jbd, MD, and the raid456 personality. These patches are currently against the RHEL 5 kernel 2.6.18-128.1.6. Porting to ext4/jbd2 and a more modern kernel is a TODO item.
Cheers,
Jody
PATCH 1/4 Add SKIP_RESYNC ioctl
Add a SKIP_RESYNC ioctl to md allowing resync to be skipped on an MD device or partition.
Design note: I expect there to be one (unpartitioned MD device) or just a few (partitioned MD device) skip_list entries, therefore searching a linked list is not a huge concern.
drivers/md/md.c +29 -1
@@ -314,12 +314,13 @@ static inline int mddev_trylock(mddev_treturn mutex_trylock (&mddev->reconfig_mutex); } -static inline void mddev_unlock (mddev_t * mddev) +inline void mddev_unlock (mddev_t * mddev) {mutex_unlock (&mddev->reconfig_mutex);md_wakeup_thread (mddev->thread); } +EXPORT_SYMBOL_GPL (mddev_unlock);static mdk_rdev_t *find_rdev_nr (mddev_t *mddev,int nr) {
@@ -4484,6 +4485,33 @@ static int md_ioctl(struct inode *inode, err =set_bitmap_file (mddev, (int )arg); goto done_unlock; +case SKIP_RESYNC : + { +struct hd_struct *part = inode->i_bdev->bd_part; +sector_t start, end; + +if (mddev->pers ==NULL ) { + err = -ENODEV ; + goto abort_unlock; + } + +if (mddev->pers->skip_resync ==NULL ) { + err = -EINVAL ; + goto abort_unlock; + } + +if (part) { + start = part->start_sect; + end = part->start_sect + part->nr_sects -1 ; + }else { + start =0 ; + end = (mddev->array_size<<1 ) -1 ; + } + + err = mddev->pers->skip_resync (mddev, start, end); + goto done_unlock; + } +default : err = -EINVAL ; goto abort_unlock;
include/linux/raid/md_u.h +1 -0
@@ -45,6 +45,7 @@#define STOP_ARRAY _IO (MD_MAJOR, 0x32)#define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33)#define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34) +#define SKIP_RESYNC _IO (MD_MAJOR, 0x40) typedef struct mdu_version_s { int major;
include/linux/raid/md_k.h +1 -0
@@ -283,6 +283,7 @@ struct mdk_personality* others - reserved */ void (*quiesce) (mddev_t *mddev,int state); +int (*skip_resync) (mddev_t *mddev,sector_t start,sector_t end); };
drivers/md/raid5.c +79 -0
127 lines not on this page · raw message
include/linux/raid/raid5.h +8 -0
@@ -260,6 +260,7 @@ struct raid5_private_data {int pool_size;/* number of disks in stripeheads in pool */ spinlock_t device_lock;struct disk_info *disks; +struct list_head skip_list;/* used to skip resync on certain blocks */ /* * Stats
@@ -294,4 +295,11 @@ typedef struct raid5_private_data raid5_#define ALGORITHM_LEFT_SYMMETRIC 2#define ALGORITHM_RIGHT_SYMMETRIC 3 +struct skip_entry { +struct list_head skip_list; + +sector_t start; +sector_t end; +}; +#endif
include/linux/raid/md.h +2 -0
@@ -95,5 +95,7 @@ extern void md_new_event(mddev_t *mddev)extern void md_update_sb (mddev_t * mddev); +extern void mddev_unlock (mddev_t * mddev); +#endif
PATCH 2/4 Add RESYNC_RANGE ioctl
In reply to [email protected]Add the RESYNC_RANGE ioctl and implement it for RAID 4/5/6. This causes an immediate resync of the requested sectors if the device is under resync.
TODO: In raid456 (and probably in any other personality that implements this), there should be some concept of the last blocks that were resynced since the personality can resync more data than requested which will result in multiple resyncs of the same data with this implementation.
drivers/md/md.c +68 -0
@@ -4512,6 +4512,71 @@ static int md_ioctl(struct inode *inode, goto done_unlock; } +case RESYNC_RANGE : + { +mdu_range_t range; +struct hd_struct *part = inode->i_bdev->bd_part; +int ret; + +if (!arg) { + err = -EINVAL ; + goto abort_unlock; + } + + ret =copy_from_user (&range, argp,sizeof (range)); +if (ret) { + err = -EFAULT ; + goto abort_unlock; + } + +if (range.start > range.end) { + err = -EINVAL ; + goto abort_unlock; + } + +if (part) { +sector_t part_end; + + range.start += part->start_sect; + range.end += part->start_sect; + + part_end = part->start_sect + part->nr_sects -1 ; + +if (range.end > part_end) { + err = -EINVAL ; + goto abort_unlock; + } + } + +if (range.end >= mddev->array_size<<1 ) { + err = -EINVAL ; + goto abort_unlock; + } + +if (!test_bit (MD_RECOVERY_RUNNING , &mddev->recovery)) { +/* We are already in sync; return success */ + err =0 ; + goto abort_unlock; + } + +if (!test_bit (MD_RECOVERY_SYNC , &mddev->recovery)) { +/* Something is running but not a resync. */ + err = -EBUSY ; + goto abort_unlock; + } + +if (mddev->pers->resync_range ==NULL ) { + err = -EINVAL ; + goto abort_unlock; + } + + + err = mddev->pers->resync_range (mddev, range.start, + range.end); + + goto done_unlock; + } +default : err = -EINVAL ; goto abort_unlock;
@@ -4865,6 +4930,7 @@ static int md_seq_show(struct seq_file *mdk_rdev_t *rdev;struct mdstat_info *mi = seq->private;struct bitmap *bitmap; +unsigned long resync;if (v == (void *)1 ) {struct mdk_personality *pers;
@@ -4883,6 +4949,8 @@ static int md_seq_show(struct seq_file *return 0 ; } + resync = (mddev->curr_resync -atomic_read (&mddev->recovery_active))/2 ; +if (mddev_lock (mddev) <0 )return -EINTR ;
include/linux/raid/md_u.h +7 -0
@@ -46,6 +46,7 @@#define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33)#define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34)#define SKIP_RESYNC _IO (MD_MAJOR, 0x40) +#define RESYNC_RANGE _IO (MD_MAJOR, 0x41) typedef struct mdu_version_s { int major;
@@ -121,5 +122,11 @@ typedef struct mdu_param_s int max_fault; /* unused for now */ } mdu_param_t; +typedef struct mdu_range_s +{ +__u64 start;/* starting sector */ +__u64 end;/* ending sector */ +}mdu_range_t ; +#endif
drivers/md/raid5.c +61 -4
117 lines not on this page · raw message
include/linux/raid/md_k.h +1 -0
@@ -284,6 +284,7 @@ struct mdk_personality*/ void (*quiesce) (mddev_t *mddev,int state);int (*skip_resync) (mddev_t *mddev,sector_t start,sector_t end); +int (*resync_range) (mddev_t *mddev,sector_t start,sector_t end); };
include/linux/raid/raid5.h +2 -0
@@ -180,6 +180,8 @@ struct stripe_head {#define STRIPE_EXPANDING 9#define STRIPE_EXPAND_SOURCE 10#define STRIPE_EXPAND_READY 11 +#define STRIPE_RESYNC_RANGE 12 +/* * Plugging: *
PATCH 3/4 Add support for journal guided resync.
In reply to [email protected]Adds support for declare blocks, used by ext3's journal guided resync (declared mode.) A declare block is added to the journal to list blocks to be written during the current transaction. During journal replay, we perform a RAID resync of only these blocks and skip the rest of the resync.
fs/jbd/checkpoint.c +2 -0
@@ -712,6 +712,8 @@ void __journal_drop_transaction(journal_J_ASSERT (transaction->t_state ==T_FINISHED );J_ASSERT (transaction->t_buffers ==NULL ); +J_ASSERT (transaction->t_declare_root.rnode ==NULL ); +J_ASSERT (transaction->t_declare_done_root.rnode ==NULL );J_ASSERT (transaction->t_sync_datalist ==NULL );J_ASSERT (transaction->t_forget ==NULL );J_ASSERT (transaction->t_iobuf_list ==NULL );
fs/jbd/commit.c +275 -49
376 lines not on this page · raw message
fs/jbd/journal.c +49 -0
79 lines not on this page · raw message
fs/jbd/recovery.c +181 -9
304 lines not on this page · raw message
fs/jbd/transaction.c +66 -7
151 lines not on this page · raw message
include/linux/jbd.h +32 -2
@@ -26,6 +26,7 @@#include <linux/types.h> #include <linux/buffer_head.h> #include <linux/journal-head.h> +#include <linux/radix-tree.h> #include <linux/stddef.h> #include <linux/bit_spinlock.h> #include <linux/mutex.h>
@@ -137,6 +138,7 @@ typedef struct journal_s journal_t; /* J#define JFS_SUPERBLOCK_V1 3#define JFS_SUPERBLOCK_V2 4#define JFS_REVOKE_BLOCK 5 +#define JFS_DECLARE_BLOCK 6/* * Standard header for all descriptor blocks:
@@ -261,12 +263,14 @@ typedef struct journal_superblock_s#define JFS_FEATURE_INCOMPAT_REVOKE 0x00000001#define JFS_FEATURE_INCOMPAT_ASYNC_COMMIT 0x00000004 +#define JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS 0x00000008/* Features known to this kernel version: */ #define JFS_KNOWN_COMPAT_FEATURES JFS_FEATURE_COMPAT_CHECKSUM#define JFS_KNOWN_ROCOMPAT_FEATURES 0#define JFS_KNOWN_INCOMPAT_FEATURES (JFS_FEATURE_INCOMPAT_REVOKE | \ - JFS_FEATURE_INCOMPAT_ASYNC_COMMIT) + JFS_FEATURE_INCOMPAT_ASYNC_COMMIT | \ + JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS)#ifdef __KERNEL__
@@ -559,6 +563,15 @@ struct transaction_sstruct journal_head *t_sync_datalist;/* +* Radix tree of all data buffers that must be declared before being +* written, declare mode counters [j_list_lock] +*/ +struct radix_tree_root t_declare_root; +struct radix_tree_root t_declare_done_root; +unsigned int t_declare_count; +unsigned int t_declare_request; + +/* * Doubly-linked circular list of all forget buffers (superseded * buffers which we can un-checkpoint once this transaction commits) * [j_list_lock]
@@ -730,6 +743,7 @@ jbd_time_diff(unsigned int start, unsign* @j_wait_checkpoint: Wait queue to trigger checkpointing * @j_wait_commit: Wait queue to trigger commit * @j_wait_updates: Wait queue to wait for updates to complete +* @j_wait_declare: Wait queue to wait for declarations to complete * @j_checkpoint_mutex: Mutex for locking against concurrent checkpoints * @j_head: Journal head - identifies the first unused block in the journal * @j_tail: Journal tail - identifies the oldest still-used block in the
@@ -768,6 +782,8 @@ jbd_time_diff(unsigned int start, unsign* @j_wbufsize: maximum number of buffer_heads allowed in j_wbuf, the * number that will fit in j_blocksize * @j_last_sync_writer: most recent pid which did a synchronous write +* @j_declare_jhs: array of journal_heads for write_declare_blocks +* @j_declare_bhs: array of buffer_heads for write_declare_blocks * @j_private: An opaque pointer to fs-private information. */
@@ -841,6 +857,9 @@ struct journal_s/* Wait queue to wait for updates to complete */ wait_queue_head_t j_wait_updates; +/* Wait queue to wait for declarations to complete */ +wait_queue_head_t j_wait_declare; +/* Semaphore for locking against concurrent checkpoints */ struct mutex j_checkpoint_mutex;
@@ -970,6 +989,13 @@ struct journal_sstruct transaction_stats_s j_stats;/* +* Arrays of jhs and bhs for write_declare_blocks, to avoid +* having to allocate them each time. +*/ +void *j_declare_jhs[64 ]; +struct buffer_head *j_declare_bhs[64 ]; + +/* * An opaque pointer to fs-private information. ext3 puts its * superblock pointer here */
@@ -985,6 +1011,7 @@ struct journal_s#define JFS_FLUSHED 0x008/* The journal superblock has been flushed */ #define JFS_LOADED 0x010/* The journal superblock has been loaded */ #define JFS_BARRIER 0x020/* Use IDE barriers */ +#define JFS_DECLARE 0x040/* Declare data blocks before writing */ /* * Function declarations for the journaling transaction and buffer
@@ -1100,6 +1127,7 @@ extern void journal_ack_err (journextern int journal_clear_err (journal_t *);extern int journal_bmap (journal_t *,unsigned long ,unsigned long *);extern int journal_force_commit (journal_t *); +extern int journal_write_declare (journal_t *);/* * journal_head management
@@ -1244,7 +1272,9 @@ static inline int jbd_space_needed(journ#define BJ_LogCtl 6/* Buffer contains log descriptors */ #define BJ_Reserved 7/* Buffer is reserved for access by journal */ #define BJ_Locked 8/* Locked for I/O during commit */ -#define BJ_Types 9 +#define BJ_Declare 9/* Needs to be declared first */ +#define BJ_DeclareDone 10/* Has been declared */ +#define BJ_Types 11extern int jbd_blocks_per_page (struct inode *inode);
PATCH 4/4 Add journal guided resync (data=declared mode)
In reply to [email protected]We introduce a new data write mode known as declared mode. This is based on ordered mode except that a list of blocks to be written during the current transaction is added to the journal before the blocks themselves are written to the disk. Then, if the system crashes, we can resync only those blocks during journal replay and skip the rest of the resync of the RAID array.
TODO: Add support to e2fsck.
TODO: The following sequence of events could cause resync to be skipped incorrectly:
- An MD array that supports RESYNC_RANGE is undergoing resync.
- A filesystem on that array is mounted with data=declared.
- The machine crashes before the resync completes.
- The array is restarted and the filesystem is remounted.
- Recovery resyncs only the blocks that were undergoing writes during the crash and skips the rest.
Addressing this requires even more communication between MD and ext and I need to think more about how to do this.
fs/ext3/file.c +2 -1
@@ -78,7 +78,8 @@ ext3_file_write(struct kiocb *iocb, cons* Open question --- do we care about flushing timestamps too * if the inode is IS_SYNC? */ -if (!ext3_should_journal_data (inode)) +if (!ext3_should_journal_data (inode) && + !ext3_should_declare_data (inode))return ret; goto force_commit;
fs/ext3/fsync.c +6 -1
@@ -66,8 +66,13 @@ int ext3_sync_file(struct file * file, s* filemap_fdatawait() will encounter a ton of newly-dirtied pages * (they were dirtied by commit). But that's OK - the blocks are * safe in-journal, which is all fsync() needs to ensure. +* +* data=declared: +* Declare blocks are written before data blocks, then the +* sync proceeds as for data=ordered. */ -if (ext3_should_journal_data (inode)) { +if (ext3_should_journal_data (inode) || +ext3_should_declare_data (inode)) { ret =ext3_force_commit (inode->i_sb); goto out; }
fs/ext3/inode.c +62 -3
97 lines not on this page · raw message
fs/ext3/super.c +26 -2
@@ -391,6 +391,9 @@ static void ext3_put_super (struct superint i, err;ext3_xattr_put_super (sb); +journal_clear_features (sbi->s_journal,0 ,0 , +JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS ); +journal_update_superblock (sbi->s_journal,1 ); err =journal_destroy (sbi->s_journal); sbi->s_journal =NULL ;if (err <0 )
@@ -553,6 +556,8 @@ static int ext3_show_options(struct seq_seq_puts (seq,",data=ordered" ); else if (test_opt(sb, DATA_FLAGS) == EXT3_MOUNT_WRITEBACK_DATA) seq_puts(seq, ",data=writeback"); + else if (test_opt(sb, DATA_FLAGS) == EXT3_MOUNT_DECLARED_DATA) +seq_puts (seq,",data=declared" );ext3_show_quota_options (seq, sb);
@@ -682,7 +687,7 @@ enum { Opt_usrjquota, Opt_grpjquota, Opt_offusrjquota, Opt_offgrpjquota, Opt_jqfmt_vfsold, Opt_jqfmt_vfsv0, Opt_quota, Opt_noquota, Opt_ignore, Opt_barrier, Opt_err, Opt_resize, Opt_usrquota, - Opt_grpquota + Opt_grpquota, Opt_data_declared }; static match_table_t tokens = {
@@ -721,6 +726,7 @@ static match_table_t tokens = { {Opt_data_journal, "data=journal"}, {Opt_data_ordered, "data=ordered"}, {Opt_data_writeback, "data=writeback"}, + {Opt_data_declared, "data=declared"}, {Opt_offusrjquota, "usrjquota="}, {Opt_usrjquota, "usrjquota=%s"}, {Opt_offgrpjquota, "grpjquota="},
@@ -922,6 +928,9 @@ static int parse_options (char *options, goto datacheck;case Opt_data_writeback: data_opt =EXT3_MOUNT_WRITEBACK_DATA ; + goto datacheck; +case Opt_data_declared: + data_opt =EXT3_MOUNT_DECLARED_DATA ; datacheck:if (is_remount) { if ((sbi->s_mount_opt & EXT3_MOUNT_DATA_FLAGS)
@@ -1740,7 +1749,21 @@ static int ext3_fill_super (struct superelse set_opt (sbi->s_mount_opt,JOURNAL_DATA );break ; - +case EXT3_MOUNT_DECLARED_DATA : +if (!journal_check_available_features + (sbi->s_journal,0 ,0 ,JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS )) { +printk (KERN_ERR "EXT3-fs: Journal does not support " +"declared data journaling mode\n" ); + goto failed_mount4; + } +spin_lock (&sbi->s_journal->j_state_lock); + sbi->s_journal->j_flags |=JFS_DECLARE ; +spin_unlock (&sbi->s_journal->j_state_lock); +if (!journal_set_features (sbi->s_journal,0 ,0 , +JFS_FEATURE_INCOMPAT_DECLARE_BLOCKS )) { +printk (KERN_ERR "EXT3-fs: Cannot set declared mode.\n" ); + goto failed_mount4; + }case EXT3_MOUNT_ORDERED_DATA :case EXT3_MOUNT_WRITEBACK_DATA : if (!journal_check_available_features
@@ -1797,6 +1820,7 @@ static int ext3_fill_super (struct superprintk (KERN_INFO "EXT3-fs: mounted filesystem with %s data mode.\n" ,test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_JOURNAL_DATA ?"journal" :test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_ORDERED_DATA ?"ordered" : +test_opt (sb,DATA_FLAGS ) ==EXT3_MOUNT_DECLARED_DATA ?"declared" :"writeback" );lock_kernel ();
include/linux/ext3_fs.h +3 -2
@@ -357,11 +357,11 @@ struct ext3_inode {#define EXT3_MOUNT_MINIX_DF 0x00080/* Mimics the Minix statfs */ #define EXT3_MOUNT_NOLOAD 0x00100/* Don't use existing journal*/ #define EXT3_MOUNT_ABORT 0x00200/* Fatal error detected */ -#define EXT3_MOUNT_DATA_FLAGS 0x00C00/* Mode for data writes: */ +#define EXT3_MOUNT_DATA_FLAGS 0x01C00/* Mode for data writes: */ #define EXT3_MOUNT_JOURNAL_DATA 0x00400/* Write data to journal */ #define EXT3_MOUNT_ORDERED_DATA 0x00800/* Flush data before commit */ #define EXT3_MOUNT_WRITEBACK_DATA 0x00C00/* No data ordering */ -#define EXT3_MOUNT_UPDATE_JOURNAL 0x01000/* Update the journal format */ +#define EXT3_MOUNT_DECLARED_DATA 0x01000/* Declare data blocks before writing */ #define EXT3_MOUNT_NO_UID32 0x02000/* Disable 32-bit UIDs */ #define EXT3_MOUNT_XATTR_USER 0x04000/* Extended user attributes */ #define EXT3_MOUNT_POSIX_ACL 0x08000/* POSIX Access Control Lists */
@@ -383,6 +383,7 @@ struct ext3_inode {#define EXT2_MOUNT_ABORT EXT3_MOUNT_ABORT#define EXT2_MOUNT_DATA_FLAGS EXT3_MOUNT_DATA_FLAGS#endif +#define EXT3_MOUNT_UPDATE_JOURNAL 0x40000000/* Update the journal format */ #define ext3_set_bit ext2_set_bit#define ext3_set_bit_atomic ext2_set_bit_atomic
include/linux/ext3_jbd.h +11 -0
@@ -265,4 +265,15 @@ static inline int ext3_should_writeback_ return 0; } +static inline int ext3_should_declare_data (struct inode *inode) +{ +if (!S_ISREG (inode->i_mode)) +return 0 ; +if (EXT3_I (inode)->i_flags &EXT3_JOURNAL_DATA_FL ) +return 0 ; +if (test_opt (inode->i_sb,DATA_FLAGS ) ==EXT3_MOUNT_DECLARED_DATA ) +return 1 ; +return 0 ; +} +#endif /* _LINUX_EXT3_JBD_H */
Re: [patch 3/4] [jbd] Add support for journal guided resync.
In reply to [email protected]Quoted 8 lines · On Thu, 01 Oct 2009 18:39:32 -0400 [email protected] wrote:
Adds support for declare blocks, used by ext3's journal guided resync (declared mode.) A declare block is added to the journal to list blocks to be written during the current transaction. During journal replay, we perform a RAID resync of only these blocks and skip the rest of the resync.
...
+int
wait_for_descriptors(journal_t *journal, transaction_t *trans){
Opening brace goes on the next line, please. Several instances.
Quoted 22 lines
fs/jbd/commit.c +22
-0+struct journal_head *jh; +struct buffer_head *bh; +int err =0 ; + +wait_for_ctlbuf: + +while (trans->t_log_list !=NULL ) { + + jh = trans->t_log_list->b_tprev; + bh =jh2bh (jh); +if (buffer_locked (bh)) { +wait_on_buffer (bh); + goto wait_for_ctlbuf; + } +if (cond_resched ()) + goto wait_for_ctlbuf; + +if (unlikely (!buffer_uptodate (bh))) + err = -EIO ; + +BUFFER_TRACE (bh,"ph5: control buffer writeout done: unfile" ); +clear_buffer_jwrite (bh);
fs/jbd/commit.c static inline __u32 jbd_checksum_data(__ · line 408
Quoted 10 lines
fs/jbd/commit.c +10
-0+journal_unfile_buffer (journal, jh); +journal_put_journal_head (jh); +__brelse (bh);/* One for getblk */ + } + +return err; +} + +struct journal_head *get_descriptor (journal_t *journal,transaction_t *trans, +int blocktype,char **tagp,int *space_left) {
Did all these functions need global scope? Some do, but I didn't check them all.
Those which cannot be made static should be given appropriate names, such as jbd_get_descriptor().
Quoted 100 lines
fs/jbd/commit.c +97
-0+struct journal_head *descriptor; +struct buffer_head *dbh; +journal_header_t *header; + +jbd_debug (4 ,"JBD: get descriptor\n" ); + + descriptor =journal_get_descriptor_buffer (journal); +if (!descriptor) +return NULL ; + + dbh =jh2bh (descriptor); +jbd_debug (4 ,"JBD: got buffer %llu (%p)\n" , + (unsigned long long )dbh->b_blocknr, dbh->b_data); + header = (journal_header_t *)&dbh->b_data[0 ]; + header->h_magic =cpu_to_be32 (JFS_MAGIC_NUMBER ); + header->h_blocktype =cpu_to_be32 (blocktype); + header->h_sequence =cpu_to_be32 (trans->t_tid); + + *tagp = &dbh->b_data[sizeof (journal_header_t)]; + *space_left = dbh->b_size -sizeof (journal_header_t); + +set_buffer_jwrite (dbh); +set_buffer_dirty (dbh); + +/* Record it so that we can wait for it later */ +BUFFER_TRACE (dbh,"ph3: file as descriptor" ); +journal_file_buffer (descriptor, trans, BJ_LogCtl); + +return descriptor; +} + +void write_declare_blocks (journal_t *journal,transaction_t *transaction, +int committing) +{ +struct journal_head *jh, *descriptor =NULL ; +struct buffer_head *bh; +int i, bufs =0 , err; +unsigned int n, count =0 , to_write; +unsigned long nextblock =0 ; +char *tagp =NULL ; +journal_block_tag_t *tag =NULL ; +int space_left =0 , first_tag =0 , tag_flag; +struct radix_tree_root *root; + + root = &transaction->t_declare_root; + +spin_lock (&journal->j_list_lock); + to_write = transaction->t_declare_request; + transaction->t_declare_request =0 ; +spin_unlock (&journal->j_list_lock); + +if (to_write ==UINT_MAX ) +jbd_debug (1 ,"jbd: tid %d write declare request for ALL " +"blocks\n" , transaction->t_tid); +else +jbd_debug (1 ,"jbd: tid %d write declare request for %u " +"blocks\n" , transaction->t_tid, to_write); +write_declare: +cond_resched (); +spin_lock (&journal->j_list_lock); + + n =radix_tree_gang_lookup (root, journal->j_declare_jhs, nextblock,1 ); +while (n) { +if (!descriptor) { +J_ASSERT (bufs ==0 ); + +spin_unlock (&journal->j_list_lock); + + descriptor =get_descriptor (journal, transaction, +JFS_DECLARE_BLOCK , + &tagp, &space_left); + +if (!descriptor) { +journal_abort (journal, -EIO ); +return ; + } + + first_tag =1 ; + journal->j_declare_bhs[bufs++] =jh2bh (descriptor); + + goto write_declare; + } + + jh = (struct journal_head *)journal->j_declare_jhs[0 ]; + bh =jh2bh (jh); + +/* refile the buffer as having been declared */ +if (!inverted_lock (journal, bh)) + goto write_declare; +__journal_unfile_buffer (jh); +__journal_file_buffer (jh, transaction, BJ_DeclareDone); + +jbd_unlock_bh_state (bh); + +/* record the block's tag in the current descriptor buffer */ + tag_flag =0 ; +if (!first_tag)
fs/jbd/commit.c static inline __u32 jbd_checksum_data(__ · line 519
Quoted 10 lines
fs/jbd/commit.c +10
-0+ tag_flag |=JFS_FLAG_SAME_UUID ; + + tag = (journal_block_tag_t *)tagp; + tag->t_blocknr =cpu_to_be32 (bh->b_blocknr); + tag->t_flags =cpu_to_be32 (tag_flag); + tagp +=sizeof (journal_block_tag_t); + space_left -=sizeof (journal_block_tag_t); + +if (first_tag) { +memcpy (tagp, journal->j_uuid,16 );
Please pass the patches through scripts/checkpatch.pl and review the result.
Quoted 45 lines
fs/jbd/commit.c +42
-0+ tagp +=16 ; + space_left -=16 ; + first_tag =0 ; + } + + count++; + +/* advance to the next journal head and buffer */ + nextblock = bh->b_blocknr +1 ; + n =radix_tree_gang_lookup (root, journal->j_declare_jhs, + nextblock,1 ); + +/* If there's no more to do, or if the descriptor is full, +let the IO rip! */ + +if (bufs ==ARRAY_SIZE (journal->j_declare_bhs) || n ==0 || + count == to_write || + space_left <sizeof (journal_block_tag_t) +16 ) { + +jbd_debug (4 ,"JBD: Submit %d IOs\n" , bufs); + +/* Write an end-of-descriptor marker before +* submitting the IOs. "tag" still points to +* the last tag we set up. +*/ + + tag->t_flags |=cpu_to_be32 (JFS_FLAG_LAST_TAG ); + +int journal_write_declare (journal_t *journal) +{ +transaction_t *transaction = journal->j_running_transaction; +DEFINE_WAIT (wait); + +if (transaction ==NULL ) +return 0 ; + +spin_lock (&journal->j_list_lock); + +if (transaction->t_declare_root.rnode ==NULL ) { +spin_unlock (&journal->j_list_lock); +return 0 ; + }
fs/jbd/commit.c static inline __u32 jbd_checksum_data(__
Quoted 10 lines
fs/jbd/commit.c +10
-0+ + transaction->t_declare_request =UINT_MAX ; + +jbd_debug (1 ,"waking commit thread for fsync declare\n" ); +wake_up (&journal->j_wait_commit); + +prepare_to_wait (&journal->j_wait_declare, &wait,TASK_INTERRUPTIBLE ); +spin_unlock (&journal->j_list_lock); +schedule (); +finish_wait (&journal->j_wait_declare, &wait);
If this function is only ever called by a kernel thread then TASK_INTERRUPTIBLE is OK.
If this function can be called by a userspace process then TASK_INTERRUPTIBLE is buggy - a signal_pending() state will cause the schedule() to fall straight through.
Quoted 19 lines
fs/jbd/recovery.c +16
-0+return 0 ; +} + +static int resync_range (journal_t *j,unsigned long start, +unsigned long end) +{ +int err; +struct inode *fake_inode =kmalloc (sizeof (*fake_inode),GFP_KERNEL ); +mdu_range_t range; +sector_t sectors_per_block = j->j_blocksize >>9 ; +mm_segment_t old_fs; + +if (fake_inode ==NULL ) { +printk (KERN_ERR "JBD: Out of memory during recovery.\n" ); +return -ENOMEM ; + }
fs/jbd/recovery.c void journal_brelse_array(struct buffer_ · line 83
Quoted 10 lines
fs/jbd/recovery.c +10
-0+ + fake_inode->i_bdev = j->j_fs_dev; + range.start = start * sectors_per_block; + range.end = end * sectors_per_block + sectors_per_block -1 ; + + old_fs =get_fs (); +set_fs (KERNEL_DS ); + err =blkdev_driver_ioctl (fake_inode,NULL , j->j_fs_dev->bd_disk, +RESYNC_RANGE , (long )&range); +set_fs (old_fs);
erk, what's happening here.
I can't find the blkdev_driver_ioctl() definition. If I could, I'd be asking if it can be refactored so we can call a regular kernel function and avoid the nasty set_fs(), and do away with the nasty fake_inode.
Quoted 11 lines
fs/jbd/recovery.c +7
-0+jbd_debug (3 ,"RESYNC_RANGE of sectors %llu - %llu returned %d\n" , + range.start, range.end, err); + +kfree (fake_inode); + +return err; +}
Quoted 4 lines · [email protected] writes:
The changes consist of patches to ext3, jbd, MD, and the raid456 personality. These patches are currently against the RHEL 5 kernel 2.6.18-128.1.6. Porting to ext4/jbd2 and a more modern kernel is a TODO item.
The basic idea seems very cool, but the code would probably need porting to mainline before anything could be done with it on this list. Also most likely you would need a ext4 version.
(my experience is that the ext3 maintainers don't accept new features anymore)
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [patch 4/4] [ext3] Add journal guided resync (data=declared mode)
In reply to [email protected]Quoted 8 lines · On Thursday October 1, [email protected] wrote:
We introduce a new data write mode known as declared mode. This is based on ordered mode except that a list of blocks to be written during the current transaction is added to the journal before the blocks themselves are written to the disk. Then, if the system crashes, we can resync only those blocks during journal replay and skip the rest of the resync of the RAID array.
TODO: Add support to e2fsck.
Quoted 10 lines
TODO: The following sequence of events could cause resync to be skipped incorrectly:
- An MD array that supportsRESYNC_RANGEis undergoing resync.
- A filesystem on that array is mounted with data=declared.
- The machine crashes before the resync completes.
- The array is restarted and the filesystem is remounted.
- Recovery resyncs only the blocks that were undergoing writes during the crash and skips the rest.
Addressing this requires even more communication between MD and ext and I need to think more about how to do this.
I have thought about this sort of thing from time to time and I have a very different idea for how the necessary communication between the filesystem and MD would happen. I think my approach would completely address this problem, and doesn't need to add any ioctls (which I am not keen on).
I would add two new BIO_RW_ flags to be used with WRITE requests. The first flag would mean "don't worry about a crash in the middle of this write, I will validate it after a crash before I rely on the data."
The second would mean "last time I wrote data near here there might have been a failure, be extra careful".
So the first flag would be used during normal filesystem writes for every block that gets recorded in the journal, and for every write to the journal.
The second flag is used after a crash to re-write every block that could have been in-flight during the crash. Some of those blocks will be read from the journal and written to their proper home, other will be read from wherever they are and written back there.
The first flag would be interpreted by MD as "don't set the bitmap bit". The second flag would be interpreted as "don't trust the parity block, but do a reconstruct-write".
With this scheme you would still need a write-intent-bitmap on the MD array, but no bits would ever be set if the filesystem were using the new flags, so no performance impact. You probably could run without a bitmap, in which case the flag would me "don't mark the array as active".
I'm not entirely sure about the second flag. Maybe it would be better to make it a flag for READ and have it mean "validate and correct any redundancy information (duplicates or parity) for this block before returning it. Then we could have just one flag, which meant different things for READ and WRITE.
What do you think?
NeilBrown
Re: [patch 4/4] [ext3] Add journal guided resync (data=declared mode)
Quoted 26 lines · Neil Brown
I have thought about this sort of thing from time to time and I have a very different idea for how the necessary communication between the filesystem and MD would happen. I think my approach would completely address this problem, and doesn't need to add any ioctls (which I am not keen on).
I would add two new
BIO_RW_flags to be used with WRITE requests. The first flag would mean "don't worry about a crash in the middle of this write, I will validate it after a crash before I rely on the data."
The second would mean "last time I wrote data near here there might have been a failure, be extra careful".So the first flag would be used during normal filesystem writes for every block that gets recorded in the journal, and for every write to the journal.
The second flag is used after a crash to re-write every block that could have been in-flight during the crash. Some of those blocks will be read from the journal and written to their proper home, other will be read from wherever they are and written back there.
The first flag would be interpreted by MD as "don't set the bitmap bit". The second flag would be interpreted as "don't trust the parity block, but do a reconstruct-write".
Quoted 5 lines
With this scheme you would still need a write-intent-bitmap on the MD array, but no bits would ever be set if the filesystem were using the new flags, so no performance impact. You probably could run without a bitmap, in which case the flag would me "don't mark the array as active".
That makes a lot of sense. The original design used something similar to your second flag to do all the resync work. I changed to an IOCTL approach because there's no good way for a flagged IO to return "the resync you requested could not be done so don't skip the automated resync." But with your suggestion, there's no need to explicitly skip the automated resync at the end either - it just won't act on those blocks, assuming all blocks were previously written with the first flag.
I don't like the IOCTLs either :)
Quoted 5 lines
I'm not entirely sure about the second flag. Maybe it would be better to make it a flag for READ and have it mean "validate and correct any redundancy information (duplicates or parity) for this block before returning it. Then we could have just one flag, which meant different things for READ and WRITE.
Yes, it needs to be a READ flag to avoid an unnecessary read-write cycle. The list of declare blocks in the ext3 journal just lists the block numbers, not their contents (this is equivalent to data=ordered, not data=journaled).
It would be nice to also have a WRITE flag though, for those cases (metadata) where we do know what's being written to the block.
In any case, I think it's less confusing to have two flags: the "don't worry about this block for now" flag and the "resync this block now" flag.
There's still one complication though, and an IOCTL may still be necessary:
at fsck time, we need to perform a journal recovery from e2fsck (userland.) Clearly, resyncing the declared blocks (plus the journal and affected metadata blocks) must be done before e2fsck does any writes to these blocks. An IOCTL seemed to be the best way to do that, but I'm open to other suggestions.
Quoted 1 line
What do you think?
I'll send a new patchset, hopefully today.
Cheers,
Jody
Quoted 1 line
NeilBrown