A dm-raid45 target implemented using md raid5.
Subsystems (2)
cover A dm-raid45 target implemented using md raid5.
Greetings Heinz, Alasdair, and all,
(Alasdair and Heinz cc:ed on this intro, but the patches are
only going to the lists).
Some months ago I posted a proof-of-concept patch which attempted to provide RAID4/5/6 functionality to 'dm' using md/raid5.c. While it did a least partly work it contained lots of hacks and was very ugly.
I finally made time to do the job "properly".
The following series, when applied on top of a bunch of patches I just submitted for linux-next, provides a 'dm-raid45' target which is largely compatible with the one that Heinz has written (and several distros are shipping), but which uses md/raid5.c for the core IO processing.
I have tried to split the patch up into easy-to-handle pieces. You will note that some changes to core-dm are required, in particular to pass back 'congestion' information and to handle plugging (which raid5 uses to improve throughput). I hope the approach I have taken is suitable, but it can obviously be changed if necessary.
The create/status/message interface differs from the one in Heinz's patch, but should be close enough to work with current 'dmraid'.
If you want to try the patches (rather than just read them) you should probably "git pull" (please don't clone) from
git://neil.brown.name/md md-dm-raid45
so as to get all the prior refactoring patches in md.
Some advantages of this over Heinz's patch (at least as it was when I last looked at it) are:
- raid6 support
- support for XOR-offload hardware where present
- less code duplication
- a single dm device can include multiple dm-raid45 targets. (Heinz' code accesses dm_disk(md)->queue directly which is a layering violations and assumes that there is no other target in the mapped_device).
There is a lot more that could be done to this such as getting to work with a disk based dirty-log and making the reshape options available. But this patch set should provide all basic RAID5 functionality.
Would the dm community be interested in including this work upstream (after suitable review and testing)?
Thanks,
NeilBrown
---
NeilBrown (12):
md: reduce dependence on sysfs.
md/raid5: factor out code for changing size of stripe cache. md/dm: create dm-raid456 module using md/raid5
dm-raid456: add support for raising events to userspace.
raid5: Don't set read-ahead when there is no queue
dm-raid456: add congestion checking.
md/raid5: add simple plugging infrastructure.
md/plug: optionally use plugger to unplug an array during resync/recovery.
dm-raid456: support unplug
dm-raid456: add support for setting IO hints.
dm-raid456: add suspend/resume method
dm-raid456: add message handler.
drivers/md/Kconfig | 8 + drivers/md/Makefile | 1 drivers/md/dm-raid456.c | 540 +++++++++++++++++++++++++++++++++++++++++ drivers/md/dm-table.c | 19 + drivers/md/md.c | 211 ++++++++++------ drivers/md/md.h | 43 +++ drivers/md/raid5.c | 155 +++++++----- drivers/md/raid5.h | 6 include/linux/device-mapper.h | 13 + 9 files changed, 859 insertions(+), 137 deletions(-) create mode 100644 drivers/md/dm-raid456.c
PATCH 1/12 md: reduce dependence on sysfs.
We will want md devices to live as dm targets where sysfs is not visible. So allow md to not connect to sysfs.
Signed-off-by; NeilBrown <[email protected]>
--- drivers/md/md.c | 101 ++++++++++++++++++++++++---------------------------- drivers/md/md.h | 12 ++++++ drivers/md/raid5.c | 8 ++-- 3 files changed, 62 insertions(+), 59 deletions(-)
drivers/md/md.c +47 -54
@@ -540,14 +540,16 @@ static void mddev_unlock(mddev_t * mddev)mutex_lock (&mddev->open_mutex);mutex_unlock (&mddev->reconfig_mutex); -if (to_remove != &md_redundancy_group) -sysfs_remove_group (&mddev->kobj, to_remove); -if (mddev->pers ==NULL || - mddev->pers->sync_request ==NULL ) { -sysfs_remove_group (&mddev->kobj, &md_redundancy_group); -if (mddev->sysfs_action) -sysfs_put (mddev->sysfs_action); - mddev->sysfs_action =NULL ; +if (mddev->kobj.sd) { +if (to_remove != &md_redundancy_group) +sysfs_remove_group (&mddev->kobj, to_remove); +if (mddev->pers ==NULL || + mddev->pers->sync_request ==NULL ) { +sysfs_remove_group (&mddev->kobj, &md_redundancy_group); +if (mddev->sysfs_action) +sysfs_put (mddev->sysfs_action); + mddev->sysfs_action =NULL ; + } }mutex_unlock (&mddev->open_mutex); }else
@@ -1803,11 +1805,9 @@ static int bind_rdev_to_array(mdk_rdev_t * rdev, mddev_t * mddev) goto fail; ko = &part_to_dev (rdev->bdev->bd_part)->kobj; -if ((err =sysfs_create_link (&rdev->kobj, ko,"block" ))) { -kobject_del (&rdev->kobj); - goto fail; - } - rdev->sysfs_state =sysfs_get_dirent (rdev->kobj.sd,"state" ); +if (sysfs_create_link (&rdev->kobj, ko,"block" )) +/* failure here is OK */ + rdev->sysfs_state =sysfs_get_dirent_safe (rdev->kobj.sd,"state" );list_add_rcu (&rdev->same_set, &mddev->disks);bd_claim_by_disk (rdev->bdev, rdev->bdev->bd_holder, mddev->gendisk);
@@ -2334,8 +2334,8 @@ state_store(mdk_rdev_t *rdev, const char *buf, size_t len)set_bit (In_sync, &rdev->flags); err =0 ; } -if (!err && rdev->sysfs_state) -sysfs_notify_dirent (rdev->sysfs_state); +if (!err) +sysfs_notify_dirent_safe (rdev->sysfs_state);return err ? err : len; } static struct rdev_sysfs_entry rdev_state =
@@ -2430,14 +2430,10 @@ slot_store(mdk_rdev_t *rdev, const char *buf, size_t len) rdev->raid_disk =-1 ;return err; }else -sysfs_notify_dirent (rdev->sysfs_state); +sysfs_notify_dirent_safe (rdev->sysfs_state);sprintf (nm,"rd%d" , rdev->raid_disk);if (sysfs_create_link (&rdev->mddev->kobj, &rdev->kobj, nm)) -printk (KERN_WARNING -"md: cannot register " -"%s for %s\n" , - nm,mdname (rdev->mddev)); - +/* failure here is OK */ ;/* don't wakeup anyone, leave that to userspace. */ }else {if (slot >= rdev->mddev->raid_disks)
@@ -2447,7 +2443,7 @@ slot_store(mdk_rdev_t *rdev, const char *buf, size_t len)clear_bit (Faulty, &rdev->flags);clear_bit (WriteMostly, &rdev->flags);set_bit (In_sync, &rdev->flags); -sysfs_notify_dirent (rdev->sysfs_state); +sysfs_notify_dirent_safe (rdev->sysfs_state); }return len; }
@@ -3406,7 +3402,7 @@ array_state_store(mddev_t *mddev, const char *buf, size_t len)if (err)return err;else { -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state);return len; } }
@@ -3704,7 +3700,7 @@ action_store(mddev_t *mddev, const char *page, size_t len) }set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);md_wakeup_thread (mddev->thread); -sysfs_notify_dirent (mddev->sysfs_action); +sysfs_notify_dirent_safe (mddev->sysfs_action);return len; }
@@ -4250,13 +4246,14 @@ static int md_alloc(dev_t dev, char *name) disk->disk_name); error =0 ; } -if (sysfs_create_group (&mddev->kobj, &md_bitmap_group)) +if (mddev->kobj.sd && +sysfs_create_group (&mddev->kobj, &md_bitmap_group))printk (KERN_DEBUG "pointless warning\n" ); abort:mutex_unlock (&disks_mutex); -if (!error) { +if (!error && mddev->kobj.sd) {kobject_uevent (&mddev->kobj,KOBJ_ADD ); - mddev->sysfs_state =sysfs_get_dirent (mddev->kobj.sd,"array_state" ); + mddev->sysfs_state =sysfs_get_dirent_safe (mddev->kobj.sd,"array_state" ); }mddev_put (mddev);return error;
@@ -4294,7 +4291,7 @@ static void md_safemode_timeout(unsigned long data)if (!atomic_read (&mddev->writes_pending)) { mddev->safemode =1 ;if (mddev->external) -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state); }md_wakeup_thread (mddev->thread); }
@@ -4366,7 +4363,7 @@ static int md_run(mddev_t *mddev)return -EINVAL ; } } -sysfs_notify_dirent (rdev->sysfs_state); +sysfs_notify_dirent_safe (rdev->sysfs_state); }spin_lock (&pers_lock);
@@ -4465,11 +4462,12 @@ static int md_run(mddev_t *mddev)return err; }if (mddev->pers->sync_request) { -if (sysfs_create_group (&mddev->kobj, &md_redundancy_group)) +if (mddev->kobj.sd && +sysfs_create_group (&mddev->kobj, &md_redundancy_group))printk (KERN_WARNING "md: cannot register extra attributes for %s\n" ,mdname (mddev)); - mddev->sysfs_action =sysfs_get_dirent (mddev->kobj.sd,"sync_action" ); + mddev->sysfs_action =sysfs_get_dirent_safe (mddev->kobj.sd,"sync_action" ); }else if (mddev->ro ==2 )/* auto-readonly not meaningful */ mddev->ro =0 ;
@@ -4487,8 +4485,7 @@ static int md_run(mddev_t *mddev)char nm[20 ];sprintf (nm,"rd%d" , rdev->raid_disk);if (sysfs_create_link (&mddev->kobj, &rdev->kobj, nm)) -printk ("md: cannot register %s for %s\n" , - nm,mdname (mddev)); +/* failure here is OK */ ; }set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);
@@ -4500,9 +4497,8 @@ static int md_run(mddev_t *mddev)md_wakeup_thread (mddev->sync_thread);/* possibly kick off a reshape */ md_new_event (mddev); -sysfs_notify_dirent (mddev->sysfs_state); -if (mddev->sysfs_action) -sysfs_notify_dirent (mddev->sysfs_action); +sysfs_notify_dirent_safe (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_action);sysfs_notify (&mddev->kobj,NULL ,"degraded" );return 0 ; }
@@ -4542,7 +4538,7 @@ static int restart_array(mddev_t *mddev)set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);md_wakeup_thread (mddev->thread);md_wakeup_thread (mddev->sync_thread); -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state);return 0 ; }
@@ -4665,7 +4661,7 @@ static int md_set_readonly(mddev_t *mddev, int is_open) mddev->ro =1 ;set_disk_ro (mddev->gendisk,1 );clear_bit (MD_RECOVERY_FROZEN , &mddev->recovery); -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state); err =0 ; } out:
@@ -4698,7 +4694,7 @@ static int do_md_stop(mddev_t * mddev, int mode, int is_open) mddev->queue->backing_dev_info.congested_fn =NULL ;/* tell userspace to handle 'inactive' */ -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state);list_for_each_entry (rdev, &mddev->disks, same_set)if (rdev->raid_disk >=0 ) {
@@ -4744,7 +4740,7 @@ static int do_md_stop(mddev_t * mddev, int mode, int is_open) err =0 ;blk_integrity_unregister (disk);md_new_event (mddev); -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state);return err; }
@@ -5106,7 +5102,7 @@ static int add_new_disk(mddev_t * mddev, mdu_disk_info_t *info)if (err)export_rdev (rdev);else -sysfs_notify_dirent (rdev->sysfs_state); +sysfs_notify_dirent_safe (rdev->sysfs_state);md_update_sb (mddev,1 );if (mddev->degraded)
@@ -5752,7 +5748,7 @@ static int md_ioctl(struct block_device *bdev, fmode_t mode,if (_IOC_TYPE (cmd) ==MD_MAJOR && mddev->ro && mddev->pers) {if (mddev->ro ==2 ) { mddev->ro =0 ; -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state);set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);md_wakeup_thread (mddev->thread); }else {
@@ -5997,7 +5993,7 @@ void md_error(mddev_t *mddev, mdk_rdev_t *rdev) mddev->pers->error_handler (mddev,rdev);if (mddev->degraded)set_bit (MD_RECOVERY_RECOVER , &mddev->recovery); -sysfs_notify_dirent (rdev->sysfs_state); +sysfs_notify_dirent_safe (rdev->sysfs_state);set_bit (MD_RECOVERY_INTR , &mddev->recovery);set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);md_wakeup_thread (mddev->thread);
@@ -6458,7 +6454,7 @@ void md_write_start(mddev_t *mddev, struct bio *bi)spin_unlock_irq (&mddev->write_lock); }if (did_change) -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state);wait_event (mddev->sb_wait, !test_bit (MD_CHANGE_CLEAN , &mddev->flags) && !test_bit (MD_CHANGE_PENDING , &mddev->flags));
@@ -6501,7 +6497,7 @@ int md_allow_write(mddev_t *mddev) mddev->safemode =1 ;spin_unlock_irq (&mddev->write_lock);md_update_sb (mddev,0 ); -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state); }else spin_unlock_irq (&mddev->write_lock);
@@ -6887,10 +6883,7 @@ static int remove_and_add_spares(mddev_t *mddev)sprintf (nm,"rd%d" , rdev->raid_disk);if (sysfs_create_link (&mddev->kobj, &rdev->kobj, nm)) -printk (KERN_WARNING -"md: cannot register " -"%s for %s\n" , - nm,mdname (mddev)); +/* failure here is OK */ spares++;md_new_event (mddev);set_bit (MD_CHANGE_DEVS , &mddev->flags);
@@ -6983,7 +6976,7 @@ void md_check_recovery(mddev_t *mddev) mddev->safemode =0 ;spin_unlock_irq (&mddev->write_lock);if (did_change) -sysfs_notify_dirent (mddev->sysfs_state); +sysfs_notify_dirent_safe (mddev->sysfs_state); }if (mddev->flags)
@@ -7022,7 +7015,7 @@ void md_check_recovery(mddev_t *mddev) mddev->recovery =0 ;/* flag recovery needed just to double check */ set_bit (MD_RECOVERY_NEEDED , &mddev->recovery); -sysfs_notify_dirent (mddev->sysfs_action); +sysfs_notify_dirent_safe (mddev->sysfs_action);md_new_event (mddev); goto unlock; }
@@ -7084,7 +7077,7 @@ void md_check_recovery(mddev_t *mddev) mddev->recovery =0 ; }else md_wakeup_thread (mddev->sync_thread); -sysfs_notify_dirent (mddev->sysfs_action); +sysfs_notify_dirent_safe (mddev->sysfs_action);md_new_event (mddev); } unlock:
@@ -7093,7 +7086,7 @@ void md_check_recovery(mddev_t *mddev)if (test_and_clear_bit (MD_RECOVERY_RECOVER , &mddev->recovery))if (mddev->sysfs_action) -sysfs_notify_dirent (mddev->sysfs_action); +sysfs_notify_dirent_safe (mddev->sysfs_action); }mddev_unlock (mddev); }
@@ -7101,7 +7094,7 @@ void md_check_recovery(mddev_t *mddev)void md_wait_for_blocked_rdev (mdk_rdev_t *rdev,mddev_t *mddev) { -sysfs_notify_dirent (rdev->sysfs_state); +sysfs_notify_dirent_safe (rdev->sysfs_state);wait_event_timeout (rdev->blocked_wait, !test_bit (Blocked, &rdev->flags),msecs_to_jiffies (5000 ));
drivers/md/md.h +12 -0
@@ -373,6 +373,18 @@ struct md_sysfs_entry { };extern struct attribute_group md_bitmap_group; +static inline struct sysfs_dirent *sysfs_get_dirent_safe (struct sysfs_dirent *sd,char *name) +{ +if (sd) +return sysfs_get_dirent (sd, name); +return sd; +} +static inline void sysfs_notify_dirent_safe (struct sysfs_dirent *sd) +{ +if (sd) +sysfs_notify_dirent (sd); +} +static inline char *mdname (mddev_t * mddev) {return mddev->gendisk ? mddev->gendisk->disk_name :"mdX" ;
drivers/md/raid5.c +3 -5
@@ -5085,7 +5085,8 @@ static int run(mddev_t *mddev)/* Ok, everything is just fine now */ if (mddev->to_remove == &raid5_attrs_group) mddev->to_remove =NULL ; -else if (sysfs_create_group (&mddev->kobj, &raid5_attrs_group)) +else if (mddev->kobj.sd && +sysfs_create_group (&mddev->kobj, &raid5_attrs_group))printk (KERN_WARNING "raid5: failed to create sysfs attributes for %s\n" ,mdname (mddev));
@@ -5466,10 +5467,7 @@ static int raid5_start_reshape(mddev_t *mddev)sprintf (nm,"rd%d" , rdev->raid_disk);if (sysfs_create_link (&mddev->kobj, &rdev->kobj, nm)) -printk (KERN_WARNING -"raid5: failed to create " -" link %s for %s\n" , - nm,mdname (mddev)); +/* Failure here is OK */ ; }else break ; }
PATCH 2/12 md/raid5: factor out code for changing size of stripe cache.
In reply to NeilBrownSeparate the actual 'change' code from the sysfs interface so that it can eventually be called internally.
--- drivers/md/raid5.c | 38 +++++++++++++++++++++++++------------- 1 files changed, 25 insertions(+), 13 deletions(-)
drivers/md/raid5.c +25 -13
@@ -4487,23 +4487,15 @@ raid5_show_stripe_cache_size(mddev_t *mddev, char *page)return 0 ; } -static ssize_t -raid5_store_stripe_cache_size (mddev_t *mddev,const char *page,size_t len) +static int +raid5_set_cache_size (mddev_t *mddev,int size) {raid5_conf_t *conf = mddev->private; -unsigned long new;int err; -if (len >=PAGE_SIZE ) +if (size <=16 || size >32768 )return -EINVAL ; -if (!conf) -return -ENODEV ; - -if (strict_strtoul (page,10 , &new)) -return -EINVAL ; -if (new <=16 || new >32768 ) -return -EINVAL ; -while (new < conf->max_nr_stripes) { +while (size < conf->max_nr_stripes) {if (drop_one_stripe (conf)) conf->max_nr_stripes--;else
@@ -4512,11 +4504,31 @@ raid5_store_stripe_cache_size(mddev_t *mddev, const char *page, size_t len) err =md_allow_write (mddev);if (err)return err; -while (new > conf->max_nr_stripes) { +while (size > conf->max_nr_stripes) {if (grow_one_stripe (conf)) conf->max_nr_stripes++;else break ; } +return 0 ; +} + +static ssize_t +raid5_store_stripe_cache_size (mddev_t *mddev,const char *page,size_t len) +{ +raid5_conf_t *conf = mddev->private; +unsigned long new; +int err; + +if (len >=PAGE_SIZE ) +return -EINVAL ; +if (!conf) +return -ENODEV ; + +if (strict_strtoul (page,10 , &new)) +return -EINVAL ; + err =raid5_set_cache_size (mddev, new); +if (err) +return err;return len; }
PATCH 3/12 md/dm: create dm-raid456 module using md/raid5
In reply to NeilBrown--- drivers/md/Kconfig | 8 + drivers/md/Makefile | 1 drivers/md/dm-raid456.c | 436 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/md/md.c | 42 +++-- drivers/md/md.h | 4 5 files changed, 474 insertions(+), 17 deletions(-) create mode 100644 drivers/md/dm-raid456.c
drivers/md/Kconfig +8 -0
@@ -256,6 +256,14 @@ config DM_MIRROR Allow volume managers to mirror logical volumes, also needed for live data migration tools such as 'pvmove'. +config DM_RAID456 + tristate "RAID 4/5/6 target (EXPERIMENTAL)" + depends on BLK_DEV_DM && MD_RAID456 && EXPERIMENTAL + ---help--- + A dm target that supports RAID4 RAID5 and RAID6 mapping + + If unsure, say N. + config DM_LOG_USERSPACE tristate "Mirror userspace logging (EXPERIMENTAL)" depends on DM_MIRROR && EXPERIMENTAL && NET
drivers/md/Makefile +1 -0
@@ -44,6 +44,7 @@ obj-$(CONFIG_DM_SNAPSHOT) += dm-snapshot.o obj-$(CONFIG_DM_MIRROR) += dm-mirror.o dm-log.o dm-region-hash.o obj-$(CONFIG_DM_LOG_USERSPACE) += dm-log-userspace.o obj-$(CONFIG_DM_ZERO) += dm-zero.o +obj-$(CONFIG_DM_RAID456) += dm-raid456.o quiet_cmd_unroll = UNROLL $@ cmd_unroll = $(AWK) -f$(srctree)/$(src)/unroll.awk -vN=$(UNROLL) \
drivers/md/dm-raid456.c new file +436 -0
436 lines not on this page · raw message
drivers/md/md.c +25 -17
102 lines not on this page · raw message
drivers/md/md.h +4 -0
@@ -478,4 +478,8 @@ extern void md_integrity_add_rdev(mdk_rdev_t *rdev, mddev_t *mddev);extern int strict_strtoul_scaled (const char *cp,unsigned long *res,int scale);extern void restore_bitmap_write_access (struct file *file); +extern void mddev_init (mddev_t *mddev); +extern int md_run (mddev_t *mddev); +extern void md_stop (mddev_t *mddev); +extern void md_rdev_init (mdk_rdev_t *rdev);#endif /* _MD_MD_H */
PATCH 4/12 dm-raid456: add support for raising events to userspace.
In reply to NeilBrownUserspace needs to know about failure events. DM handles though through the DM_DEV_WAIT_CMD ioctl.
So allow md_error to be given some work to do on an error, and arrange that work to signal dm.
--- drivers/md/dm-raid456.c | 8 ++++++++ drivers/md/md.c | 2 ++ drivers/md/md.h | 1 + 3 files changed, 11 insertions(+), 0 deletions(-)
drivers/md/dm-raid456.c +8 -0
@@ -138,6 +138,13 @@ static int dev_parms(struct raid_set *rs, char **argv)return 0 ; } +static void do_table_event (struct work_struct *ws) +{ + struct raid_set *rs = container_of(ws, struct raid_set, + md.event_work); +dm_table_event (rs->ti->table); +} +/* * Construct a RAID4/5/6 mapping: * Args:
@@ -289,6 +296,7 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv)if (rs->md.raid_disks - in_sync > rt->parity_devs) goto err; +INIT_WORK (&rs->md.event_work, do_table_event); ti->split_io = rs->md.chunk_sectors; ti->private = rs;
drivers/md/md.c +2 -0
@@ -6005,6 +6005,8 @@ void md_error(mddev_t *mddev, mdk_rdev_t *rdev)set_bit (MD_RECOVERY_INTR , &mddev->recovery);set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);md_wakeup_thread (mddev->thread); +if (mddev->event_work.func) +schedule_work (&mddev->event_work);md_new_event_inintr (mddev); }
drivers/md/md.h +1 -0
@@ -309,6 +309,7 @@ struct mddev_sstruct bio *barrier;atomic_t flush_pending;struct work_struct barrier_work; +struct work_struct event_work; };
PATCH 5/12 raid5: Don't set read-ahead when there is no queue
In reply to NeilBrowndm-raid456 does not provide a 'queue' for raid5 to use, so we must make raid5 stop depending on the queue.
First: read_ahead
dm handles read-ahead adjustment fully in userspace, so simply don't do any readahead adjustments if there is no queue.
Also re-arrange code slightly so all the accesses to ->queue are together.
Finally, move the blk_queue_merge_bvec function into the 'if' as the ->split_io setting in dm-raid456 has the same effect.
--- drivers/md/raid5.c | 30 ++++++++++++++++-------------- 1 files changed, 16 insertions(+), 14 deletions(-)
drivers/md/raid5.c +16 -14
@@ -5083,16 +5083,6 @@ static int run(mddev_t *mddev) "reshape"); } -/* read-ahead size must cover two whole stripes, which is -* 2 * (datadisks) * chunksize where 'n' is the number of raid devices -*/ - { -int data_disks = conf->previous_raid_disks - conf->max_degraded; -int stripe = data_disks * - ((mddev->chunk_sectors <<9 ) /PAGE_SIZE ); -if (mddev->queue->backing_dev_info.ra_pages <2 * stripe) - mddev->queue->backing_dev_info.ra_pages =2 * stripe; - }/* Ok, everything is just fine now */ if (mddev->to_remove == &raid5_attrs_group)
@@ -5102,6 +5092,21 @@ static int run(mddev_t *mddev)printk (KERN_WARNING "raid5: failed to create sysfs attributes for %s\n" ,mdname (mddev)); +md_set_array_sectors (mddev,raid5_size (mddev,0 ,0 )); + +if (mddev->queue) { +/* read-ahead size must cover two whole stripes, which +* is 2 * (datadisks) * chunksize where 'n' is the +* number of raid devices +*/ +int data_disks = conf->previous_raid_disks - conf->max_degraded; +int stripe = data_disks * + ((mddev->chunk_sectors <<9 ) /PAGE_SIZE ); +if (mddev->queue->backing_dev_info.ra_pages <2 * stripe) + mddev->queue->backing_dev_info.ra_pages =2 * stripe; + +blk_queue_merge_bvec (mddev->queue, raid5_mergeable_bvec); + } mddev->queue->queue_lock = &conf->device_lock;
@@ -5109,9 +5114,6 @@ static int run(mddev_t *mddev) mddev->queue->backing_dev_info.congested_data = mddev; mddev->queue->backing_dev_info.congested_fn = raid5_congested; -md_set_array_sectors (mddev,raid5_size (mddev,0 ,0 )); - -blk_queue_merge_bvec (mddev->queue, raid5_mergeable_bvec); chunk_size = mddev->chunk_sectors <<9 ;blk_queue_io_min (mddev->queue, chunk_size); blk_queue_io_opt(mddev->queue, chunk_size *
@@ -5534,7 +5536,7 @@ static void end_reshape(raid5_conf_t *conf)/* read-ahead size must cover two whole stripes, which is * 2 * (datadisks) * chunksize where 'n' is the number of raid devices */ - { +if (conf->mddev->queue) {int data_disks = conf->raid_disks - conf->max_degraded;int stripe = data_disks * ((conf->chunk_sectors <<9 ) /PAGE_SIZE );
PATCH 6/12 dm-raid456: add congestion checking.
In reply to NeilBrowndm currently implements congestion checking by checking on congestion in each component device.
For raid456 we need to also check if the stripe cache is congested. So add support to dm for a target to register a congestion checker, then registered such a checker for dm-raid456.
We add support for multiple callbacks as we will need one for unplug too.
Finally, we move the setting for congested_fn for the mddev->queue into the "if (mddev->queue)" protected branch as it is not needed for dm-raid456 now.
--- drivers/md/dm-raid456.c | 13 +++++++++++++ drivers/md/dm-table.c | 15 +++++++++++++++ drivers/md/raid5.c | 22 +++++++++++++++------- drivers/md/raid5.h | 1 + include/linux/device-mapper.h | 12 ++++++++++++ 5 files changed, 56 insertions(+), 7 deletions(-)
drivers/md/dm-raid456.c +13 -0
@@ -16,6 +16,7 @@ struct raid_set {struct dm_target *ti;struct mddev_s md;struct raid_type *raid_type; +struct target_callbacks callbacks;struct raid_dev dev[0 ]; };
@@ -145,6 +146,13 @@ static void do_table_event(struct work_struct *ws)dm_table_event (rs->ti->table); } +static int raid_is_congested (void *v,int bits) +{ +struct target_callbacks *cb = v; + struct raid_set *rs = container_of(cb, struct raid_set, + callbacks); +return md_raid5_congested (&rs->md, bits); +}/* * Construct a RAID4/5/6 mapping: * Args:
@@ -308,6 +316,10 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv)if (errnum) goto err; + + rs->callbacks.congested_fn = raid_is_congested; +dm_table_add_callbacks (ti->table, &rs->callbacks); +return 0 ; err:if (rs)
@@ -320,6 +332,7 @@ static void raid_dtr(struct dm_target *ti) {struct raid_set *rs = ti->private; +list_del_init (&rs->callbacks.list);md_stop (&rs->md);context_free (rs); }
drivers/md/dm-table.c +15 -0
@@ -68,6 +68,8 @@ struct dm_table {void (*event_fn)(void *);void *event_context; +struct list_head target_callbacks; +struct dm_md_mempools *mempools; };
@@ -202,6 +204,7 @@ int dm_table_create(struct dm_table **result, fmode_t mode,return -ENOMEM ;INIT_LIST_HEAD (&t->devices); +INIT_LIST_HEAD (&t->target_callbacks);atomic_set (&t->holders,0 );if (!num_targets)
@@ -1174,10 +1177,18 @@ int dm_table_resume_targets(struct dm_table *t)return 0 ; } +void dm_table_add_callbacks (struct dm_table *t, +struct target_callbacks *cb) +{ +list_add (&cb->list, &t->target_callbacks); +} +EXPORT_SYMBOL_GPL (dm_table_add_callbacks); +int dm_table_any_congested (struct dm_table *t,int bdi_bits) {struct dm_dev_internal *dd;struct list_head *devices =dm_table_get_devices (t); +struct target_callbacks *cb;int r =0 ;list_for_each_entry (dd, devices, list) {
@@ -1192,6 +1203,10 @@ int dm_table_any_congested(struct dm_table *t, int bdi_bits) bdevname(dd->dm_dev.bdev, b)); } +list_for_each_entry (cb, &t->target_callbacks, list) +if (cb->congested_fn) + r |= cb->congested_fn (cb, bdi_bits); +return r; }
drivers/md/raid5.c +15 -7
@@ -3595,17 +3595,14 @@ static void raid5_unplug_device(struct request_queue *q)unplug_slaves (mddev); } -static int raid5_congested (void *data,int bits) +int md_raid5_congested (mddev_t *mddev,int bits) { -mddev_t *mddev = data;raid5_conf_t *conf = mddev->private;/* No difference between reads and writes. Just check * how busy the stripe_cache is */ -if (mddev_congested (mddev, bits)) -return 1 ;if (conf->inactive_blocked)return 1 ;if (conf->quiesce)
@@ -3615,6 +3612,15 @@ static int raid5_congested(void *data, int bits)return 0 ; } +EXPORT_SYMBOL_GPL (md_raid5_congested); + +static int raid5_congested (void *data,int bits) +{ +mddev_t *mddev = data; + +return mddev_congested (mddev, bits) || +md_raid5_congested (mddev, bits); +}/* We want read requests to align with chunks where possible, * but write requests don't need to.
@@ -5106,13 +5112,14 @@ static int run(mddev_t *mddev) mddev->queue->backing_dev_info.ra_pages =2 * stripe;blk_queue_merge_bvec (mddev->queue, raid5_mergeable_bvec); + + mddev->queue->backing_dev_info.congested_data = mddev; + mddev->queue->backing_dev_info.congested_fn = raid5_congested; } mddev->queue->queue_lock = &conf->device_lock; mddev->queue->unplug_fn = raid5_unplug_device; - mddev->queue->backing_dev_info.congested_data = mddev; - mddev->queue->backing_dev_info.congested_fn = raid5_congested; chunk_size = mddev->chunk_sectors <<9 ;blk_queue_io_min (mddev->queue, chunk_size);
@@ -5144,7 +5151,8 @@ static int stop(mddev_t *mddev)md_unregister_thread (mddev->thread); mddev->thread =NULL ; - mddev->queue->backing_dev_info.congested_fn =NULL ; +if (mddev->queue) + mddev->queue->backing_dev_info.congested_fn =NULL ;blk_sync_queue (mddev->queue);/* the unplug fn references 'conf'*/ free_conf (conf); mddev->private =NULL ;
drivers/md/raid5.h +1 -0
@@ -497,4 +497,5 @@ static inline int algorithm_is_DDF(int layout) {return layout >=8 && layout <=10 ; } +extern int md_raid5_congested (mddev_t *mddev,int bits);#endif
include/linux/device-mapper.h +12 -0
@@ -187,6 +187,12 @@ struct dm_target {char *error; }; +/* Each target can link one of these into the table */ +struct target_callbacks { +struct list_head list; +congested_fn *congested_fn; +}; +int dm_register_target (struct target_type *t);void dm_unregister_target (struct target_type *t);
@@ -263,6 +269,12 @@ int dm_table_add_target(struct dm_table *t, const char *type, sector_t start, sector_t len, char *params);/* +* Target_ctr should call this if they need to add any +* callback +*/ +void dm_table_add_callbacks (struct dm_table *t, +struct target_callbacks *cb); +/* * Finally call this to make the table ready for use. */ int dm_table_complete (struct dm_table *t);
PATCH 7/12 md/raid5: add simple plugging infrastructure.
In reply to NeilBrownmd/raid5 uses the plugging infrastructure provided by the block layer and 'struct request_queue'. However when we plug raid5 under dm there is no request queue so we cannot use that.
So create a similar infrastructure that is much lighter weight and use it for raid5.
--- drivers/md/md.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ drivers/md/md.h | 21 +++++++++++++++++++++ drivers/md/raid5.c | 39 +++++++++++++++++++++++++-------------- drivers/md/raid5.h | 3 +++ 4 files changed, 94 insertions(+), 14 deletions(-)
drivers/md/md.c +45 -0
@@ -383,6 +383,51 @@ void md_barrier_request(mddev_t *mddev, struct bio *bio) }EXPORT_SYMBOL (md_barrier_request); +/* Support for plugging. +* This mirrors the plugging support in request_queue, but does not +* require having a whole queue +*/ +static void plugger_work (struct work_struct *work) +{ +struct plug_handle *plug = + container_of(work, struct plug_handle, unplug_work); + plug->unplug_fn (plug); +} +static void plugger_timeout (unsigned long data) +{ +struct plug_handle *plug = (void *)data; +kblockd_schedule_work (NULL , &plug->unplug_work); +} +void plugger_init (struct plug_handle *plug, +void (*unplug_fn)(struct plug_handle *)) +{ + plug->unplug_flag =0 ; + plug->unplug_fn = unplug_fn; +init_timer (&plug->unplug_timer); + plug->unplug_timer.function = plugger_timeout; + plug->unplug_timer.data = (unsigned long )plug; +INIT_WORK (&plug->unplug_work, plugger_work); +} +EXPORT_SYMBOL_GPL (plugger_init); + +void plugger_set_plug (struct plug_handle *plug) +{ +if (!test_and_set_bit (PLUGGED_FLAG , &plug->unplug_flag)) +mod_timer (&plug->unplug_timer, jiffies +msecs_to_jiffies (3 )+1 ); +} +EXPORT_SYMBOL_GPL (plugger_set_plug); + +int plugger_remove_plug (struct plug_handle *plug) +{ +if (test_and_clear_bit (PLUGGED_FLAG , &plug->unplug_flag)) { +del_timer (&plug->unplug_timer); +return 1 ; + }else +return 0 ; +} +EXPORT_SYMBOL_GPL (plugger_remove_plug); + +static inline mddev_t *mddev_get (mddev_t *mddev) {atomic_inc (&mddev->active);
drivers/md/md.h +21 -0
@@ -29,6 +29,26 @@typedef struct mddev_s mddev_t ;typedef struct mdk_rdev_s mdk_rdev_t ; +/* generic plugging support - like that provided with request_queue, +* but does not require a request_queue +*/ +struct plug_handle { +void (*unplug_fn)(struct plug_handle *); +struct timer_list unplug_timer; +struct work_struct unplug_work; +unsigned long unplug_flag; +}; +#define PLUGGED_FLAG 1 +void plugger_init (struct plug_handle *plug, +void (*unplug_fn)(struct plug_handle *)); +void plugger_set_plug (struct plug_handle *plug); +int plugger_remove_plug (struct plug_handle *plug); +static inline void plugger_flush (struct plug_handle *plug) +{ +del_timer_sync (&plug->unplug_timer); +cancel_work_sync (&plug->unplug_work); +} +/* * MD's 'extended' device */
@@ -449,6 +469,7 @@ static inline void safe_put_page(struct page *p)if (p)put_page (p); } +extern int register_md_personality (struct mdk_personality *p);extern int unregister_md_personality (struct mdk_personality *p); extern mdk_thread_t * md_register_thread(void (*run) (mddev_t *mddev),
drivers/md/raid5.c +25 -14
@@ -200,11 +200,11 @@ static void __release_stripe(raid5_conf_t *conf, struct stripe_head *sh)if (test_bit (STRIPE_HANDLE , &sh->state)) {if (test_bit (STRIPE_DELAYED , &sh->state)) {list_add_tail (&sh->lru, &conf->delayed_list); -blk_plug_device (conf->mddev->queue); +plugger_set_plug (&conf->plug); }else if (test_bit (STRIPE_BIT_DELAY , &sh->state) && sh->bm_seq - conf->seq_write >0 ) {list_add_tail (&sh->lru, &conf->bitmap_list); -blk_plug_device (conf->mddev->queue); +plugger_set_plug (&conf->plug); }else {clear_bit (STRIPE_BIT_DELAY , &sh->state);list_add_tail (&sh->lru, &conf->handle_list);
@@ -364,7 +364,7 @@ static struct stripe_head *__find_stripe(raid5_conf_t *conf, sector_t sector, }static void unplug_slaves (mddev_t *mddev); -static void raid5_unplug_device (struct request_queue *q); +static void raid5_unplug_device (raid5_conf_t *conf);static struct stripe_head * get_active_stripe(raid5_conf_t *conf, sector_t sector,
@@ -394,7 +394,7 @@ get_active_stripe(raid5_conf_t *conf, sector_t sector, < (conf->max_nr_stripes *3/4) || !conf->inactive_blocked), conf->device_lock, -raid5_unplug_device (conf->mddev->queue) +raid5_unplug_device (conf) ); conf->inactive_blocked =0 ; }else
@@ -3535,7 +3535,7 @@ static void raid5_activate_delayed(raid5_conf_t *conf)list_add_tail (&sh->lru, &conf->hold_list); } }else -blk_plug_device (conf->mddev->queue); +plugger_set_plug (&conf->plug); }static void activate_bit_delay (raid5_conf_t *conf)
@@ -3576,23 +3576,33 @@ static void unplug_slaves(mddev_t *mddev)rcu_read_unlock (); } -static void raid5_unplug_device (struct request_queue *q) +static void raid5_unplug_device (raid5_conf_t *conf) { -mddev_t *mddev = q->queuedata; -raid5_conf_t *conf = mddev->private;unsigned long flags;spin_lock_irqsave (&conf->device_lock, flags); -if (blk_remove_plug (q)) { +if (plugger_remove_plug (&conf->plug)) { conf->seq_flush++;raid5_activate_delayed (conf); } -md_wakeup_thread (mddev->thread); +md_wakeup_thread (conf->mddev->thread);spin_unlock_irqrestore (&conf->device_lock, flags); -unplug_slaves (mddev); +unplug_slaves (conf->mddev); +} + +static void raid5_unplug (struct plug_handle *plug) +{ +raid5_conf_t *conf =container_of (plug, raid5_conf_t, plug); +raid5_unplug_device (conf); +} + +static void raid5_unplug_queue (struct request_queue *q) +{ +mddev_t *mddev = q->queuedata; +raid5_unplug_device (mddev->private); }int md_raid5_congested (mddev_t *mddev,int bits)
@@ -4002,7 +4012,7 @@ static int make_request(mddev_t *mddev, struct bio * bi)* add failed due to overlap. Flush everything * and wait a while */ -raid5_unplug_device (mddev->queue); +raid5_unplug_device (conf);release_stripe (sh);schedule (); goto retry;
@@ -5100,6 +5110,7 @@ static int run(mddev_t *mddev) mdname(mddev));md_set_array_sectors (mddev,raid5_size (mddev,0 ,0 )); +plugger_init (&conf->plug, raid5_unplug);if (mddev->queue) {/* read-ahead size must cover two whole stripes, which * is 2 * (datadisks) * chunksize where 'n' is the
@@ -5119,7 +5130,7 @@ static int run(mddev_t *mddev) mddev->queue->queue_lock = &conf->device_lock; - mddev->queue->unplug_fn = raid5_unplug_device; + mddev->queue->unplug_fn = raid5_unplug_queue; chunk_size = mddev->chunk_sectors <<9 ;blk_queue_io_min (mddev->queue, chunk_size);
@@ -5153,7 +5164,7 @@ static int stop(mddev_t *mddev) mddev->thread =NULL ;if (mddev->queue) mddev->queue->backing_dev_info.congested_fn =NULL ; -blk_sync_queue (mddev->queue);/* the unplug fn references 'conf'*/ +plugger_flush (&conf->plug);/* the unplug fn references 'conf'*/ free_conf (conf); mddev->private =NULL ; mddev->to_remove = &raid5_attrs_group;
drivers/md/raid5.h +3 -0
@@ -398,6 +398,9 @@ struct raid5_private_data {* (fresh device added). * Cleared when a sync completes. */ + +struct plug_handle plug; +/* per cpu variables */ struct raid5_percpu {struct page *spare_page;/* Used when checking P/Q in raid6 */
PATCH 8/12 md/plug: optionally use plugger to unplug an array during resync/recovery.
In reply to NeilBrownIf an array doesn't have a 'queue' then md_do_sync cannot unplug it.
In that case it will have a 'plugger', so make that available to the mddev, and us it to unplug the array if needed.
--- drivers/md/md.c | 15 ++++++++++++--- drivers/md/md.h | 2 ++ drivers/md/raid5.c | 1 + 3 files changed, 15 insertions(+), 3 deletions(-)
drivers/md/md.c +12 -3
@@ -4660,6 +4660,7 @@ static void md_clean(mddev_t *mddev) mddev->bitmap_info.chunksize =0 ; mddev->bitmap_info.daemon_sleep =0 ; mddev->bitmap_info.max_write_behind =0 ; + mddev->plug =NULL ; }static void md_stop_writes (mddev_t *mddev)
@@ -6563,6 +6564,14 @@ int md_allow_write(mddev_t *mddev) }EXPORT_SYMBOL_GPL (md_allow_write); +static void md_unplug (mddev_t *mddev) +{ +if (mddev->queue) +blk_unplug (mddev->queue); +if (mddev->plug) + mddev->plug->unplug_fn (mddev->plug); +} +#define SYNC_MARKS 10#define SYNC_MARK_STEP (3*HZ)void md_do_sync (mddev_t *mddev)
@@ -6741,7 +6750,7 @@ void md_do_sync(mddev_t *mddev) >= mddev->resync_max - mddev->curr_resync_completed )) {/* time to update curr_resync_completed */ -blk_unplug (mddev->queue); +md_unplug (mddev);wait_event (mddev->recovery_wait,atomic_read (&mddev->recovery_active) ==0 ); mddev->curr_resync_completed =
@@ -6818,7 +6827,7 @@ void md_do_sync(mddev_t *mddev)* about not overloading the IO subsystem. (things like an * e2fsck being done on the RAID array should execute fast) */ -blk_unplug (mddev->queue); +md_unplug (mddev);cond_resched (); currspeed = ((unsigned long )(io_sectors-mddev->resync_mark_cnt))/2
@@ -6837,7 +6846,7 @@ void md_do_sync(mddev_t *mddev)* this also signals 'finished resyncing' to md_stop */ out: -blk_unplug (mddev->queue); +md_unplug (mddev);wait_event (mddev->recovery_wait, !atomic_read (&mddev->recovery_active));
drivers/md/md.h +2 -0
@@ -319,6 +319,8 @@ struct mddev_sstruct list_head all_mddevs;struct attribute_group *to_remove; +struct plug_handle *plug;/* if used by personality */ +/* Generic barrier handling. * If there is a pending barrier request, all other * writes are blocked while the devices are flushed.
drivers/md/raid5.c +1 -0
@@ -5111,6 +5111,7 @@ static int run(mddev_t *mddev)md_set_array_sectors (mddev,raid5_size (mddev,0 ,0 ));plugger_init (&conf->plug, raid5_unplug); + mddev->plug = &conf->plug;if (mddev->queue) {/* read-ahead size must cover two whole stripes, which * is 2 * (datadisks) * chunksize where 'n' is the
PATCH 9/12 dm-raid456: support unplug
In reply to NeilBrownIn a similar manner to congestion checking, per-target unplug support for raid456 under dm.
--- drivers/md/dm-raid456.c | 9 +++++++++ drivers/md/dm-table.c | 4 ++++ drivers/md/raid5.c | 11 +++++------ drivers/md/raid5.h | 1 + include/linux/device-mapper.h | 1 + 5 files changed, 20 insertions(+), 6 deletions(-)
drivers/md/dm-raid456.c +9 -0
@@ -153,6 +153,14 @@ static int raid_is_congested(void *v, int bits) callbacks);return md_raid5_congested (&rs->md, bits); } +static void raid_unplug (void *v) +{ +struct target_callbacks *cb = v; + struct raid_set *rs = container_of(cb, struct raid_set, + callbacks); +raid5_unplug_device (rs->md.private); +} +/* * Construct a RAID4/5/6 mapping: * Args:
@@ -288,6 +296,7 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv) goto err;clear_bit (In_sync, &rs->dev[rebuildA].rdev.flags); rs->dev[rebuildA].rdev.recovery_offset =0 ; + rs->callbacks.unplug_fn = raid_unplug; }if (rebuildB >=0 ) {if (rs->dev[rebuildB].dev ==NULL )
drivers/md/dm-table.c +4 -0
@@ -1228,6 +1228,7 @@ void dm_table_unplug_all(struct dm_table *t) {struct dm_dev_internal *dd;struct list_head *devices =dm_table_get_devices (t); +struct target_callbacks *cb;list_for_each_entry (dd, devices, list) {struct request_queue *q =bdev_get_queue (dd->dm_dev.bdev);
@@ -1240,6 +1241,9 @@ void dm_table_unplug_all(struct dm_table *t)dm_device_name (t->md), bdevname(dd->dm_dev.bdev, b)); } +list_for_each_entry (cb, &t->target_callbacks, list) +if (cb->unplug_fn) + cb->unplug_fn (cb); }struct mapped_device *dm_table_get_md (struct dm_table *t)
drivers/md/raid5.c +5 -6
@@ -364,7 +364,6 @@ static struct stripe_head *__find_stripe(raid5_conf_t *conf, sector_t sector, }static void unplug_slaves (mddev_t *mddev); -static void raid5_unplug_device (raid5_conf_t *conf);static struct stripe_head * get_active_stripe(raid5_conf_t *conf, sector_t sector,
@@ -3576,7 +3575,7 @@ static void unplug_slaves(mddev_t *mddev)rcu_read_unlock (); } -static void raid5_unplug_device (raid5_conf_t *conf) +void raid5_unplug_device (raid5_conf_t *conf) {unsigned long flags;
@@ -3592,6 +3591,7 @@ static void raid5_unplug_device(raid5_conf_t *conf)unplug_slaves (conf->mddev); } +EXPORT_SYMBOL_GPL (raid5_unplug_device);static void raid5_unplug (struct plug_handle *plug) {
@@ -5127,11 +5127,10 @@ static int run(mddev_t *mddev) mddev->queue->backing_dev_info.congested_data = mddev; mddev->queue->backing_dev_info.congested_fn = raid5_congested; - } - - mddev->queue->queue_lock = &conf->device_lock; - mddev->queue->unplug_fn = raid5_unplug_queue; + mddev->queue->queue_lock = &conf->device_lock; + mddev->queue->unplug_fn = raid5_unplug_queue; + } chunk_size = mddev->chunk_sectors <<9 ;blk_queue_io_min (mddev->queue, chunk_size);
drivers/md/raid5.h +1 -0
@@ -501,4 +501,5 @@ static inline int algorithm_is_DDF(int layout) return layout >= 8 && layout <= 10; }extern int md_raid5_congested (mddev_t *mddev,int bits); +extern void raid5_unplug_device (raid5_conf_t *conf);#endif
include/linux/device-mapper.h +1 -0
@@ -191,6 +191,7 @@ struct dm_target {struct target_callbacks {struct list_head list;congested_fn *congested_fn; +void (*unplug_fn)(void *); };int dm_register_target (struct target_type *t);
PATCH 10/12 dm-raid456: add support for setting IO hints.
In reply to NeilBrown--- drivers/md/dm-raid456.c | 33 +++++++++++++++++++++++++++++++++ drivers/md/raid5.c | 19 ++++++++++--------- 2 files changed, 43 insertions(+), 9 deletions(-)
drivers/md/dm-raid456.c +33 -0
@@ -434,6 +434,37 @@ static int raid_status(struct dm_target *ti, status_type_t type,return 0 ; } +static int raid_iterate_devices (struct dm_target *ti, +iterate_devices_callout_fn fn, +void *data) +{ +struct raid_set *rs = ti->private; +int ret =0 ; +unsigned i =0 ; + +for (i=0 ; !ret && i < rs->md.raid_disks; i++) +if (rs->dev[i].dev) + ret =fn (ti, + rs->dev[i].dev, + rs->dev[i].rdev.data_offset, + rs->md.dev_sectors, + data); + +return ret; +} + +static void raid_io_hints (struct dm_target *ti, +struct queue_limits *limits) +{ +struct raid_set *rs = ti->private; +unsigned chunk_size = rs->md.chunk_sectors <<9 ; +raid5_conf_t *conf = rs->md.private; + +blk_limits_io_min (limits, chunk_size); +blk_limits_io_opt (limits, chunk_size * + (conf->raid_disks - conf->max_degraded)); +} +static struct target_type raid_target = { .name ="raid45" , .version = {1 ,0 ,0 },
@@ -442,6 +473,8 @@ static struct target_type raid_target = { .dtr = raid_dtr, .map = raid_map, .status = raid_status, + .iterate_devices = raid_iterate_devices, + .io_hints = raid_io_hints, };static int __init dm_raid_init (void )
drivers/md/raid5.c +10 -9
@@ -4906,7 +4906,7 @@ static int only_parity(int raid_disk, int algo, int raid_disks, int max_degradedstatic int run (mddev_t *mddev) {raid5_conf_t *conf; -int working_disks =0 , chunk_size; +int working_disks =0 ;int dirty_parity_disks =0 ;mdk_rdev_t *rdev;sector_t reshape_offset =0 ;
@@ -5113,6 +5113,7 @@ static int run(mddev_t *mddev)plugger_init (&conf->plug, raid5_unplug); mddev->plug = &conf->plug;if (mddev->queue) { +int chunk_size;/* read-ahead size must cover two whole stripes, which * is 2 * (datadisks) * chunksize where 'n' is the * number of raid devices
@@ -5130,16 +5131,16 @@ static int run(mddev_t *mddev) mddev->queue->queue_lock = &conf->device_lock; mddev->queue->unplug_fn = raid5_unplug_queue; - } - chunk_size = mddev->chunk_sectors <<9 ; -blk_queue_io_min (mddev->queue, chunk_size); -blk_queue_io_opt (mddev->queue, chunk_size * - (conf->raid_disks - conf->max_degraded)); + chunk_size = mddev->chunk_sectors <<9 ; +blk_queue_io_min (mddev->queue, chunk_size); +blk_queue_io_opt (mddev->queue, chunk_size * + (conf->raid_disks - conf->max_degraded)); -list_for_each_entry (rdev, &mddev->disks, same_set) -disk_stack_limits (mddev->gendisk, rdev->bdev, - rdev->data_offset <<9 ); +list_for_each_entry (rdev, &mddev->disks, same_set) +disk_stack_limits (mddev->gendisk, rdev->bdev, + rdev->data_offset <<9 ); + }return 0 ; abort:
PATCH 11/12 dm-raid456: add suspend/resume method
In reply to NeilBrownThese just call in to the md methods.
--- drivers/md/dm-raid456.c | 15 +++++++++++++++ drivers/md/md.c | 6 ++++-- drivers/md/md.h | 3 +++ 3 files changed, 22 insertions(+), 2 deletions(-)
drivers/md/dm-raid456.c +15 -0
@@ -465,6 +465,19 @@ static void raid_io_hints(struct dm_target *ti, (conf->raid_disks - conf->max_degraded)); } +static void raid_presuspend (struct dm_target *ti) +{ +struct raid_set *rs = ti->private; +mddev_suspend (&rs->md); +} + +static void raid_resume (struct dm_target *ti) +{ +struct raid_set *rs = ti->private; + +mddev_resume (&rs->md); +} +static struct target_type raid_target = { .name ="raid45" , .version = {1 ,0 ,0 },
@@ -475,6 +488,8 @@ static struct target_type raid_target = { .status = raid_status, .iterate_devices = raid_iterate_devices, .io_hints = raid_io_hints, + .presuspend = raid_presuspend, + .resume = raid_resume, };static int __init dm_raid_init (void )
drivers/md/md.c +4 -2
@@ -260,7 +260,7 @@ static int md_make_request(struct request_queue *q, struct bio *bio)* Once ->stop is called and completes, the module will be completely * unused. */ -static void mddev_suspend (mddev_t *mddev) +void mddev_suspend (mddev_t *mddev) {BUG_ON (mddev->suspended); mddev->suspended =1 ;
@@ -268,13 +268,15 @@ static void mddev_suspend(mddev_t *mddev)wait_event (mddev->sb_wait,atomic_read (&mddev->active_io) ==0 ); mddev->pers->quiesce (mddev,1 ); } +EXPORT_SYMBOL_GPL (mddev_suspend); -static void mddev_resume (mddev_t *mddev) +void mddev_resume (mddev_t *mddev) { mddev->suspended =0 ;wake_up (&mddev->sb_wait); mddev->pers->quiesce (mddev,0 ); } +EXPORT_SYMBOL_GPL (mddev_resume);int mddev_congested (mddev_t *mddev,int bits) {
drivers/md/md.h +3 -0
@@ -506,4 +506,7 @@ extern void mddev_init(mddev_t *mddev);extern int md_run (mddev_t *mddev);extern void md_stop (mddev_t *mddev);extern void md_rdev_init (mdk_rdev_t *rdev); + +extern void mddev_suspend (mddev_t *mddev); +extern void mddev_resume (mddev_t *mddev);#endif /* _MD_MD_H */
PATCH 12/12 dm-raid456: add message handler.
In reply to NeilBrownSupport messages to:
- change the size of the stripe cache
- change the speed limiter on resync.
--- drivers/md/dm-raid456.c | 26 ++++++++++++++++++++++++++ drivers/md/raid5.c | 3 ++- drivers/md/raid5.h | 1 + 3 files changed, 29 insertions(+), 1 deletions(-)
drivers/md/dm-raid456.c +26 -0
@@ -478,6 +478,31 @@ static void raid_resume(struct dm_target *ti)mddev_resume (&rs->md); } +/* Parse and handle a message from userspace +* Messages are: +* stripecache N (pages per devices) +* minspeed N (kibibytes per seconds) +*/ +static int raid_message (struct dm_target *ti,unsigned argc,char **argv) +{ +struct raid_set *rs = ti->private; + +if (argc ==2 &&strcmp (argv[0 ],"stripecache" ) ==0 ) { +unsigned long size; +if (strict_strtoul (argv[1 ],10 , &size)) +return -EINVAL ; +return raid5_set_cache_size (&rs->md, size); + } +if (argc ==2 &&strcmp (argv[0 ],"minspeed" ) ==0 ) { +unsigned long speed; +if (strict_strtoul (argv[1 ],10 , &speed)) +return -EINVAL ; + rs->md.sync_speed_min = speed; +return 0 ; + } +return -EINVAL ; +} +static struct target_type raid_target = { .name ="raid45" , .version = {1 ,0 ,0 },
@@ -490,6 +515,7 @@ static struct target_type raid_target = { .io_hints = raid_io_hints, .presuspend = raid_presuspend, .resume = raid_resume, + .message = raid_message, };static int __init dm_raid_init (void )
drivers/md/raid5.c +2 -1
@@ -4503,7 +4503,7 @@ raid5_show_stripe_cache_size(mddev_t *mddev, char *page)return 0 ; } -static int +int raid5_set_cache_size (mddev_t *mddev,int size) {raid5_conf_t *conf = mddev->private;
@@ -4527,6 +4527,7 @@ raid5_set_cache_size(mddev_t *mddev, int size) }return 0 ; } +EXPORT_SYMBOL (raid5_set_cache_size);static ssize_t raid5_store_stripe_cache_size (mddev_t *mddev,const char *page,size_t len)
drivers/md/raid5.h +1 -0
@@ -502,4 +502,5 @@ static inline int algorithm_is_DDF(int layout) }extern int md_raid5_congested (mddev_t *mddev,int bits);extern void raid5_unplug_device (raid5_conf_t *conf); +extern int raid5_set_cache_size (mddev_t *mddev,int size);#endif
Quoted 2 lines · NeilBrown
Would the dm community be interested in including this work upstream (after suitable review and testing)?
I certainly hope so. I was able to get this to the "it works" stage, and the overall effort is something upstream has needed for a long while.
Jeff
Hi Neil,
had a first go reading through your patch series w/o finding any major issues. The only important feature for an initial release which needs adding (as you mentioned) is (persistent) dirty log support.
Because you're using a persistent bitmap in the MD RAID personalities, this looks like a bit more surgery to factor it out to potentially enhance dm-log.c. For an initial solution we can as well just go with MDs existing bitmap while keeping the dm-raid456 ctr support for explicit dirty logging in order to avoid compatibility issues (there's obviously no parameter to support bitmap chunk sizes so far).
Reshaping could be triggered either preferably via the constructor involving MD metadata reads to be able to recognize the size change requested or the message interface. Both ctr/message support could be implemented sharing the same functions. Enhancements in the status interface and dm_table_event() throwing on error/finish are mandatory if we support reshaping.
A shortcoming of this MD wrapping solution vs. dm-raid45 is, that there is no obvious way to leverage it to be a clustered RAID456 mapping target. dm-raid45 has been designed with that future enhancement possibility in mind.
Will try testing your code tomorrow.
Regards,
Heinz
Quoted 83 lines · NeilBrown
Greetings Heinz, Alasdair, and all,
(Alasdair and Heinz cc:ed on this intro, but the patches are
only going to the lists).Some months ago I posted a proof-of-concept patch which attempted to provide RAID4/5/6 functionality to 'dm' using
md/raid5.c. While it did a least partly work it contained lots of hacks and was very ugly.I finally made time to do the job "properly".
The following series, when applied on top of a bunch of patches I just submitted for linux-next, provides a 'dm-raid45' target which is largely compatible with the one that Heinz has written (and several distros are shipping), but which uses
md/raid5.cfor the core IO processing.I have tried to split the patch up into easy-to-handle pieces. You will note that some changes to core-dm are required, in particular to pass back 'congestion' information and to handle plugging (which raid5 uses to improve throughput). I hope the approach I have taken is suitable, but it can obviously be changed if necessary.
The create/status/message interface differs from the one in Heinz's patch, but should be close enough to work with current 'dmraid'.
If you want to try the patches (rather than just read them) you should probably "git pull" (please don't clone) from
git://neil.brown.name/md md-dm-raid45so as to get all the prior refactoring patches in md.
Some advantages of this over Heinz's patch (at least as it was when I last looked at it) are:
- raid6 support
- support for XOR-offload hardware where present
- less code duplication
- a single dm device can include multiple dm-raid45 targets. (Heinz' code accesses dm_disk(md)->queue directly which is a layering violations and assumes that there is no other target in themapped_device).There is a lot more that could be done to this such as getting to work with a disk based dirty-log and making the reshape options available. But this patch set should provide all basic RAID5 functionality.
Would the dm community be interested in including this work upstream (after suitable review and testing)?
Thanks,
NeilBrown---
NeilBrown (12):
md: reduce dependence on sysfs.
md/raid5: factor out code for changing size of stripe cache. md/dm: create dm-raid456 module using md/raid5
dm-raid456: add support for raising events to userspace.
raid5: Don't set read-ahead when there is no queue
dm-raid456: add congestion checking.
md/raid5: add simple plugging infrastructure.
md/plug: optionally use plugger to unplug an array during resync/recovery.
dm-raid456: support unplug
dm-raid456: add support for setting IO hints.
dm-raid456: add suspend/resume method
dm-raid456: add message handler.drivers/md/Kconfig | 8 + drivers/md/Makefile | 1 drivers/md/dm-raid456.c | 540 +++++++++++++++++++++++++++++++++++++++++ drivers/md/dm-table.c | 19 + drivers/md/md.c | 211 ++++++++++------ drivers/md/md.h | 43 +++ drivers/md/raid5.c | 155 +++++++----- drivers/md/raid5.h | 6 include/linux/device-mapper.h | 13 + 9 files changed, 859 insertions(+), 137 deletions(-) create mode 100644 drivers/md/dm-raid456.c
Quoted 7 lines · On Thu, 15 Apr 2010 19:27:15 +0200 Heinz Mauelshagen <[email protected]> wrote:
Hi Neil,
had a first go reading through your patch series w/o finding any major issues. The only important feature for an initial release which needs adding (as you mentioned) is (persistent) dirty log support.
Quoted 6 lines
Because you're using a persistent bitmap in the MD RAID personalities, this looks like a bit more surgery to factor it out to potentially enhance
dm-log.c. For an initial solution we can as well just go with MDs existing bitmap while keeping the dm-raid456 ctr support for explicit dirty logging in order to avoid compatibility issues (there's obviously no parameter to support bitmap chunk sizes so far).
I don't think we can use md's existing bitmap support as there is no easy way to store it on an arbitrary target: it either lives near the metadata or on a file (not a device).
There a just a few calls in the interface to md/bitmap.c - it shouldn't be too hard to make those selectively call into a dm_dirty_log instead. I want to do something like that anyway as I want to optionally be able to use a dirty log which is a list of dirty sector addresses rather than a bitmap. I'll have a look next week.
And the "bitmap chunk size" is exactly the same as the dm "region size". (which would probably have been a better name to choose for md too).
Quoted 7 lines
Reshaping could be triggered either preferably via the constructor involving MD metadata reads to be able to recognize the size change requested or the message interface. Both ctr/message support could be implemented sharing the same functions. Enhancements in the status interface and
dm_table_event()throwing on error/finish are mandatory if we support reshaping.
I imagine enhancing the constructor to take before/after values for type, disks, chunksize, and a sector which marks where "after" starts. You also need to know which direction the reshape is going (low addresses to high addresses, or the reverse) though that might be implicit in the other values.
Quoted 6 lines
A shortcoming of this MD wrapping solution vs. dm-raid45 is, that there is no obvious way to leverage it to be a clustered RAID456 mapping target. dm-raid45 has been designed with that future enhancement possibility in mind.
I haven't given cluster locking a lot of thought...
I would probably do the locking on a per-"stripe_head" basis as everything revolves around that structure.
Get a shared lock when servicing a read (Which would only happen on a degraded array - normally reads bypass the stripe cache), or a write lock when servicing a write or a resync.
It should all interface with DLM quite well - when DLM tries to reclaim a lock we first mark all the stripe as not up-to-date...
Does DM simply use DLM for locking or something else?
Quoted 1 line
Will try testing your code tomorrow.
Thanks,
NeilBrown
Quoted 17 lines · Neil Brown
Quoted 7 lines · On Thu, 15 Apr 2010 19:27:15 +0200 Heinz Mauelshagen <[email protected]> wrote:
Hi Neil,
had a first go reading through your patch series w/o finding any major issues. The only important feature for an initial release which needs adding (as you mentioned) is (persistent) dirty log support.
Quoted 6 lines
Because you're using a persistent bitmap in the MD RAID personalities, this looks like a bit more surgery to factor it out to potentially enhance
dm-log.c. For an initial solution we can as well just go with MDs existing bitmap while keeping the dm-raid456 ctr support for explicit dirty logging in order to avoid compatibility issues (there's obviously no parameter to support bitmap chunk sizes so far).
Quoted 5 lines
I don't think we can use md's existing bitmap support as there is no easy way to store it on an arbitrary target: it either lives near the metadata or on a file (not a device).
There a just a few calls in the interface tomd/bitmap.c- it shouldn't be too hard to make those selectively call into adm_dirty_loginstead.
Good, it was my thinking if using dm-dirty-log interface, that there are some MD bitmap code valuables we could factor out (bitmap flushing enhancements?).
Quoted 4 lines
I want to do something like that anyway as I want to optionally be able to use a dirty log which is a list of dirty sector addresses rather than a bitmap. I'll have a look next week.
Ok.
Quoted 3 lines
And the "bitmap chunk size" is exactly the same as the dm "region size". (which would probably have been a better name to choose for md too).
Fair enough.
Quoted 9 lines
Quoted 8 lines
Reshaping could be triggered either preferably via the constructor involving MD metadata reads to be able to recognize the size change requested or the message interface. Both ctr/message support could be implemented sharing the same functions. Enhancements in the status interface and
dm_table_event()throwing on error/finish are mandatory if we support reshaping.
Quoted 5 lines
I imagine enhancing the constructor to take before/after values for type, disks, chunksize, and a sector which marks where "after" starts. You also need to know which direction the reshape is going (low addresses to high addresses, or the reverse) though that might be implicit in the other values.
Yes, that can be additional ctr variable parameters allowing for a compatible enhancement.
One possibility could be using variable parameters from free #8 on:
o to_raid_type # may be existing one; eg. raid6_zr
o to_chunk_size # new requested chunk size in sectors
o old_size # actual size of the array
o low_to_high/high_to_low # low->high or high->low addresses
ti->len defines the new intended size while old_size provides the actual size of the array.
Quoted 11 lines
Quoted 7 lines
A shortcoming of this MD wrapping solution vs. dm-raid45 is, that there is no obvious way to leverage it to be a clustered RAID456 mapping target. dm-raid45 has been designed with that future enhancement possibility in mind.
I haven't given cluster locking a lot of thought...
I would probably do the locking on a per-"stripe_head" basis as everything revolves around that structure.
Makes sense. I was also thinking about tying stripe invalidation to lock state changes.
Quoted 3 lines
Get a shared lock when servicing a read (Which would only happen on a degraded array - normally reads bypass the stripe cache), or a write lock when servicing a write or a resync.
Yes, an exclusive DLM lock.
Quoted 2 lines
It should all interface with DLM quite well - when DLM tries to reclaim a lock we first mark all the stripe as not up-to-date...
When a dm-raid45(6) instance tries to reclaim either lock *after* it had to drop it before, it has to invalidate the respective stripe date.
Quoted 2 lines
Does DM simply use DLM for locking or something else?
We don't use the DLM from DM yet, but essentially: yes, you'ld call dlm_new_lockspace(), dlm_lock(..., DLM_LOCK_{CR|EX}, ...), ...
Of course such locking has to be abstracted in dm-raid456 in order to plug in NULL, clustered, locking modules.
Cheers,
Heinz
Quoted 12 lines
Quoted 3 lines
Will try testing your code tomorrow.
Thanks,
NeilBrown
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel