sysfs representation of stacked devices (dm/md) (rev.2)
Subsystems (3)
cover sysfs representation of stacked devices (dm/md) (rev.2)
Hello,
This is a revised set of pathces which provides common representation of dependencies between stacked devices (dm and md) in sysfs.
Variants of bd_claim/bd_release are added to accept a kobject and create symlinks between the claimed bdev and the holder.
dm/md will give a child of its gendisk kobject to bd_claim. For example, if dm-0 maps to sda, we have the following symlinks;
/sys/block/dm-0/slaves/sda --> /sys/block/sda /sys/block/sda/holders/dm-0 --> /sys/block/dm-0
Comments are welcome.
A few points I would appreciate comments/reviews from maintainers:
About sysfs
- I confirmed sysfs_remove_symlink() and kobject_del() don't allocate memory in 2.6.15 and it seems true on the git head. I would like to make sure it's true in future versions of kernel because they are called during device-mapper's table swapping where I/O to free memory could deadlock on the dm device. What is the recommended way to do that?
Or can I just expect these functions will not allocate memory in future versions of kernel?
About dm
- To get a reference to mapped_device, table_load() do dm_get() before populating table. It will dm_put() when the table is being discarded or the table is being activated.
About md
- Rather than carrying mddev pointer around, bd_claim is now made twice. First is not changed at lock_rdev().
The second is at bind_rdev_to_array() where kobject is passed and symlinks are created.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
PATCH 1/3 sysfs representation of stacked devices (common) (rev.2)
This patch adds bd_claim_by_kobject and bd_release_from_kobject which create/remove symlinks between the claimed bdev and the holder.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
Adding bd_claim_by_kobject/bd_release_from_kobject for use by stacked device drivers (dm and md)
include/linux/fs.h +4 -0
@@ -373,6 +373,8 @@ struct block_device {struct list_head bd_inodes;void * bd_holder;int bd_holders; +struct kobject bd_holder_dir; +struct list_head bd_holder_list;struct block_device * bd_contains;unsigned bd_block_size;struct hd_struct * bd_part;
@@ -1351,6 +1353,8 @@ extern int blkdev_get(struct block_devicextern int blkdev_put (struct block_device *);extern int bd_claim (struct block_device *,void *);extern void bd_release (struct block_device *); +extern int bd_claim_by_kobject (struct block_device *,void *,struct kobject *); +extern void bd_release_from_kobject (struct block_device *,struct kobject *);/* fs/char_dev.c */ extern int alloc_chrdev_region (dev_t *,unsigned ,unsigned ,const char *);
include/linux/genhd.h +1 -0
@@ -114,6 +114,7 @@ struct gendisk {int number;/* more of the same */ struct device *driverfs_dev;struct kobject kobj; +struct kobject slave_dir;struct timer_rand_state *random;int policy;
fs/block_dev.c +174 -2
210 lines not on this page · raw message
fs/partitions/check.c +14 -0
@@ -293,6 +293,13 @@ struct kobj_type ktype_part = { .sysfs_ops = &part_sysfs_ops, }; +/* This is a mere directory in sysfs. No methods are needed. */ +static struct kobj_type slave_ktype = { + .release =NULL , + .sysfs_ops =NULL , + .default_attrs =NULL , +}; +void delete_partition (struct gendisk *disk,int part) {struct hd_struct *p = disk->part[part-1 ];
@@ -343,6 +350,12 @@ static void disk_sysfs_symlinks(struct gsysfs_create_link (&disk->kobj,&target->kobj,"device" );sysfs_create_link (&target->kobj,&disk->kobj,"block" ); } + +/* create subdirectory for symlinks to underlying device */ + disk->slave_dir.ktype = &slave_ktype; +kobject_set_name (&disk->slave_dir,"slaves" ); + disk->slave_dir.parent = &disk->kobj; +kobject_register (&disk->slave_dir); }/* Not exported, helper to add_disk(). */
@@ -460,6 +473,7 @@ void del_gendisk(struct gendisk *disk)devfs_remove_disk (disk); +kobject_unregister (&disk->slave_dir);if (disk->driverfs_dev) {sysfs_remove_link (&disk->kobj,"device" );sysfs_remove_link (&disk->driverfs_dev->kobj,"block" );
Re: [PATCH 1/3] sysfs representation of stacked devices (common) (rev.2)
fs/block_dev.c void bd_forget(struct inode *inode) · line 518
Quoted 6 lines · Jun'ichi Nomura
fs/block_dev.c +6
-0+/* This is a mere directory in sysfs. No methods are needed. */ +static struct kobj_type bd_holder_ktype = { + .release =NULL , + .sysfs_ops =NULL , + .default_attrs =NULL , +};
That doesn't look right. You always need a release function.
Quoted 11 lines
fs/block_dev.c +11
-0+static inline void add_holder_dir (struct block_device *bdev) +{ +struct kobject *kobj = &bdev->bd_holder_dir; + + kobj->ktype = &bd_holder_ktype; +kobject_set_name (kobj,"holders" ); + kobj->parent =bdev_get_kobj (bdev); +kobject_init (kobj); +kobject_add (kobj); +kobject_put (kobj->parent); +}
fs/block_dev.c void bd_forget(struct inode *inode) · line 540
Quoted 10 lines
fs/block_dev.c +10
-0+ +static inline void del_holder_dir (struct block_device *bdev) +{ +/* +* Don't kobject_unregister to avoid memory allocation +* in kobject_hotplug. +*/ +kobject_del (&bdev->bd_holder_dir); +kobject_put (&bdev->bd_holder_dir); +}
No, do it correctly please.
thanks,
greg k-h
Re: [PATCH 1/3] sysfs representation of stacked devices (common) (rev.2)
Hi Greg,
Thanks for comments.
Quoted 8 lines · Greg KH
fs/block_dev.c void bd_forget(struct inode *inode) · line 518
Quoted 6 lines
fs/block_dev.c +6
-0+/* This is a mere directory in sysfs. No methods are needed. */ +static struct kobj_type bd_holder_ktype = { + .release =NULL , + .sysfs_ops =NULL , + .default_attrs =NULL , +};That doesn't look right. You always need a release function.
I'll move them out to gendisk/hd_struct creation with proper release function.
I thought it's correct because NULL release function is just ignored in kobject_cleanup() and it let outside function to release the whole structure.
But it seems wrong to embed these additional kobjects in the structures which are logically separate from them.
Quoted 11 lines
fs/block_dev.c void bd_forget(struct inode *inode) · line 540
Quoted 9 lines
fs/block_dev.c +9
-0+static inline void del_holder_dir (struct block_device *bdev) +{ +/* +* Don't kobject_unregister to avoid memory allocation +* in kobject_hotplug. +*/ +kobject_del (&bdev->bd_holder_dir); +kobject_put (&bdev->bd_holder_dir); +}No, do it correctly please.
OK, I'll change them to kobject_unregister() and do it when gendisk/hd_struct is removed.
Then we can avoid possible memory allocation in dm's atomic operation, too.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
Re: [PATCH 1/3] sysfs representation of stacked devices (common) (rev.2)
Quoted 35 lines · Jun'ichi Nomura
Hi Greg,
Thanks for comments.
Quoted 8 lines · Greg KH
fs/block_dev.c void bd_forget(struct inode *inode) · line 518
Quoted 6 lines
fs/block_dev.c +6
-0+/* This is a mere directory in sysfs. No methods are needed. */ +static struct kobj_type bd_holder_ktype = { + .release =NULL , + .sysfs_ops =NULL , + .default_attrs =NULL , +};That doesn't look right. You always need a release function.
I'll move them out to gendisk/hd_struct creation with proper release function.
I thought it's correct because NULL release function is just ignored in
kobject_cleanup()and it let outside function to release the whole structure.
But it seems wrong to embed these additional kobjects in the structures which are logically separate from them.Quoted 11 lines
fs/block_dev.c void bd_forget(struct inode *inode) · line 540
Quoted 9 lines
fs/block_dev.c +9
-0+static inline void del_holder_dir (struct block_device *bdev) +{ +/* +* Don't kobject_unregister to avoid memory allocation +* in kobject_hotplug. +*/ +kobject_del (&bdev->bd_holder_dir); +kobject_put (&bdev->bd_holder_dir); +}No, do it correctly please.
Quoted 4 lines
OK, I'll change them to
kobject_unregister()and do it when gendisk/hd_struct is removed.
Then we can avoid possible memory allocation in dm's atomic operation, too.
That sounds great.
thanks,
greg k-h
Re: [PATCH 1/3] sysfs representation of stacked devices (common) (rev.2)
Hello Greg,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+/* This is a mere directory in sysfs. No methods are needed. */ +static struct kobj_type bd_holder_ktype = { + .release = NULL, + .sysfs_ops = NULL, + .default_attrs = NULL, +};That doesn't look right. You always need a release function.
I updated the patch based your comments. Could you take a look at this version whether there's any problematic use of sysfs/kobjects?
- I removed embedded child-kobjects from struct block_device and struct gendisk which I added in my previous patch. Kobject registration occurs when gendisk or hd_struct is registered. Release function of the kobject type is added.
- Reference counting of kobjects is done in much symmetric manner than before.
- Added bd_claim_by_disk/bd_release_from_disk inline functions to help proper reference counting.
Thanks,
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
This patch provides a common functions/structures for
sysfs representation of dependencies between stacked devices (dm and md).
Variants of bd_claim/bd_release are added to accept a kobject (or a gendisk containing it) and create symlinks between the claimed bdev and the holder.
Other patches for dm/md will make use of these functions.
For example, if dm-0 maps to sda, we have the following symlinks;
/sys/block/dm-0/slaves/sda --> /sys/block/sda
/sys/block/sda/holders/dm-0 --> /sys/block/dm-0fs/block_dev.c | 196 ++++++++++++++++++++++++++++++++++++++++++++++++- fs/partitions/check.c | 55 ++++++++++++++ include/linux/fs.h | 3 include/linux/genhd.h | 16 ++++ 4 files changed, 268 insertions(+), 2 deletions(-)
include/linux/fs.h +3 -0
@@ -401,6 +401,7 @@ struct block_device {struct list_head bd_inodes;void * bd_holder;int bd_holders; +struct list_head bd_holder_list;struct block_device * bd_contains;unsigned bd_block_size;struct hd_struct * bd_part;
@@ -1380,6 +1381,8 @@ extern int blkdev_get(struct block_devicextern int blkdev_put (struct block_device *);extern int bd_claim (struct block_device *,void *);extern void bd_release (struct block_device *); +extern int bd_claim_by_kobject (struct block_device *,void *,struct kobject *); +extern void bd_release_from_kobject (struct block_device *,struct kobject *);/* fs/char_dev.c */ extern int alloc_chrdev_region (dev_t *,unsigned ,unsigned ,const char *);
include/linux/genhd.h +16 -0
@@ -78,6 +78,7 @@ struct hd_struct {sector_t start_sect;sector_t nr_sects;struct kobject kobj; +struct kobject *holder_dir;unsigned ios[2 ], sectors[2 ];/* READs and WRITEs */ int policy, partno; };
@@ -114,6 +115,8 @@ struct gendisk {int number;/* more of the same */ struct device *driverfs_dev;struct kobject kobj; +struct kobject *holder_dir; +struct kobject *slave_dir;struct timer_rand_state *random;int policy;
@@ -421,6 +424,19 @@ static inline struct block_device *bdgetreturn bdget (MKDEV (disk->major, disk->first_minor) + index); } +static inline int bd_claim_by_disk (struct block_device *bdev, +void *holder,struct gendisk *disk) +{ +return bd_claim_by_kobject (bdev, holder,kobject_get (disk->slave_dir)); +} + +static inline void bd_release_from_disk (struct block_device *bdev, +struct gendisk *disk) +{ +bd_release_from_kobject (bdev, disk->slave_dir); +kobject_put (disk->slave_dir); +} +#endif #endif
fs/block_dev.c +194 -2
230 lines not on this page · raw message
fs/partitions/check.c +55 -0
85 lines not on this page · raw message
Re: [PATCH 1/3] sysfs representation of stacked devices (common) (rev.2)
Quoted 15 lines · Jun'ichi Nomura
Hello Greg,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/partitions/check.c struct kobj_type ktype_part = { · line 309
Quoted 6 lines
fs/partitions/check.c +6
-0+/* This is a mere directory in sysfs. No methods are needed. */ +static struct kobj_type bd_holder_ktype = { + .release =NULL , + .sysfs_ops =NULL , + .default_attrs =NULL , +};That doesn't look right. You always need a release function.
I updated the patch based your comments. Could you take a look at this version whether there's any problematic use of sysfs/kobjects?
Quoted 8 lines
- I removed embedded child-kobjects from struct
block_deviceand struct gendisk which I added in my previous patch. Kobject registration occurs when gendisk orhd_structis registered. Release function of the kobject type is added.
- Reference counting of kobjects is done in much symmetric manner than before.
- Added bd_claim_by_disk/bd_release_from_disk inline functions to help proper reference counting.
Looks great, only one comment:
Quoted 26 lines
fs/partitions/check.c +20
-0@@ -297,6 +297,56 @@ struct kobj_type ktype_part = { .sysfs_ops = &part_sysfs_ops, }; +static void dir_release (struct kobject *kobj) +{ +kfree (kobj); +} + +static struct kobj_type dir_ktype = { + .release = dir_release, + .sysfs_ops =NULL , + .default_attrs =NULL , +}; + +static inline struct kobject *add_dir (struct kobject *parent,const char *name) +{ +struct kobject *k; + +if (!parent) +return NULL ; + + k =kmalloc (sizeof (*k),GFP_KERNEL ); +if (!k)
fs/partitions/check.c struct kobj_type ktype_part = { · line 329
Quoted 10 lines
fs/partitions/check.c +10
-0+return NULL ; + +memset (k,0 ,sizeof (*k)); + k->parent = parent; + k->ktype = &dir_ktype; +kobject_set_name (k, name); +kobject_register (k); + +return k; +}
This code looks good enough that we should add it to the core kobject code, don't you think? Also, you might use kzalloc instead of kmalloc here.
thanks,
greg k-h
Re: [PATCH 1/3] sysfs representation of stacked devices (common) (rev.2)
Hi Greg,
Thank you for the comments.
Quoted 15 lines · Greg KH
Quoted 15 lines
fs/partitions/check.c +15
-0+static inline struct kobject *add_dir (struct kobject *parent,const char *name) +{ +struct kobject *k; + +if (!parent) +return NULL ; + + k =kmalloc (sizeof (*k),GFP_KERNEL ); +if (!k) +return NULL ; + +memset (k,0 ,sizeof (*k)); + k->parent = parent; + k->ktype = &dir_ktype; +kobject_set_name (k, name);
Quoted 8 lines
fs/partitions/check.c struct kobj_type ktype_part = { · line 329
Quoted 4 lines
fs/partitions/check.c +4
-0+kobject_register (k); + +return k; +}This code looks good enough that we should add it to the core kobject code, don't you think? Also, you might use kzalloc instead of kmalloc here.
Yes, it would be nice if kobject core has this function. I'll move them to lib/kobject.c.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
PATCH 2/3 sysfs representation of stacked devices (dm) (rev.2)
In reply to Jun'ichi NomuraThis patch modifies dm driver to call bd_claim_by_kobject and bd_release_from_kobject.
To do that, reference to the mapped_device is added in dm_table.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
Exporting stacked device relationship to sysfs (dm)
drivers/md/dm.h +2 -0
@@ -102,6 +102,8 @@ int dm_table_create(struct dm_table **revoid dm_table_get (struct dm_table *t);void dm_table_put (struct dm_table *t); +void dm_table_set_md (struct dm_table *t,struct mapped_device *md); +struct mapped_device *dm_table_device (struct dm_table *t);int dm_table_add_target (struct dm_table *t,const char *type,sector_t start,sector_t len,char *params);
drivers/md/dm-ioctl.c +25 -4
@@ -228,6 +228,16 @@ static int dm_hash_insert(const char *nareturn -EBUSY ; } +/* called when the populated table is no longer needed */ +static void __release_new_map (struct dm_table *t) +{ +struct mapped_device *md =dm_table_device (t); + +dm_table_put (t); +if (md) +dm_put (md); +} +static void __hash_remove (struct hash_cell *hc) {struct dm_table *table;
@@ -246,7 +256,7 @@ static void __hash_remove(struct hash_cedm_put (hc->md);if (hc->new_map) -dm_table_put (hc->new_map); +__release_new_map (hc->new_map);free_cell (hc); }
@@ -719,6 +729,7 @@ static int do_resume(struct dm_ioctl *padm_get (md); new_map = hc->new_map; +dm_put (md);/* drop reference by dm_table_set_md() */ hc->new_map =NULL ; param->flags &= ~DM_INACTIVE_PRESENT_FLAG ;
@@ -963,9 +974,19 @@ static int table_load(struct dm_ioctl *pif (r)return r; +down_read (&_hash_lock); + hc =__find_device_hash_cell (param); +if (!hc) { +DMWARN ("device doesn't appear to be in the dev hash table." ); +up_read (&_hash_lock); +return -ENXIO ; + } +dm_table_set_md (t, hc->md); +up_read (&_hash_lock); + r =populate_table (t, param, param_size);if (r) { -dm_table_put (t); +__release_new_map (t);return r; }
@@ -979,7 +1000,7 @@ static int table_load(struct dm_ioctl *p }if (hc->new_map) -dm_table_put (hc->new_map); +__release_new_map (hc->new_map); hc->new_map = t; param->flags |=DM_INACTIVE_PRESENT_FLAG ;
@@ -1003,7 +1024,7 @@ static int table_clear(struct dm_ioctl * }if (hc->new_map) { -dm_table_put (hc->new_map); +__release_new_map (hc->new_map); hc->new_map =NULL ; }
drivers/md/dm-table.c +28 -10
@@ -53,6 +53,8 @@ struct dm_table {/* events get handed up using this callback */ void (*event_fn)(void *);void *event_context; + +struct mapped_device *md; };/*
@@ -345,7 +347,7 @@ static struct dm_dev *find_device(struct/* * Open a device so we can use it as a map destination. */ -static int open_dev (struct dm_dev *d,dev_t dev) +static int open_dev (struct dm_dev *d,dev_t dev,struct mapped_device *md) {static char *_claim_ptr ="I belong to device-mapper" ;struct block_device *bdev;
@@ -358,7 +360,7 @@ static int open_dev(struct dm_dev *d, de bdev =open_by_devnum (dev, d->mode);if (IS_ERR (bdev))return PTR_ERR (bdev); - r =bd_claim (bdev, _claim_ptr); + r =bd_claim_by_kobject (bdev, _claim_ptr, &dm_disk (md)->slave_dir);if (r)blkdev_put (bdev);else
@@ -369,12 +371,12 @@ static int open_dev(struct dm_dev *d, de/* * Close a device that we've been using. */ -static void close_dev (struct dm_dev *d) +static void close_dev (struct dm_dev *d,struct mapped_device *md) {if (!d->bdev)return ; -bd_release (d->bdev); +bd_release_from_kobject (d->bdev, &dm_disk (md)->slave_dir);blkdev_put (d->bdev); d->bdev =NULL ; }
@@ -395,7 +397,7 @@ static int check_device_area(struct dm_d* careful to leave things as they were if we fail to reopen the * device. */ -static int upgrade_mode (struct dm_dev *dd,int new_mode) +static int upgrade_mode (struct dm_dev *dd,int new_mode,struct mapped_device *md) {int r;struct dm_dev dd_copy;
@@ -405,9 +407,9 @@ static int upgrade_mode(struct dm_dev *d dd->mode |= new_mode; dd->bdev =NULL ; - r =open_dev (dd, dev); + r =open_dev (dd, dev, md);if (!r) -close_dev (&dd_copy); +close_dev (&dd_copy, md);else *dd = dd_copy;
@@ -450,7 +452,7 @@ static int __table_get_device(struct dm_ dd->mode = mode; dd->bdev =NULL ; -if ((r =open_dev (dd, dev))) { +if ((r =open_dev (dd, dev, t->md))) {kfree (dd);return r; }
@@ -461,7 +463,7 @@ static int __table_get_device(struct dm_list_add (&dd->list, &t->devices); }else if (dd->mode != (mode | dd->mode)) { - r =upgrade_mode (dd, mode); + r =upgrade_mode (dd, mode, t->md);if (r)return r; }
@@ -536,7 +538,7 @@ int dm_get_device(struct dm_target *ti,void dm_put_device (struct dm_target *ti,struct dm_dev *dd) {if (atomic_dec_and_test (&dd->count)) { -close_dev (dd); +close_dev (dd, ti->table->md);list_del (&dd->list);kfree (dd); }
@@ -945,6 +947,22 @@ int dm_table_flush_all(struct dm_table *return ret; } +void dm_table_set_md (struct dm_table *t,struct mapped_device *md) +{ +if (t->md) { +dm_put (t->md); + t->md =NULL ; + } +if (md) { +dm_get (md); + t->md = md; + } +} +struct mapped_device *dm_table_device (struct dm_table *t) +{ +return t->md; +} +EXPORT_SYMBOL (dm_vcalloc);EXPORT_SYMBOL (dm_get_device);EXPORT_SYMBOL (dm_put_device);
Alasdair G Kergon
maintainerRe: [PATCH 2/3] sysfs representation of stacked devices (dm) (rev.2)
Quoted 4 lines · Jun'ichi Nomura
This patch modifies dm driver to call
bd_claim_by_kobjectandbd_release_from_kobject.
To do that, reference to themapped_deviceis added indm_table.
This patch needs splitting up so that independent changes can be considered separately.
c.f. The proposal from Mike Anderson (repeated below) which I prefer because it makes it clear that a table always belongs to exactly one md.
Exposing dm_table_set_md() suggests a table can have its owning md changed - it can't.
Alasdair
Signature · 140 lines
[email protected] This patch adds a mapped_device member to the dm_table struct. Signed-off-by: Mike Anderson <[email protected]> drivers/md/dm-ioctl.c | 32 +++++++++++++++++++------------- drivers/md/dm-table.c | 12 +++++++++++- drivers/md/dm.h | 4 +++- 3 files changed, 33 insertions(+), 15 deletions(-) Index: sas-2.6-patched/drivers/md/dm.h =================================================================== --- sas-2.6-patched.orig/drivers/md/dm.h 2006-02-20 01:05:32.000000000 -0800 +++ sas-2.6-patched/drivers/md/dm.h 2006-02-20 01:42:29.000000000 -0800 @@ -99,7 +99,8 @@ int dm_suspended(struct mapped_device *m * Functions for manipulating a table. Tables are also reference * counted. *---------------------------------------------------------------*/ -int dm_table_create(struct dm_table **result, int mode, unsigned num_targets); +int dm_table_create(struct dm_table **result, int mode, unsigned + num_targets, struct mapped_device *md); void dm_table_get(struct dm_table *t); void dm_table_put(struct dm_table *t); @@ -123,6 +124,7 @@ void dm_table_resume_targets(struct dm_t int dm_table_any_congested(struct dm_table *t, int bdi_bits); void dm_table_unplug_all(struct dm_table *t); int dm_table_flush_all(struct dm_table *t); +struct mapped_device *dm_table_get_md(struct dm_table *t); /*----------------------------------------------------------------- * A registry of target types. Index: sas-2.6-patched/drivers/md/dm-table.c =================================================================== --- sas-2.6-patched.orig/drivers/md/dm-table.c 2006-02-20 01:05:32.000000000 -0800 +++ sas-2.6-patched/drivers/md/dm-table.c 2006-02-20 01:42:29.000000000 -0800 @@ -33,6 +33,7 @@ struct dm_table { unsigned int num_allocated; sector_t *highs; struct dm_target *targets; + struct mapped_device *md; /* * Indicates the rw permissions for the new logical @@ -204,7 +205,8 @@ static int alloc_targets(struct dm_table return 0; } -int dm_table_create(struct dm_table **result, int mode, unsigned num_targets) +int dm_table_create(struct dm_table **result, int mode, + unsigned num_targets, struct mapped_device *md) { struct dm_table *t = kmalloc(sizeof(*t), GFP_KERNEL); @@ -227,6 +229,7 @@ int dm_table_create(struct dm_table **re } t->mode = mode; + t->md = md; *result = t; return 0; } @@ -945,6 +948,12 @@ int dm_table_flush_all(struct dm_table * return ret; } +struct mapped_device *dm_table_get_md(struct dm_table *t) +{ + dm_get(t->md); + return t->md; +} + EXPORT_SYMBOL(dm_vcalloc); EXPORT_SYMBOL(dm_get_device); EXPORT_SYMBOL(dm_put_device); @@ -955,3 +964,4 @@ EXPORT_SYMBOL(dm_table_put); EXPORT_SYMBOL(dm_table_get); EXPORT_SYMBOL(dm_table_unplug_all); EXPORT_SYMBOL(dm_table_flush_all); +EXPORT_SYMBOL(dm_table_get_md); Index: sas-2.6-patched/drivers/md/dm-ioctl.c =================================================================== --- sas-2.6-patched.orig/drivers/md/dm-ioctl.c 2006-02-20 01:05:32.000000000 -0800 +++ sas-2.6-patched/drivers/md/dm-ioctl.c 2006-02-20 01:42:29.000000000 -0800 @@ -972,27 +972,26 @@ static int populate_table(struct dm_tabl static int table_load(struct dm_ioctl *param, size_t param_size) { - int r; + int r = -ENXIO; struct hash_cell *hc; struct dm_table *t; - r = dm_table_create(&t, get_mode(param), param->target_count); - if (r) - return r; - - r = populate_table(t, param, param_size); - if (r) { - dm_table_put(t); - return r; - } down_write(&_hash_lock); hc = __find_device_hash_cell(param); if (!hc) { DMWARN("device doesn't appear to be in the dev hash table."); - up_write(&_hash_lock); - dm_table_put(t); - return -ENXIO; + goto out; + } + + r = dm_table_create(&t, get_mode(param), param->target_count, + hc->md); + if (r) + goto out; + + r = populate_table(t, param, param_size); + if (r) { + goto table_out; } if (hc->new_map) @@ -1001,6 +1000,13 @@ static int table_load(struct dm_ioctl *p param->flags |= DM_INACTIVE_PRESENT_FLAG; r = __dev_status(hc->md, param); + + up_write(&_hash_lock); + return r; + +table_out: + dm_table_put(t); +out: up_write(&_hash_lock); return r; }
Re: [PATCH 2/3] sysfs representation of stacked devices (dm) (rev.2)
Hi,
Quoted 5 lines · Alasdair G Kergon
This patch needs splitting up so that independent changes can be considered separately.
c.f. The proposal from Mike Anderson (repeated below) which I prefer because it makes it clear that a table always belongs to exactly one md.
I like his proposed patch.
The interface is useful for my purpose too and moving table creation inside _hash_lock means I don't need dm_get() neither.
Is it going to be pushed to upstream?
I'll remake my patch based on it.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
Alasdair G Kergon
maintainerRe: [PATCH 2/3] sysfs representation of stacked devices (dm) (rev.2)
Quoted 9 lines · Jun'ichi Nomura
Quoted 4 lines · Alasdair G Kergon
This patch needs splitting up so that independent changes can be considered separately.
c.f. The proposal from Mike Anderson (repeated below) which I prefer because it makes it clear that a table always belongs to exactly one md.I like his proposed patch.
The interface is useful for my purpose too and moving table creation inside_hash_lockmeans I don't needdm_get()neither.
The global _hash_lock should not be held (thereby locking out most dm ioctl operations on any device) while the slow populate_table() runs.
I'm trying out a variant of the patch that drops and reacquires that lock.
Alasdair
Signature · 1 line
[email protected]
Re: [PATCH 2/3] sysfs representation of stacked devices (dm) (rev.2)
Hi,
Quoted 3 lines · Alasdair G Kergon
The global
_hash_lockshould not be held (thereby locking out most dm ioctl operations on any device) while the slowpopulate_table()runs.I'm trying out a variant of the patch that drops and reacquires that lock.
OK, thanks for the confirmation.
I guess the variant itself will need dm_get() to avoid md being stolen. Depending on that, I might change my patch.
Attached is a revised patch based on Mike Anderson's patch.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
drivers/md/dm-table.c +10 -10
@@ -348,7 +348,7 @@ static struct dm_dev *find_device(struct/* * Open a device so we can use it as a map destination. */ -static int open_dev (struct dm_dev *d,dev_t dev) +static int open_dev (struct dm_dev *d,dev_t dev,struct mapped_device *md) {static char *_claim_ptr ="I belong to device-mapper" ;struct block_device *bdev;
@@ -361,7 +361,7 @@ static int open_dev(struct dm_dev *d, de bdev =open_by_devnum (dev, d->mode);if (IS_ERR (bdev))return PTR_ERR (bdev); - r =bd_claim (bdev, _claim_ptr); + r =bd_claim_by_kobject (bdev, _claim_ptr, &dm_disk (md)->slave_dir);if (r)blkdev_put (bdev);else
@@ -372,12 +372,12 @@ static int open_dev(struct dm_dev *d, de/* * Close a device that we've been using. */ -static void close_dev (struct dm_dev *d) +static void close_dev (struct dm_dev *d,struct mapped_device *md) {if (!d->bdev)return ; -bd_release (d->bdev); +bd_release_from_kobject (d->bdev, &dm_disk (md)->slave_dir);blkdev_put (d->bdev); d->bdev =NULL ; }
@@ -398,7 +398,7 @@ static int check_device_area(struct dm_d* careful to leave things as they were if we fail to reopen the * device. */ -static int upgrade_mode (struct dm_dev *dd,int new_mode) +static int upgrade_mode (struct dm_dev *dd,int new_mode,struct mapped_device *md) {int r;struct dm_dev dd_copy;
@@ -408,9 +408,9 @@ static int upgrade_mode(struct dm_dev *d dd->mode |= new_mode; dd->bdev =NULL ; - r =open_dev (dd, dev); + r =open_dev (dd, dev, md);if (!r) -close_dev (&dd_copy); +close_dev (&dd_copy, md);else *dd = dd_copy;
@@ -453,7 +453,7 @@ static int __table_get_device(struct dm_ dd->mode = mode; dd->bdev =NULL ; -if ((r =open_dev (dd, dev))) { +if ((r =open_dev (dd, dev, t->md))) {kfree (dd);return r; }
@@ -464,7 +464,7 @@ static int __table_get_device(struct dm_list_add (&dd->list, &t->devices); }else if (dd->mode != (mode | dd->mode)) { - r =upgrade_mode (dd, mode); + r =upgrade_mode (dd, mode, t->md);if (r)return r; }
@@ -539,7 +539,7 @@ int dm_get_device(struct dm_target *ti,void dm_put_device (struct dm_target *ti,struct dm_dev *dd) {if (atomic_dec_and_test (&dd->count)) { -close_dev (dd); +close_dev (dd, ti->table->md);list_del (&dd->list);kfree (dd); }
PATCH 3/3 sysfs representation of stacked devices (md) (rev.2)
In reply to Jun'ichi NomuraThis patch modifies md driver to call bd_claim_by_kobject and bd_release_from_kobject.
Signature · 1 line
Jun'ichi Nomura, NEC Solutions (America), Inc.
Exporting stacked device relationship to sysfs (md)
drivers/md/md.c +2 -0
@@ -1226,6 +1226,7 @@ static int bind_rdev_to_array(mdk_rdev_telse ko = &rdev->bdev->bd_disk->kobj;sysfs_create_link (&rdev->kobj, ko,"block" ); +bd_claim_by_kobject (rdev->bdev, rdev, &mddev->gendisk->slave_dir);return 0 ; }
@@ -1236,6 +1237,7 @@ static void unbind_rdev_from_array(mdk_rMD_BUG ();return ; } +bd_release_from_kobject (rdev->bdev, &rdev->mddev->gendisk->slave_dir);list_del_init (&rdev->same_set);printk (KERN_INFO "md: unbind<%s>\n" ,bdevname (rdev->bdev,b)); rdev->mddev =NULL ;
Quoted 16 lines · Jun'ichi Nomura
Hello,
This is a revised set of pathces which provides common representation of dependencies between stacked devices (dm and md) in sysfs.
Variants of bd_claim/bd_release are added to accept a kobject and create symlinks between the claimed bdev and the holder.
dm/md will give a child of its gendisk kobject to
bd_claim. For example, if dm-0 maps to sda, we have the following symlinks;
/sys/block/dm-0/slaves/sda-->/sys/block/sda/sys/block/sda/holders/dm-0-->/sys/block/dm-0Comments are welcome.
Quoted 8 lines
A few points I would appreciate comments/reviews from maintainers:
About sysfs
- I confirmedsysfs_remove_symlink()andkobject_del()don't allocate memory in 2.6.15 and it seems true on the git head. I would like to make sure it's true in future versions of kernel because they are called during device-mapper's table swapping where I/O to free memory could deadlock on the dm device. What is the recommended way to do that?
But it can possibly sleep.
Hm, wait, the put_device stuff can possibly sleep, the "raw" kobject_del() stuff looks safe. Either way, they don't create new memory, unless you do something really wierd in your release callback.
So you should be safe here.
But if you want to be absolutly safe, look at the thread on lkml about changing the scsi code to do the final release in a non-interrupt context. That looks like it might be the same thing you want to do here to guarantee that nothing bad happens.
thanks,
greg k-h