md: bitmap grow fixes
v2 cover md: bitmap grow fixes
Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.
The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.
v2:
Add a dummy bitmap operations per Kuai's suggestion.
To Yu Kuai:
A NULL group can't used for internal_bitmap_group since
the entries from internal_bitmap_attrs should be under bitmap.
So instead of sysfs_create_groups(), sysfs_create_group() and sysfs_merge_group() are still needed /sigh.
Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy in location_store md/md-bitmap: add an extra sysfs argument to md_bitmap_create and
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skip ID_BITMAP_NONE when show available bitmap types md/md-bitmap: remove member group from bitmap_operations
drivers/md/md-bitmap.c | 121 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 +- drivers/md/md-llbitmap.c | 13 ++++- drivers/md/md.c | 55 +++++++++--------- drivers/md/md.h | 2 + 5 files changed, 155 insertions(+), 39 deletions(-)
Signature · 1 line
2.53.0
PATCH v2 1/5 md/md-bitmap: call md_bitmap_create,destroy in location_store
If bitmap/location is present, mdadm will call update_array_info() while growing bitmap from none to internal via location_store(). md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger kernel NULL pointer dereference.
--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +8 -3
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set bitmap_id +* using bitmap_type, so always ID_BITMAP. +*/ + mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); goto out; } }
drivers/md/md.c +2 -2
@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -static int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL ;
@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev *mddev)return mddev->bitmap_ops->create (mddev); } -static void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev) {if (!md_bitmap_registered (mddev))return ;
drivers/md/md.h +2 -0
@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); +int md_bitmap_create (struct mddev *mddev); +void md_bitmap_destroy (struct mddev *mddev); extern struct md_thread *md_register_thread( void (*run)(struct md_thread *thread),
Signature · 1 line
2.53.0
Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
Quoted 33 lines · Su Yue
If bitmap/location is present, mdadm will call
update_array_info()while growing bitmap from none to internal vialocation_store().md_bitmap_create()is needed to setmddev->bitmap_opsotherwisemddev->bitmap_ops->get_stats()inupdate_array_info()will trigger kernel NULL pointer dereference.Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)drivers/md/md-bitmap.c +3
-1@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set bitmap_id
drivers/md/md-bitmap.c location_store(struct mddev *mddev, const char *buf, size_t len) · line 2666
Quoted 10 lines
drivers/md/md-bitmap.c +4
-1+* using bitmap_type, so always ID_BITMAP. +*/ + mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);
mddev->bitmap_ops->load() should also be used here.
Signature · 2 lines
Thanks, Nan
Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
Quoted 36 lines · Li Nan
Quoted 35 lines · Su Yue
If bitmap/location is present, mdadm will call
update_array_info()while growing bitmap from none to internal vialocation_store().md_bitmap_create()is needed to setmddev->bitmap_opsotherwisemddev->bitmap_ops->get_stats()inupdate_array_info()will trigger kernel NULL pointer dereference.Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops untilcreating bitmap")
Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)drivers/md/md-bitmap.c +5
-0@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set bitmap_id +* using bitmap_type, so always ID_BITMAP. +*/ + mddev->bitmap_id =ID_BITMAP ;
Quoted 8 lines
drivers/md/md-bitmap.c location_store(struct mddev *mddev, const char *buf, size_t len) · line 2666
Quoted 6 lines
drivers/md/md-bitmap.c +1
-1mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);
mddev->bitmap_ops->load()should also be used here.
/NOD.location_store() is only used for ID_BITMAP_NONE and ID_BITMAP, so mddev->bitmap_ops->load() is always bitmap_load().
But for code consistency, mddev->bitmap_ops->load() is better, will fix it.
--
Su
Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
In reply to Su YueQuoted 6 lines · Su Yue
If bitmap/location is present, mdadm will call
update_array_info()while growing bitmap from none to internal vialocation_store().md_bitmap_create()is needed to setmddev->bitmap_opsotherwisemddev->bitmap_ops->get_stats()inupdate_array_info()will trigger kernel NULL pointer dereference.
Hi Su Yue
How can bitmap/location be present when bitmap is none? Could you provide the test commands that reproduce this problem?
mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --bitmap=none (There is not bitmap/location, because bitmap directory is not created) mdadm /dev/md0 --grow --bitmap=internalGrow.c md_set_array_info runs
451 array.state |= (1 << MD_SB_BITMAP_PRESENT);
452 rv = md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv = md_bitmap_create(mddev);
8126 if (!rv)
8127 rv = mddev->bitmap_ops->load(mddev);
Best Regards
Xiao
Quoted 83 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)drivers/md/md-bitmap.c +8
-3@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set bitmap_id +* using bitmap_type, so always ID_BITMAP. +*/ + mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); goto out; } }drivers/md/md.c +2
-2@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -static int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL ;@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev *mddev)return mddev->bitmap_ops->create (mddev); } -static void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev) {if (!md_bitmap_registered (mddev))return ;drivers/md/md.h +2
-1@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); +int md_bitmap_create (struct mddev *mddev); +void md_bitmap_destroy (struct mddev *mddev); extern struct md_thread *md_register_thread( void (*run)(struct md_thread *thread), --2.53.0
Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
Quoted 9 lines · Xiao Ni
Quoted 6 lines · Su Yue
If bitmap/location is present, mdadm will call
update_array_info()while growing bitmap from none to internal vialocation_store().md_bitmap_create()is needed to setmddev->bitmap_opsotherwisemddev->bitmap_ops->get_stats()inupdate_array_info()will trigger kernel NULL pointer dereference.
Quoted 5 lines
Hi Su Yue
How can bitmap/location be present when bitmap is none? Could you provide the test commands that reproduce this problem?
Sorry for the misleading commit message. It can only be reproduced patch 3 is appiled. I adjusted the sequence of this patch for easy review because md_bitmap_create,destroy are touched in patch1,2 and 3. Also if put the patch after 3rd patch,
it will break ability to bisect.
# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal --raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc /dev/vdd # mdadm --grow /dev/md0 --bitmap=none # mdadm --grow /dev/md0 --bitmap=internal # step 3 # mdadm --grow /dev/md0 --bitmap=none # step 4 [1] 2325 killed mdadm --grow /dev/md0 --bitmap=none
When step 3 is called,md_bitmap_destroy() is called in update_array_info() to set NULL mddev->bitmap_ops then in step 4 kernel Oops is triggered.
I am willing to amend commit message or move it after patch 3 if you would like.
--
Su
Quoted 106 lines
mdadm -CR
/dev/md0-l1 -n2/dev/loop0/dev/loop1--bitmap=none (There is not bitmap/location, because bitmap directory is not created) mdadm/dev/md0--grow --bitmap=internalGrow.cmd_set_array_inforuns
451 array.state |= (1 <<MD_SB_BITMAP_PRESENT);
452 rv =md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv =md_bitmap_create(mddev);
8126 if (!rv)
8127 rv =mddev->bitmap_ops->load(mddev);Best Regards
XiaoQuoted 91 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)drivers/md/md-bitmap.c +1
-1@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f =
mddev->bitmap_info.file;drivers/md/md-bitmap.c +2
-0@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set
bitmap_id
+ * usingbitmap_type, so alwaysID_BITMAP.drivers/md/md-bitmap.c +4
-2+*/ + mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); goto out; } }drivers/md/md.c +1
-1@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -static int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL;drivers/md/md.c +1
-1@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev *mddev)return mddev->bitmap_ops->create (mddev); } -static void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev) {if (!md_bitmap_registered (mddev))return;drivers/md/md.h +2
-0@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); +int md_bitmap_create (struct mddev *mddev); +void md_bitmap_destroy (struct mddev *mddev); extern struct md_thread *md_register_thread(void (*run)(struct
md_thread*thread),
--
2.53.0
Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
Quoted 36 lines · Su Yue
Quoted 12 lines · Xiao Ni
Quoted 9 lines · Su Yue
If bitmap/location is present, mdadm will call
update_array_info()
while growing bitmap from none to internal vialocation_store().md_bitmap_create()is needed to setmddev->bitmap_opsotherwisemddev->bitmap_ops->get_stats()inupdate_array_info()will trigger
kernel NULL pointer dereference.Quoted 6 lines
Hi Su Yue
How can bitmap/location be present when bitmap is none? Could you
provide the test commands that reproduce this problem?Sorry for the misleading commit message. It can only be reproduced patch 3 is appiled.
I adjusted the sequence of this patch for easy review because md_bitmap_create,destroy
are touched in patch1,2 and 3. Also if put the patch after 3rd patch,
it will break ability to bisect.# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal --raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc /dev/vdd # mdadm --grow /dev/md0 --bitmap=none # mdadm --grow /dev/md0 --bitmap=internal # step 3 # mdadm --grow /dev/md0 --bitmap=none # step 4 [1] 2325 killed mdadm --grow /dev/md0 --bitmap=none
Quoted 8 lines
When step 3 is called,
md_bitmap_destroy()is called inupdate_array_info()to set NULLmddev->bitmap_ops
then in step 4 kernel Oops is triggered.I am willing to amend commit message or move it after patch 3 if you would like.
Hi Su
Thanks for the detail explanation. After reading patch3, I totoally understand. The sequence is good to me. And yes, it's better to explain that this is needed after patch3.
Best Regards
Xiao
Quoted 113 lines
--
SuQuoted 108 lines
mdadm -CR
/dev/md0-l1 -n2/dev/loop0/dev/loop1--bitmap=none (There
is not bitmap/location, because bitmap directory is not created) mdadm/dev/md0--grow --bitmap=internalGrow.cmd_set_array_inforuns
451 array.state |= (1 <<MD_SB_BITMAP_PRESENT);
452 rv =md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv =md_bitmap_create(mddev);
8126 if (!rv)
8127 rv =mddev->bitmap_ops->load(mddev);Best Regards
XiaoQuoted 92 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of
bitmap_opsuntil creating bitmap")Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)drivers/md/md-bitmap.c +1
-1@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f =
mddev->bitmap_info.file;drivers/md/md-bitmap.c +2
-0@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set
bitmap_id
+ * usingbitmap_type, so alwaysID_BITMAP.drivers/md/md-bitmap.c +4
-2+*/ + mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); goto out; } }drivers/md/md.c +1
-1@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -static int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL;drivers/md/md.c +1
-1@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev *mddev)return mddev->bitmap_ops->create (mddev); } -static void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev) {if (!md_bitmap_registered (mddev))return;drivers/md/md.h +2
-0@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); +int md_bitmap_create (struct mddev *mddev); +void md_bitmap_destroy (struct mddev *mddev); extern struct md_thread *md_register_thread(void (*run)(struct
md_thread*thread),
--
2.53.0
Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
Quoted 44 lines · Xiao Ni
Quoted 43 lines · Su Yue
Quoted 12 lines · Xiao Ni
Quoted 9 lines · Su Yue
If bitmap/location is present, mdadm will call
update_array_info()
while growing bitmap from none to internal vialocation_store().md_bitmap_create()is needed to setmddev->bitmap_opsotherwisemddev->bitmap_ops->get_stats()inupdate_array_info()will trigger
kernel NULL pointer dereference.Quoted 6 lines
Hi Su Yue
How can bitmap/location be present when bitmap is none? Could you
provide the test commands that reproduce this problem?Sorry for the misleading commit message. It can only be reproduced patch 3 is appiled.
I adjusted the sequence of this patch for easy review because md_bitmap_create,destroy
are touched in patch1,2 and 3. Also if put the patch after 3rd patch,
it will break ability to bisect.# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal --raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc /dev/vdd # mdadm --grow /dev/md0 --bitmap=none # mdadm --grow /dev/md0 --bitmap=internal # step 3 # mdadm --grow /dev/md0 --bitmap=none # step 4 [1] 2325 killed mdadm --grow /dev/md0 --bitmap=noneWhen step 3 is called,
md_bitmap_destroy()is called inupdate_array_info()to set NULLmddev->bitmap_ops
then in step 4 kernel Oops is triggered.
Quoted 9 lines
Quoted 2 lines
I am willing to amend commit message or move it after patch 3 if you would like.
Hi Su
Thanks for the detail explanation. After reading patch3, I totoally understand. The sequence is good to me. And yes, it's better to explain that this is needed after patch3.
Sure. I will do it in next version.
--
Su
Quoted 121 lines
Best Regards
XiaoQuoted 118 lines
--
SuQuoted 113 lines
mdadm -CR
/dev/md0-l1 -n2/dev/loop0/dev/loop1--bitmap=none
(There
is not bitmap/location, because bitmap directory is not created) mdadm/dev/md0--grow --bitmap=internalGrow.cmd_set_array_inforuns
451 array.state |= (1 <<MD_SB_BITMAP_PRESENT);
452 rv =md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv =md_bitmap_create(mddev);
8126 if (!rv)
8127 rv =mddev->bitmap_ops->load(mddev);Best Regards
XiaoQuoted 96 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of
bitmap_opsuntil creating bitmap")Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 11 ++++++++--- drivers/md/md.c | 4 ++-- drivers/md/md.h | 2 ++ 3 files changed, 12 insertions(+), 5 deletions(-)drivers/md/md-bitmap.c +0
-0@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev,const
char *buf,size_tlen)goto out; }drivers/md/md-bitmap.c +3
-1-bitmap_destroy (mddev); +md_bitmap_destroy (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } +/* +* lockless bitmap shoudle have set
bitmap_id
+ * usingbitmap_type, so alwaysID_BITMAP.drivers/md/md-bitmap.c +4
-2+*/ + mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =bitmap_create (mddev); + rv =md_bitmap_create (mddev);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -bitmap_destroy (mddev); +md_bitmap_destroy (mddev); goto out; } }drivers/md/md.c +1
-1@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -static int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL;drivers/md/md.c +0
-0@@ -6458,7 +6458,7 @@ static int md_bitmap_create(structmddev
*mddev)return mddev->bitmap_ops->create(mddev);}
drivers/md/md.c +1
-1-static void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev) {if (!md_bitmap_registered (mddev))return ;drivers/md/md.h +2
-0@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); +int md_bitmap_create (struct mddev *mddev); +void md_bitmap_destroy (struct mddev *mddev);extern struct
md_thread*md_register_thread(
void (*run)(structmd_thread*thread),
--
2.53.0
PATCH v2 2/5 md/md-bitmap: add an extra sysfs argument to md_bitmap_create and destroy
In reply to Su YueFor further use, no functional change.
--- drivers/md/md-bitmap.c | 6 +++--- drivers/md/md.c | 36 ++++++++++++++++++------------------ drivers/md/md.h | 4 ++-- 3 files changed, 23 insertions(+), 23 deletions(-)
drivers/md/md-bitmap.c +3 -3
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;
@@ -2659,14 +2659,14 @@ location_store(struct mddev *mddev, const char *buf, size_t len)*/ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev); + rv =md_bitmap_create (mddev, true);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); goto out; } }
drivers/md/md.c +18 -18
@@ -678,7 +678,7 @@ static void active_io_release(struct percpu_ref *ref)static void no_op (struct percpu_ref *r) {} -static bool mddev_set_bitmap_ops (struct mddev *mddev) +static bool mddev_set_bitmap_ops (struct mddev *mddev,bool create_sysfs) {struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head;
@@ -703,7 +703,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (!mddev_is_dm (mddev) && mddev->bitmap_ops->group) { +if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) {if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group))pr_warn ("md: cannot register extra bitmap attributes for %s\n" ,mdname (mddev));
@@ -721,9 +721,9 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev)return false; } -static void mddev_clear_bitmap_ops (struct mddev *mddev) +static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) { -if (!mddev_is_dm (mddev) && mddev->bitmap_ops && +if (remove_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops && mddev->bitmap_ops->group)sysfs_remove_group (&mddev->kobj, mddev->bitmap_ops->group);
@@ -6447,24 +6447,24 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev,bool create_sysfs) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL ; -if (!mddev_set_bitmap_ops (mddev)) +if (!mddev_set_bitmap_ops (mddev, create_sysfs))return -ENOENT ;return mddev->bitmap_ops->create (mddev); } -void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev,bool remove_sysfs) {if (!md_bitmap_registered (mddev))return ; mddev->bitmap_ops->destroy (mddev); -mddev_clear_bitmap_ops (mddev); +mddev_clear_bitmap_ops (mddev, remove_sysfs); }int md_run (struct mddev *mddev)
@@ -6612,7 +6612,7 @@ int md_run(struct mddev *mddev) }if (err ==0 && pers->sync_request && (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { - err =md_bitmap_create (mddev); + err =md_bitmap_create (mddev, true);if (err)pr_warn ("%s: failed to create bitmap (%d)\n" ,mdname (mddev), err);
@@ -6685,7 +6685,7 @@ int md_run(struct mddev *mddev) pers->free (mddev, mddev->private); mddev->private =NULL ;put_pers (pers); -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true);return err; }EXPORT_SYMBOL_GPL (md_run);
@@ -6702,7 +6702,7 @@ int do_md_run(struct mddev *mddev)if (md_bitmap_registered (mddev)) { err = mddev->bitmap_ops->load (mddev);if (err) { -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); goto out; } }
@@ -6903,7 +6903,7 @@ static void __md_stop(struct mddev *mddev) {struct md_personality *pers = mddev->pers; -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true);mddev_detach (mddev);spin_lock (&mddev->lock); mddev->pers =NULL ;
@@ -7680,16 +7680,16 @@ static int set_bitmap_file(struct mddev *mddev, int fd) err =0 ;if (mddev->pers) {if (fd >=0 ) { - err =md_bitmap_create (mddev); + err =md_bitmap_create (mddev, true);if (!err) err = mddev->bitmap_ops->load (mddev);if (err) { -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); fd =-1 ; } }else if (fd <0 ) { -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); } }
@@ -7996,12 +7996,12 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info) mddev->bitmap_info.default_offset; mddev->bitmap_info.space = mddev->bitmap_info.default_space; - rv =md_bitmap_create (mddev); + rv =md_bitmap_create (mddev, true);if (!rv) rv = mddev->bitmap_ops->load (mddev);if (rv) -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); }else {struct md_bitmap_stats stats;
@@ -8027,7 +8027,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)put_cluster_ops (mddev); mddev->safemode_delay =DEFAULT_SAFEMODE_DELAY ; } -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); mddev->bitmap_info.offset =0 ; } }
drivers/md/md.h +2 -2
@@ -895,8 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); -int md_bitmap_create (struct mddev *mddev); -void md_bitmap_destroy (struct mddev *mddev); +int md_bitmap_create (struct mddev *mddev,bool create_sysfs); +void md_bitmap_destroy (struct mddev *mddev,bool remove_sysfs); extern struct md_thread *md_register_thread( void (*run)(struct md_thread *thread),
Signature · 1 line
2.53.0
Re: [PATCH v2 2/5] md/md-bitmap: add an extra sysfs argument to md_bitmap_create and destroy
Quoted 199 lines · Su Yue
For further use, no functional change.
Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 6 +++--- drivers/md/md.c | 36 ++++++++++++++++++------------------ drivers/md/md.h | 4 ++-- 3 files changed, 23 insertions(+), 23 deletions(-)drivers/md/md-bitmap.c +3
-3@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2659,14 +2659,14 @@ location_store(struct mddev *mddev, const char *buf, size_t len)*/ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev); + rv =md_bitmap_create (mddev, true);if (rv) goto out; rv =bitmap_load (mddev);if (rv) { mddev->bitmap_info.offset =0 ; -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); goto out; } }drivers/md/md.c +18
-18@@ -678,7 +678,7 @@ static void active_io_release(struct percpu_ref *ref)static void no_op (struct percpu_ref *r) {} -static bool mddev_set_bitmap_ops (struct mddev *mddev) +static bool mddev_set_bitmap_ops (struct mddev *mddev,bool create_sysfs) {struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head;@@ -703,7 +703,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (!mddev_is_dm (mddev) && mddev->bitmap_ops->group) { +if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) {if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group))pr_warn ("md: cannot register extra bitmap attributes for %s\n" ,mdname (mddev));@@ -721,9 +721,9 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev)return false; } -static void mddev_clear_bitmap_ops (struct mddev *mddev) +static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) { -if (!mddev_is_dm (mddev) && mddev->bitmap_ops && +if (remove_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops && mddev->bitmap_ops->group)sysfs_remove_group (&mddev->kobj, mddev->bitmap_ops->group);@@ -6447,24 +6447,24 @@ static void md_safemode_timeout(struct timer_list *t)static int start_dirty_degraded; -int md_bitmap_create (struct mddev *mddev) +int md_bitmap_create (struct mddev *mddev,bool create_sysfs) {if (mddev->bitmap_id ==ID_BITMAP_NONE )return -EINVAL ; -if (!mddev_set_bitmap_ops (mddev)) +if (!mddev_set_bitmap_ops (mddev, create_sysfs))return -ENOENT ;return mddev->bitmap_ops->create (mddev); } -void md_bitmap_destroy (struct mddev *mddev) +void md_bitmap_destroy (struct mddev *mddev,bool remove_sysfs) {if (!md_bitmap_registered (mddev))return ; mddev->bitmap_ops->destroy (mddev); -mddev_clear_bitmap_ops (mddev); +mddev_clear_bitmap_ops (mddev, remove_sysfs); }int md_run (struct mddev *mddev)@@ -6612,7 +6612,7 @@ int md_run(struct mddev *mddev) }if (err ==0 && pers->sync_request && (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { - err =md_bitmap_create (mddev); + err =md_bitmap_create (mddev, true);if (err)pr_warn ("%s: failed to create bitmap (%d)\n" ,mdname (mddev), err);@@ -6685,7 +6685,7 @@ int md_run(struct mddev *mddev) pers->free (mddev, mddev->private); mddev->private =NULL ;put_pers (pers); -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true);return err; }EXPORT_SYMBOL_GPL (md_run);@@ -6702,7 +6702,7 @@ int do_md_run(struct mddev *mddev)if (md_bitmap_registered (mddev)) { err = mddev->bitmap_ops->load (mddev);if (err) { -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); goto out; } }@@ -6903,7 +6903,7 @@ static void __md_stop(struct mddev *mddev) {struct md_personality *pers = mddev->pers; -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true);mddev_detach (mddev);spin_lock (&mddev->lock); mddev->pers =NULL ;@@ -7680,16 +7680,16 @@ static int set_bitmap_file(struct mddev *mddev, int fd) err =0 ;if (mddev->pers) {if (fd >=0 ) { - err =md_bitmap_create (mddev); + err =md_bitmap_create (mddev, true);if (!err) err = mddev->bitmap_ops->load (mddev);if (err) { -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); fd =-1 ; } }else if (fd <0 ) { -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); } }@@ -7996,12 +7996,12 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info) mddev->bitmap_info.default_offset; mddev->bitmap_info.space = mddev->bitmap_info.default_space; - rv =md_bitmap_create (mddev); + rv =md_bitmap_create (mddev, true);if (!rv) rv = mddev->bitmap_ops->load (mddev);if (rv) -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); }else {struct md_bitmap_stats stats;@@ -8027,7 +8027,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)put_cluster_ops (mddev); mddev->safemode_delay =DEFAULT_SAFEMODE_DELAY ; } -md_bitmap_destroy (mddev); +md_bitmap_destroy (mddev, true); mddev->bitmap_info.offset =0 ; } }drivers/md/md.h +2
-3@@ -895,8 +895,8 @@ static inline void safe_put_page(struct page *p)int register_md_submodule (struct md_submodule_head *msh);void unregister_md_submodule (struct md_submodule_head *msh); -int md_bitmap_create (struct mddev *mddev); -void md_bitmap_destroy (struct mddev *mddev); +int md_bitmap_create (struct mddev *mddev,bool create_sysfs); +void md_bitmap_destroy (struct mddev *mddev,bool remove_sysfs); extern struct md_thread *md_register_thread( void (*run)(struct md_thread *thread), --
Quoted 2 lines
2.53.0
Looks good to me.
PATCH v2 3/5 md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
In reply to Su YueBefore commit fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap") if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.
After the commit, if bitmap is none, bitmap/location doesn't exist anymore. It breaks 'grow' behavior of a md array of madam with MD_FEATURE_BITMAP_OFFSET. Take level=mirror and metadata=1.2 as an example:
$ mdadm --create /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \ --metadata=1.2 /dev/vdd /dev/vde $ mdadm --grow /dev/md0 --bitmap=internal $ cat /sys/block/md0/md/bitmap/location Before:+8 After: +2
While growing bitmap from none to internal, clustered and llbitmap, mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first. 1)If bitmap/location exists, it sets bitmap/location after getinfo_super(). 2)If bitmap/location doesn't exist, mdadm just calls md_set_array_info() then mddev->bitmap_info.default_offset will be used.
Situation can be worse if growing none to clustered, bitmap offset of the node calling madm --grow will be changed but the other node are reading bitmap sb from the old location.
Here introducing a dummy bitmap_operations for ID_BITMAP_NONE to restore sysfs file bitmap/location for ID_BITMAP_NONE and ID_BITMAP.location_store() now calls md_bitmap_(create|destroy) with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.
--- drivers/md/md-bitmap.c | 116 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 + drivers/md/md-llbitmap.c | 12 ++++ drivers/md/md.c | 16 +++--- 4 files changed, 130 insertions(+), 17 deletions(-)
drivers/md/md-bitmap.c +107 -9
@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush) bitmap->storage.filemap !=NULL ; } +static bool dummy_bitmap_enabled (void *data,bool flush) +{ +return false; +} +/* * check a page and, if necessary, allocate it (or hijack it if the alloc fails) *
@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)return 0 ; } +static int dummy_bitmap_create (struct mddev *mddev) +{ +return 0 ; +} +static int bitmap_load (struct mddev *mddev) {int err =0 ;
@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char *page)return len; } +static int create_internal_group (struct mddev *mddev); +static void remove_internal_group (struct mddev *mddev); +static ssize_t location_store (struct mddev *mddev,const char *buf,size_t len) {
@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;
@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev, const char *buf, size_t len)*/ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev, true); + rv =md_bitmap_create (mddev, false);if (rv) goto out; + rv =create_internal_group (mddev); +if (rv) { + mddev->bitmap_info.offset =0 ; +bitmap_destroy (mddev); + goto out; + } + rv =bitmap_load (mddev);if (rv) { +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ; -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); goto out; } }
@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry max_backlog_used =__ATTR (max_backlog_used,S_IRUGO |S_IWUSR , behind_writes_used_show, behind_writes_used_reset); -static struct attribute *md_bitmap_attrs[] = { - &bitmap_location.attr, +static struct attribute *internal_bitmap_attrs[] = { &bitmap_space.attr, &bitmap_timeout.attr, &bitmap_backlog.attr,
@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = {NULL }; -static struct attribute_group md_bitmap_group = { +static struct attribute_group internal_bitmap_group = { .name ="bitmap" , - .attrs = md_bitmap_attrs, + .attrs = internal_bitmap_attrs, }; +/* Only necessary attrs for compatibility */ +static struct attribute *common_bitmap_attrs[] = { + &bitmap_location.attr, +NULL +}; + +static const struct attribute_group common_bitmap_group = { + .name ="bitmap" , + .attrs = common_bitmap_attrs, +}; + +static int create_internal_group (struct mddev *mddev) +{ +/* +* md_bitmap_group and md_bitmap_common_group are using same name +* 'bitmap'. +*/ +return sysfs_merge_group (&mddev->kobj, &internal_bitmap_group); +} + +static void remove_internal_group (struct mddev *mddev) +{ +sysfs_unmerge_group (&mddev->kobj, &internal_bitmap_group); +} + +static int bitmap_register_groups (struct mddev *mddev) +{ +int ret; + + ret =sysfs_create_group (&mddev->kobj, &common_bitmap_group); + +if (ret) +return ret; + + ret =sysfs_merge_group (&mddev->kobj, &internal_bitmap_group); +if (ret) +sysfs_remove_group (&mddev->kobj, &common_bitmap_group); + +return ret; +} + +static void bitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_unmerge_group (&mddev->kobj, &internal_bitmap_group); +} +static struct bitmap_operations bitmap_ops = { .head = { .type =MD_BITMAP ,
@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = { .set_pages = bitmap_set_pages, .free = md_bitmap_free, - .group = &md_bitmap_group, + .register_groups = bitmap_register_groups, + .unregister_groups = bitmap_unregister_groups, +}; + +static int none_bitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &common_bitmap_group); +} + +static struct bitmap_operations none_bitmap_ops = { + .head = { + .type =MD_BITMAP , + .id =ID_BITMAP_NONE , + .name ="none" , + }, + + .enabled = dummy_bitmap_enabled, + .create = dummy_bitmap_create, + .destroy = bitmap_destroy, + .load = bitmap_load, + .get_stats = bitmap_get_stats, + .free = md_bitmap_free, + + .register_groups = none_bitmap_register_groups, + .unregister_groups =NULL , };int md_bitmap_init (void ) { +int ret; + md_bitmap_wq =alloc_workqueue ("md_bitmap" ,WQ_MEM_RECLAIM |WQ_UNBOUND ,0 );if (!md_bitmap_wq)return -ENOMEM ; -return register_md_submodule (&bitmap_ops.head); + ret =register_md_submodule (&bitmap_ops.head); +if (ret) +return ret; + +return register_md_submodule (&none_bitmap_ops.head); }void md_bitmap_exit (void ) {destroy_workqueue (md_bitmap_wq);unregister_md_submodule (&bitmap_ops.head); +unregister_md_submodule (&none_bitmap_ops.head); }
drivers/md/md-bitmap.h +3 -0
@@ -125,6 +125,9 @@ struct bitmap_operations {void (*set_pages)(void *data,unsigned long pages);void (*free)(void *data); +int (*register_groups)(struct mddev *mddev); +void (*unregister_groups)(struct mddev *mddev); +struct attribute_group *group; };
drivers/md/md-llbitmap.c +12 -0
@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = { .attrs = md_llbitmap_attrs, }; +static int llbitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &md_llbitmap_group); +} + +static void llbitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_remove_group (&mddev->kobj, &md_llbitmap_group); +} +static struct bitmap_operations llbitmap_ops = { .head = { .type =MD_BITMAP ,
@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = { .dirty_bits = llbitmap_dirty_bits, .write_all = llbitmap_write_all, + .register_groups = llbitmap_register_groups, + .unregister_groups = llbitmap_unregister_groups, .group = &md_llbitmap_group, };
drivers/md/md.c +8 -8
@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head; -if (mddev->bitmap_id ==ID_BITMAP_NONE || - (old && old->head.id == mddev->bitmap_id)) +if (old && old->head.id == mddev->bitmap_id)return true;xa_lock (&md_submodule);
@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) { -if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group)) +if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->register_groups) { +if (mddev->bitmap_ops->register_groups (mddev))pr_warn ("md: cannot register extra bitmap attributes for %s\n" ,mdname (mddev));else
@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) {if (remove_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops && - mddev->bitmap_ops->group) -sysfs_remove_group (&mddev->kobj, mddev->bitmap_ops->group); + mddev->bitmap_ops->unregister_groups) + mddev->bitmap_ops->unregister_groups (mddev); mddev->bitmap_ops =NULL ; }
@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev) (unsigned long long)pers->size(mddev, 0, 0) / 2); err = -EINVAL ; } -if (err ==0 && pers->sync_request && - (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { +if (err ==0 && pers->sync_request) { +if (mddev->bitmap_info.offset ==0 ) + mddev->bitmap_id =ID_BITMAP_NONE ; err =md_bitmap_create (mddev, true);if (err) pr_warn("%s: failed to create bitmap (%d)\n",
Signature · 1 line
2.53.0
Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
Quoted 169 lines · Su Yue
Before commit
fb8cc3b0d9db("md/md-bitmap: delay registration ofbitmap_opsuntil creating bitmap") ifCONFIG_MD_BITMAPis enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.After the commit, if bitmap is none, bitmap/location doesn't exist anymore. It breaks 'grow' behavior of a md array of madam with
MD_FEATURE_BITMAP_OFFSET. Take level=mirror and metadata=1.2 as an example:$ mdadm --create /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \ --metadata=1.2 /dev/vdd /dev/vde $ mdadm --grow /dev/md0 --bitmap=internal $ cat /sys/block/md0/md/bitmap/location Before:+8 After: +2While growing bitmap from none to internal, clustered and llbitmap, mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first. 1)If bitmap/location exists, it sets bitmap/location after
getinfo_super(). 2)If bitmap/location doesn't exist, mdadm just callsmd_set_array_info()thenmddev->bitmap_info.default_offsetwill be used.
Situation can be worse if growing none to clustered, bitmap offset of the node callingmadm --growwill be changed but the other node are reading bitmap sb from the old location.Here introducing a dummy
bitmap_operationsforID_BITMAP_NONEto restore sysfs file bitmap/location forID_BITMAP_NONEandID_BITMAP.location_store()now callsmd_bitmap_(create|destroy)with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")Suggested-by Yu Kuai <[email protected]>Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 116 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 + drivers/md/md-llbitmap.c | 12 ++++ drivers/md/md.c | 16 +++--- 4 files changed, 130 insertions(+), 17 deletions(-)drivers/md/md-bitmap.c +63
-7@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush) bitmap->storage.filemap !=NULL ; } +static bool dummy_bitmap_enabled (void *data,bool flush) +{ +return false; +} +/* * check a page and, if necessary, allocate it (or hijack it if the alloc fails) * @@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)return 0 ; } +static int dummy_bitmap_create (struct mddev *mddev) +{ +return 0 ; +} +static int bitmap_load (struct mddev *mddev) {int err =0 ;@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char *page)return len; } +static int create_internal_group (struct mddev *mddev); +static void remove_internal_group (struct mddev *mddev); +static ssize_t location_store (struct mddev *mddev,const char *buf,size_t len) {@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev, const char *buf, size_t len)*/ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev, true); + rv =md_bitmap_create (mddev, false);if (rv) goto out; + rv =create_internal_group (mddev); +if (rv) { + mddev->bitmap_info.offset =0 ; +bitmap_destroy (mddev); + goto out; + } + rv =bitmap_load (mddev);if (rv) { +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ; -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); goto out; } }@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry max_backlog_used =__ATTR (max_backlog_used,S_IRUGO |S_IWUSR , behind_writes_used_show, behind_writes_used_reset); -static struct attribute *md_bitmap_attrs[] = { - &bitmap_location.attr, +static struct attribute *internal_bitmap_attrs[] = { &bitmap_space.attr, &bitmap_timeout.attr, &bitmap_backlog.attr,@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = { NULL }; -static struct attribute_group md_bitmap_group = { +static struct attribute_group internal_bitmap_group = { .name ="bitmap" , - .attrs = md_bitmap_attrs, + .attrs = internal_bitmap_attrs, }; +/* Only necessary attrs for compatibility */ +static struct attribute *common_bitmap_attrs[] = { + &bitmap_location.attr, +NULL +}; + +static const struct attribute_group common_bitmap_group = { + .name ="bitmap" , + .attrs = common_bitmap_attrs, +}; + +static int create_internal_group (struct mddev *mddev) +{ +/* +* md_bitmap_group and md_bitmap_common_group are using same name +* 'bitmap'. +*/ +return sysfs_merge_group (&mddev->kobj, &internal_bitmap_group); +} + +static void remove_internal_group (struct mddev *mddev) +{ +sysfs_unmerge_group (&mddev->kobj, &internal_bitmap_group); +} + +static int bitmap_register_groups (struct mddev *mddev) +{ +int ret; + + ret =sysfs_create_group (&mddev->kobj, &common_bitmap_group); + +if (ret) +return ret; + + ret =sysfs_merge_group (&mddev->kobj, &internal_bitmap_group);
drivers/md/md-bitmap.c static struct attribute *md_bitmap_attrs[] = { · line 3045
Quoted 10 lines
drivers/md/md-bitmap.c +10
-0+if (ret) +sysfs_remove_group (&mddev->kobj, &common_bitmap_group); + +return ret; +} + +static void bitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_unmerge_group (&mddev->kobj, &internal_bitmap_group); +}
Hi Su
bitmap_unregister_groups should also remove common_bitmap_group, right?
drivers/md/md-bitmap.c static struct attribute *md_bitmap_attrs[] = { · line 3047
Quoted 2 lines
drivers/md/md-bitmap.c +1
-0+static struct bitmap_operations bitmap_ops = {
You already changed md_bitmap_group to internal_bitmap_group. It's better to change bitmap_ops to internal_bitmap_ops?
Quoted 22 lines
drivers/md/md-bitmap.c +15
-1.head = { .type = MD_BITMAP,@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = { .set_pages = bitmap_set_pages, .free = md_bitmap_free, - .group = &md_bitmap_group, + .register_groups = bitmap_register_groups, + .unregister_groups = bitmap_unregister_groups, +}; + +static int none_bitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &common_bitmap_group); +} + +static struct bitmap_operations none_bitmap_ops = { + .head = { + .type =MD_BITMAP , + .id =ID_BITMAP_NONE , + .name ="none" , + },
drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112
Quoted 10 lines
drivers/md/md-bitmap.c +10
-0+ + .enabled = dummy_bitmap_enabled, + .create = dummy_bitmap_create, + .destroy = bitmap_destroy, + .load = bitmap_load, + .get_stats = bitmap_get_stats, + .free = md_bitmap_free, + + .register_groups = none_bitmap_register_groups, + .unregister_groups = NULL,
How does bitmap/location can be deleted if array is created with no bitmap? Can the mdadm --stop command get stuck?
Best Regards
Xiao
Quoted 120 lines
drivers/md/md-bitmap.c +8
-1};int md_bitmap_init (void ) { +int ret; + md_bitmap_wq =alloc_workqueue ("md_bitmap" ,WQ_MEM_RECLAIM |WQ_UNBOUND ,0 );if (!md_bitmap_wq)return -ENOMEM ; -return register_md_submodule (&bitmap_ops.head); + ret =register_md_submodule (&bitmap_ops.head); +if (ret) +return ret; + +return register_md_submodule (&none_bitmap_ops.head); }void md_bitmap_exit (void ) {destroy_workqueue (md_bitmap_wq);unregister_md_submodule (&bitmap_ops.head); +unregister_md_submodule (&none_bitmap_ops.head); }drivers/md/md-bitmap.h +3
-0@@ -125,6 +125,9 @@ struct bitmap_operations {void (*set_pages)(void *data,unsigned long pages);void (*free)(void *data); +int (*register_groups)(struct mddev *mddev); +void (*unregister_groups)(struct mddev *mddev); +struct attribute_group *group; };drivers/md/md-llbitmap.c +12
-0@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = { .attrs = md_llbitmap_attrs, }; +static int llbitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &md_llbitmap_group); +} + +static void llbitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_remove_group (&mddev->kobj, &md_llbitmap_group); +} +static struct bitmap_operations llbitmap_ops = { .head = { .type =MD_BITMAP ,@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = { .dirty_bits = llbitmap_dirty_bits, .write_all = llbitmap_write_all, + .register_groups = llbitmap_register_groups, + .unregister_groups = llbitmap_unregister_groups, .group = &md_llbitmap_group, };drivers/md/md.c +8
-9@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head; -if (mddev->bitmap_id ==ID_BITMAP_NONE || - (old && old->head.id == mddev->bitmap_id)) +if (old && old->head.id == mddev->bitmap_id)return true;xa_lock (&md_submodule);@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) { -if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group)) +if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->register_groups) { +if (mddev->bitmap_ops->register_groups (mddev))pr_warn ("md: cannot register extra bitmap attributes for %s\n" ,mdname (mddev));else @@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) {if (remove_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops && - mddev->bitmap_ops->group) -sysfs_remove_group (&mddev->kobj, mddev->bitmap_ops->group); + mddev->bitmap_ops->unregister_groups) + mddev->bitmap_ops->unregister_groups (mddev); mddev->bitmap_ops =NULL ; }@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev) (unsigned long long)pers->size(mddev, 0, 0) / 2); err = -EINVAL ; } -if (err ==0 && pers->sync_request && - (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { +if (err ==0 && pers->sync_request) { +if (mddev->bitmap_info.offset ==0 ) + mddev->bitmap_id =ID_BITMAP_NONE ; err =md_bitmap_create (mddev, true);if (err) pr_warn("%s: failed to create bitmap (%d)\n", --2.53.0
Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
Quoted 203 lines · Xiao Ni
Quoted 197 lines · Su Yue
Before commit
fb8cc3b0d9db("md/md-bitmap: delay registration ofbitmap_opsuntil creating bitmap") ifCONFIG_MD_BITMAPis enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.After the commit, if bitmap is none, bitmap/location doesn't exist anymore. It breaks 'grow' behavior of a md array of madam with
MD_FEATURE_BITMAP_OFFSET. Take level=mirror and metadata=1.2 as an example:$ mdadm --create /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \ --metadata=1.2 /dev/vdd /dev/vde $ mdadm --grow /dev/md0 --bitmap=internal $ cat /sys/block/md0/md/bitmap/location Before:+8 After: +2While growing bitmap from none to internal, clustered and llbitmap, mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first. 1)If bitmap/location exists, it sets bitmap/location after
getinfo_super(). 2)If bitmap/location doesn't exist, mdadm just callsmd_set_array_info()thenmddev->bitmap_info.default_offsetwill be used.
Situation can be worse if growing none to clustered, bitmap offset of the node callingmadm --growwill be changed but the other node are reading bitmap sb from the old location.Here introducing a dummy
bitmap_operationsforID_BITMAP_NONEto restore sysfs file bitmap/location forID_BITMAP_NONEandID_BITMAP.location_store()now callsmd_bitmap_(create|destroy)with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")Suggested-by Yu Kuai <[email protected]>Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 116 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 + drivers/md/md-llbitmap.c | 12 ++++ drivers/md/md.c | 16 +++--- 4 files changed, 130 insertions(+), 17 deletions(-)drivers/md/md-bitmap.c +5
-0@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush) bitmap->storage.filemap != NULL; } +static bool dummy_bitmap_enabled (void *data,bool flush) +{ +return false; +} +/* * check a page and, if necessary, allocate it (or hijack it if the alloc fails)
*drivers/md/md-bitmap.c +5
-0@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)return 0 ; } +static int dummy_bitmap_create (struct mddev *mddev) +{ +return 0 ; +} +static int bitmap_load (struct mddev *mddev) {int err = 0;drivers/md/md-bitmap.c +3
-0@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char *page)return len; } +static int create_internal_group (struct mddev *mddev); +static void remove_internal_group (struct mddev *mddev); +static ssize_t location_store(struct mddev *mddev, const char *buf, size_tlen)
{drivers/md/md-bitmap.c +2
-1@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f =
mddev->bitmap_info.file;drivers/md/md-bitmap.c +10
-2@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,const char *buf, size_t len) */ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev, true); + rv =md_bitmap_create (mddev, false);if (rv) goto out; + rv =create_internal_group (mddev); +if (rv) { + mddev->bitmap_info.offset =0 ; +bitmap_destroy (mddev); + goto out; + } + rv =bitmap_load (mddev);if (rv) { +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ; -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); goto out;} }drivers/md/md-bitmap.c +0
-0@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry
max_backlog_used=
__ATTR(max_backlog_used,S_IRUGO|S_IWUSR,behind_writes_used_show, behind_writes_used_reset);drivers/md/md-bitmap.c +18
-4-static struct attribute *md_bitmap_attrs[] = { - &bitmap_location.attr, +static struct attribute *internal_bitmap_attrs[] = { &bitmap_space.attr, &bitmap_timeout.attr, &bitmap_backlog.attr,@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = {NULL }; -static struct attribute_group md_bitmap_group = { +static struct attribute_group internal_bitmap_group = { .name ="bitmap" , - .attrs = md_bitmap_attrs, + .attrs = internal_bitmap_attrs, }; +/* Only necessary attrs for compatibility */ +static struct attribute *common_bitmap_attrs[] = { + &bitmap_location.attr, +NULL +}; + +static const struct attribute_group common_bitmap_group = { + .name ="bitmap" , + .attrs = common_bitmap_attrs, +}; + +static int create_internal_group (struct mddev *mddev) +{ +/* +* md_bitmap_group and md_bitmap_common_group are using same name
drivers/md/md-bitmap.c +3
-0+* 'bitmap'. +*/ + return sysfs_merge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +5
-0+} + +static void remove_internal_group (struct mddev *mddev) +{ + sysfs_unmerge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +7
-0+} + +static int bitmap_register_groups (struct mddev *mddev) +{ +int ret; + + ret = sysfs_create_group(&mddev->kobj,
&common_bitmap_group);drivers/md/md-bitmap.c +5
-0+ +if (ret) +return ret; + + ret = sysfs_merge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +2
-0+if (ret) + sysfs_remove_group(&mddev->kobj,
&common_bitmap_group);drivers/md/md-bitmap.c +7
-0+ +return ret; +} + +static void bitmap_unregister_groups (struct mddev *mddev) +{ + sysfs_unmerge_group(&mddev->kobj,Quoted 2 lines
&internal_bitmap_group);
+}Hi Su
Quoted 2 lines
bitmap_unregister_groupsshould also removecommon_bitmap_group, right?
As you pasted before, mdadm:Grow.c:
int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s) {
...
if (array.state & (1 << MD_SB_BITMAP_PRESENT)) {
if (s->btype == BitmapNone) {
array.state &= ~(1 << MD_SB_BITMAP_PRESENT);
if (md_set_array_info(fd, &array) != 0) {
if (array.state & (1 << MD_SB_CLUSTERED))
pr_err("failed to remove clustered
bitmap.\n");
else
pr_err("failed to remove internal bitmap.\n");
return 1;
}
return 0;
}
pr_err("bitmap already present on %s\n", devname);
return 1;
}...
}
In case of growing from internal to none, bitmap_unregister_groups() will be called, if common_bitmap_group is removed, bitmap/location won't exist. update_array_info() is a common function used by many call paths. Also llbitmap is involved here. I don't want to make situation and code more complicated like adding more codes in update_array_info().
Quoted 35 lines
Quoted 2 lines
+
static structbitmap_operationsbitmap_ops= {You already changed
md_bitmap_grouptointernal_bitmap_group. It's better to changebitmap_opstointernal_bitmap_ops?Quoted 29 lines
.head = {
.type =MD_BITMAP,drivers/md/md-bitmap.c +20
-1@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = { .set_pages = bitmap_set_pages, .free = md_bitmap_free, - .group = &md_bitmap_group, + .register_groups = bitmap_register_groups, + .unregister_groups = bitmap_unregister_groups, +}; + +static int none_bitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &common_bitmap_group); +} + +static struct bitmap_operations none_bitmap_ops = { + .head = { + .type =MD_BITMAP , + .id =ID_BITMAP_NONE , + .name ="none" , + }, + + .enabled = dummy_bitmap_enabled, + .create = dummy_bitmap_create, + .destroy = bitmap_destroy, + .load = bitmap_load,
Quoted 9 lines
drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112
Quoted 5 lines
drivers/md/md-bitmap.c +5
-0+ .get_stats = bitmap_get_stats, + .free = md_bitmap_free, + + .register_groups = none_bitmap_register_groups, + .unregister_groups = NULL,How does bitmap/location can be deleted if array is created with no bitmap? Can the
mdadm --stopcommand get stuck?
No. It wont get stuck.
I guess here your concern is the timing of removing common_bitmap_group. The life cycle is same as before fb8cc3b0d9db. At the time llbitmap is not introduced, because there is no need to switch bitmap ops, so all attrs of bitmap are removed in kobject_put() in mddev_delayed_delete() queued by mddev_put(). This is now where common_bitmap_group is being removed.
--
Su
Quoted 140 lines
Best Regards
XiaoQuoted 137 lines
drivers/md/md-bitmap.c +8
-1};int md_bitmap_init (void ) { +int ret; + md_bitmap_wq =alloc_workqueue ("md_bitmap" ,WQ_MEM_RECLAIM |WQ_UNBOUND ,0 );if (!md_bitmap_wq)return -ENOMEM ; -return register_md_submodule (&bitmap_ops.head); + ret =register_md_submodule (&bitmap_ops.head); +if (ret) +return ret; + +return register_md_submodule (&none_bitmap_ops.head); }void md_bitmap_exit (void ) {destroy_workqueue (md_bitmap_wq);unregister_md_submodule (&bitmap_ops.head); +unregister_md_submodule (&none_bitmap_ops.head); }drivers/md/md-bitmap.h +3
-0@@ -125,6 +125,9 @@ struct bitmap_operations {void (*set_pages)(void *data,unsigned long pages);void (*free)(void *data); +int (*register_groups)(struct mddev *mddev); +void (*unregister_groups)(struct mddev *mddev); +struct attribute_group *group; };drivers/md/md-llbitmap.c +10
-0@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = { .attrs = md_llbitmap_attrs, }; +static int llbitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &md_llbitmap_group); +} + +static void llbitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_remove_group (&mddev->kobj, &md_llbitmap_group); +} +static struct bitmap_operations llbitmap_ops = {.head = {
.type =MD_BITMAP,drivers/md/md-llbitmap.c +2
-0@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = { .dirty_bits = llbitmap_dirty_bits, .write_all = llbitmap_write_all, + .register_groups = llbitmap_register_groups, + .unregister_groups = llbitmap_unregister_groups, .group = &md_llbitmap_group, };drivers/md/md.c +1
-2@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head; -if (mddev->bitmap_id ==ID_BITMAP_NONE || - (old && old->head.id == mddev->bitmap_id)) +if (old && old->head.id == mddev->bitmap_id)return true;xa_lock(&md_submodule);drivers/md/md.c +1
-2@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) { -if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group)) + if (create_sysfs && !mddev_is_dm(mddev) &&
mddev->bitmap_ops->register_groups) {drivers/md/md.c +1
-2+if (mddev->bitmap_ops->register_groups (mddev)) pr_warn("md: cannot register extra bitmap attributes for %s\n", mdname(mddev));else @@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) { if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops && - mddev->bitmap_ops->group) - sysfs_remove_group(&mddev->kobj,
mddev->bitmap_ops->group);drivers/md/md.c +5
-2+ mddev->bitmap_ops->unregister_groups) + mddev->bitmap_ops->unregister_groups (mddev); mddev->bitmap_ops =NULL ; }@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev) (unsigned long long )pers->size (mddev, 0, 0) / 2); err = -EINVAL ; } -if (err ==0 && pers->sync_request && - (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { +if (err ==0 && pers->sync_request) { +if (mddev->bitmap_info.offset ==0 ) + mddev->bitmap_id =ID_BITMAP_NONE ; err =md_bitmap_create (mddev, true);if (err)
pr_warn("%s: failed to create bitmap (%d)\n",
--
2.53.0
Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
Quoted 249 lines · Su Yue
Quoted 217 lines · Xiao Ni
Quoted 211 lines · Su Yue
Before commit
fb8cc3b0d9db("md/md-bitmap: delay registration ofbitmap_opsuntil creating bitmap")
ifCONFIG_MD_BITMAPis enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.After the commit, if bitmap is none, bitmap/location doesn't exist anymore.
It breaks 'grow' behavior of a md array of madam withMD_FEATURE_BITMAP_OFFSET.
Take level=mirror and metadata=1.2 as an example:$ mdadm --create /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \ --metadata=1.2 /dev/vdd /dev/vde $ mdadm --grow /dev/md0 --bitmap=internal $ cat /sys/block/md0/md/bitmap/location Before:+8 After: +2While growing bitmap from none to internal, clustered and llbitmap,
mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first.
1)If bitmap/location exists, it sets bitmap/location aftergetinfo_super().
2)If bitmap/location doesn't exist, mdadm just callsmd_set_array_info()thenmddev->bitmap_info.default_offsetwill be used. Situation can be worse if growing none to clustered, bitmap offset of the node
callingmadm --growwill be changed but the other node are reading bitmap sb from
the old location.Here introducing a dummy
bitmap_operationsforID_BITMAP_NONEto restore sysfs
file bitmap/location forID_BITMAP_NONEandID_BITMAP.location_store()now callsmd_bitmap_(create|destroy)with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of
bitmap_opsuntil creating bitmap")Suggested-by Yu Kuai <[email protected]>Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 116 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 + drivers/md/md-llbitmap.c | 12 ++++ drivers/md/md.c | 16 +++--- 4 files changed, 130 insertions(+), 17 deletions(-)drivers/md/md-bitmap.c +5
-0@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush) bitmap->storage.filemap != NULL; } +static bool dummy_bitmap_enabled (void *data,bool flush) +{ +return false; +} +/* * check a page and, if necessary, allocate it (or hijack it if the alloc fails)
*drivers/md/md-bitmap.c +5
-0@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)return 0 ; } +static int dummy_bitmap_create (struct mddev *mddev) +{ +return 0 ; +} +static int bitmap_load (struct mddev *mddev) {int err = 0;drivers/md/md-bitmap.c +3
-0@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char *page)return len; } +static int create_internal_group (struct mddev *mddev); +static void remove_internal_group (struct mddev *mddev); +static ssize_t location_store(struct mddev *mddev, const char *buf, size_tlen)
{drivers/md/md-bitmap.c +2
-1@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const char *buf, size_t len) goto out; } -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f =
mddev->bitmap_info.file;drivers/md/md-bitmap.c +10
-2@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,const char *buf, size_t len) */ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev, true); + rv =md_bitmap_create (mddev, false);if (rv) goto out; + rv =create_internal_group (mddev); +if (rv) { + mddev->bitmap_info.offset =0 ; +bitmap_destroy (mddev); + goto out; + } + rv =bitmap_load (mddev);if (rv) { +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ; -md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); goto out;} }drivers/md/md-bitmap.c +0
-0@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry
max_backlog_used=
__ATTR(max_backlog_used,S_IRUGO|S_IWUSR,behind_writes_used_show, behind_writes_used_reset);drivers/md/md-bitmap.c +18
-4-static struct attribute *md_bitmap_attrs[] = { - &bitmap_location.attr, +static struct attribute *internal_bitmap_attrs[] = { &bitmap_space.attr, &bitmap_timeout.attr, &bitmap_backlog.attr,@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = {NULL }; -static struct attribute_group md_bitmap_group = { +static struct attribute_group internal_bitmap_group = { .name ="bitmap" , - .attrs = md_bitmap_attrs, + .attrs = internal_bitmap_attrs, }; +/* Only necessary attrs for compatibility */ +static struct attribute *common_bitmap_attrs[] = { + &bitmap_location.attr, +NULL +}; + +static const struct attribute_group common_bitmap_group = { + .name ="bitmap" , + .attrs = common_bitmap_attrs, +}; + +static int create_internal_group (struct mddev *mddev) +{ +/* +* md_bitmap_group and md_bitmap_common_group are using same name
drivers/md/md-bitmap.c +3
-0+* 'bitmap'. +*/ + return sysfs_merge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +5
-0+} + +static void remove_internal_group (struct mddev *mddev) +{ + sysfs_unmerge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +7
-0+} + +static int bitmap_register_groups (struct mddev *mddev) +{ +int ret; + + ret = sysfs_create_group(&mddev->kobj,
&common_bitmap_group);drivers/md/md-bitmap.c +5
-0+ +if (ret) +return ret; + + ret = sysfs_merge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +2
-0+if (ret) + sysfs_remove_group(&mddev->kobj,
&common_bitmap_group);drivers/md/md-bitmap.c +7
-0+ +return ret; +} + +static void bitmap_unregister_groups (struct mddev *mddev) +{ + sysfs_unmerge_group(&mddev->kobj,Quoted 2 lines
&internal_bitmap_group);
+}Hi Su
Quoted 3 lines
bitmap_unregister_groupsshould also removecommon_bitmap_group, right?As you pasted before, mdadm:Grow.c:
int
Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
{
...if (array.state & (1 << MD_SB_BITMAP_PRESENT)) { if (s->btype == BitmapNone) { array.state &= ~(1 << MD_SB_BITMAP_PRESENT); if (md_set_array_info(fd, &array) != 0) { if (array.state & (1 << MD_SB_CLUSTERED)) pr_err("failed to remove clustered bitmap.\n"); else pr_err("failed to remove internal bitmap.\n"); return 1; } return 0; } pr_err("bitmap already present on %s\n", devname); return 1; }...
}
Quoted 9 lines
In case of growing from internal to none,
bitmap_unregister_groups()will
be called, ifcommon_bitmap_groupis removed, bitmap/location won't exist.update_array_info()is a common function used by many call paths. Also
llbitmap is involved here. I don't want to make situation and code more
complicated like adding more codes inupdate_array_info().
The above mdadm codes use bitmap/location. In your patch, you already pass false to md_bitmap_destroy in location_store. So removing common_bitmap_group in bitmap_unregister_groups can't affect the above case.
But after reading your below comments. It's good to me that bitmap_unregister_groups doesn't remove common_bitmap_group.
Quoted 50 lines
Quoted 39 lines
Quoted 4 lines
+
static structbitmap_operationsbitmap_ops= {You already changed
md_bitmap_grouptointernal_bitmap_group. It's
better to changebitmap_opstointernal_bitmap_ops?Quoted 30 lines
.head = {
.type =MD_BITMAP,drivers/md/md-bitmap.c +21
-1@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = { .set_pages = bitmap_set_pages, .free = md_bitmap_free, - .group = &md_bitmap_group, + .register_groups = bitmap_register_groups, + .unregister_groups = bitmap_unregister_groups, +}; + +static int none_bitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &common_bitmap_group); +} + +static struct bitmap_operations none_bitmap_ops = { + .head = { + .type =MD_BITMAP , + .id =ID_BITMAP_NONE , + .name ="none" , + }, + + .enabled = dummy_bitmap_enabled, + .create = dummy_bitmap_create, + .destroy = bitmap_destroy, + .load = bitmap_load, + .get_stats = bitmap_get_stats,Quoted 9 lines
drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112
Quoted 4 lines
drivers/md/md-bitmap.c +4
-0+ .free = md_bitmap_free, + + .register_groups = none_bitmap_register_groups, + .unregister_groups = NULL,How does bitmap/location can be deleted if array is created with no
bitmap? Can themdadm --stopcommand get stuck?No. It wont get stuck.
Quoted 10 lines
I guess here your concern is the timing of removing
common_bitmap_group.
The life cycle is same as beforefb8cc3b0d9db. At the time llbitmap is
not introduced, because there is no need to switch bitmap ops, so all
attrs of bitmap are removed inkobject_put()inmddev_delayed_delete()
queued bymddev_put(). This is now wherecommon_bitmap_groupis being removed.
Thanks for the explanation.
Best Regards
Xiao
Quoted 145 lines
--
SuQuoted 141 lines
Best Regards
XiaoQuoted 138 lines
drivers/md/md-bitmap.c +8
-1};int md_bitmap_init (void ) { +int ret; + md_bitmap_wq =alloc_workqueue ("md_bitmap" ,WQ_MEM_RECLAIM |WQ_UNBOUND ,0 );if (!md_bitmap_wq)return -ENOMEM ; -return register_md_submodule (&bitmap_ops.head); + ret =register_md_submodule (&bitmap_ops.head); +if (ret) +return ret; + +return register_md_submodule (&none_bitmap_ops.head); }void md_bitmap_exit (void ) {destroy_workqueue (md_bitmap_wq);unregister_md_submodule (&bitmap_ops.head); +unregister_md_submodule (&none_bitmap_ops.head); }drivers/md/md-bitmap.h +3
-0@@ -125,6 +125,9 @@ struct bitmap_operations {void (*set_pages)(void *data,unsigned long pages);void (*free)(void *data); +int (*register_groups)(struct mddev *mddev); +void (*unregister_groups)(struct mddev *mddev); +struct attribute_group *group; };drivers/md/md-llbitmap.c +10
-0@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = { .attrs = md_llbitmap_attrs, }; +static int llbitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &md_llbitmap_group); +} + +static void llbitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_remove_group (&mddev->kobj, &md_llbitmap_group); +} +static struct bitmap_operations llbitmap_ops = {.head = {
.type =MD_BITMAP,drivers/md/md-llbitmap.c +2
-0@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = { .dirty_bits = llbitmap_dirty_bits, .write_all = llbitmap_write_all, + .register_groups = llbitmap_register_groups, + .unregister_groups = llbitmap_unregister_groups, .group = &md_llbitmap_group, };drivers/md/md.c +1
-2@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head; -if (mddev->bitmap_id ==ID_BITMAP_NONE || - (old && old->head.id == mddev->bitmap_id)) +if (old && old->head.id == mddev->bitmap_id)return true;xa_lock(&md_submodule);drivers/md/md.c +1
-2@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) { -if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group)) + if (create_sysfs && !mddev_is_dm(mddev) &&
mddev->bitmap_ops->register_groups) {drivers/md/md.c +1
-2+if (mddev->bitmap_ops->register_groups (mddev)) pr_warn("md: cannot register extra bitmap attributes for %s\n", mdname(mddev));else @@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) { if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops && - mddev->bitmap_ops->group) - sysfs_remove_group(&mddev->kobj,
mddev->bitmap_ops->group);drivers/md/md.c +5
-2+ mddev->bitmap_ops->unregister_groups) + mddev->bitmap_ops->unregister_groups (mddev); mddev->bitmap_ops =NULL ; }@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev) (unsigned long long )pers->size (mddev, 0, 0) / 2); err = -EINVAL ; } -if (err ==0 && pers->sync_request && - (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { +if (err ==0 && pers->sync_request) { +if (mddev->bitmap_info.offset ==0 ) + mddev->bitmap_id =ID_BITMAP_NONE ; err =md_bitmap_create (mddev, true);if (err)
pr_warn("%s: failed to create bitmap (%d)\n",
--
2.53.0
Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
Quoted 276 lines · Xiao Ni
Quoted 265 lines · Su Yue
Quoted 226 lines · Xiao Ni
Quoted 220 lines · Su Yue
Before commit
fb8cc3b0d9db("md/md-bitmap: delay registration ofbitmap_opsuntil creating bitmap")
ifCONFIG_MD_BITMAPis enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.After the commit, if bitmap is none, bitmap/location doesn't exist anymore.
It breaks 'grow' behavior of a md array of madam withMD_FEATURE_BITMAP_OFFSET.
Take level=mirror and metadata=1.2 as an example:$ mdadm --create /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \ --metadata=1.2 /dev/vdd /dev/vde $ mdadm --grow /dev/md0 --bitmap=internal $ cat /sys/block/md0/md/bitmap/location Before:+8 After: +2While growing bitmap from none to internal, clustered and llbitmap,
mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first.
1)If bitmap/location exists, it sets bitmap/location aftergetinfo_super().
2)If bitmap/location doesn't exist, mdadm just callsmd_set_array_info()thenmddev->bitmap_info.default_offsetwill be used. Situation can be worse if growing none to clustered, bitmap offset of the node
callingmadm --growwill be changed but the other node are reading bitmap sb from
the old location.Here introducing a dummy
bitmap_operationsforID_BITMAP_NONEto restore sysfs
file bitmap/location forID_BITMAP_NONEandID_BITMAP.location_store()now callsmd_bitmap_(create|destroy)with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of
bitmap_opsuntil creating bitmap")Suggested-by Yu Kuai <[email protected]>Signed-off-by Su Yue <[email protected]>--- drivers/md/md-bitmap.c | 116 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 + drivers/md/md-llbitmap.c | 12 ++++ drivers/md/md.c | 16 +++--- 4 files changed, 130 insertions(+), 17 deletions(-)drivers/md/md-bitmap.c +5
-0@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush) bitmap->storage.filemap != NULL; } +static bool dummy_bitmap_enabled (void *data,bool flush) +{ +return false; +} +/* * check a page and, if necessary, allocate it (or hijack it
if the alloc fails)
*drivers/md/md-bitmap.c +5
-0@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)return 0 ; } +static int dummy_bitmap_create (struct mddev *mddev) +{ +return 0 ; +} +static int bitmap_load (struct mddev *mddev) {int err = 0;drivers/md/md-bitmap.c +0
-0@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev,char
*page)return len;}
drivers/md/md-bitmap.c +3
-0+static int create_internal_group (struct mddev *mddev); +static void remove_internal_group (struct mddev *mddev); +static ssize_t location_store (struct mddev *mddev,const char *buf,size_t len) {@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev,const
char *buf,size_tlen)goto out; }drivers/md/md-bitmap.c +11
-3-md_bitmap_destroy (mddev, true); +md_bitmap_destroy (mddev, false); +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ;if (mddev->bitmap_info.file) {struct file *f = mddev->bitmap_info.file;@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,const char *buf, size_t len) */ mddev->bitmap_id =ID_BITMAP ; mddev->bitmap_info.offset = offset; - rv =md_bitmap_create (mddev, true); + rv =md_bitmap_create (mddev, false);if (rv) goto out; + rv =create_internal_group (mddev); +if (rv) { + mddev->bitmap_info.offset =0 ; +bitmap_destroy (mddev); + goto out; + } + rv =bitmap_load (mddev);if (rv) { +remove_internal_group (mddev); mddev->bitmap_info.offset =0 ; - md_bitmap_destroy(mddev,true);
+ md_bitmap_destroy(mddev,false); goto out; } }drivers/md/md-bitmap.c +0
-0@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry
max_backlog_used=
__ATTR(max_backlog_used,S_IRUGO|S_IWUSR,behind_writes_used_show, behind_writes_used_reset);drivers/md/md-bitmap.c +18
-4-static struct attribute *md_bitmap_attrs[] = { - &bitmap_location.attr, +static struct attribute *internal_bitmap_attrs[] = { &bitmap_space.attr, &bitmap_timeout.attr, &bitmap_backlog.attr,@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = {NULL }; -static struct attribute_group md_bitmap_group = { +static struct attribute_group internal_bitmap_group = { .name ="bitmap" , - .attrs = md_bitmap_attrs, + .attrs = internal_bitmap_attrs, }; +/* Only necessary attrs for compatibility */ +static struct attribute *common_bitmap_attrs[] = { + &bitmap_location.attr, +NULL +}; + +static const struct attribute_group common_bitmap_group = { + .name ="bitmap" , + .attrs = common_bitmap_attrs, +}; + +static int create_internal_group (struct mddev *mddev) +{ +/* +* md_bitmap_group and md_bitmap_common_group are using
same namedrivers/md/md-bitmap.c +3
-0+* 'bitmap'. +*/ + return sysfs_merge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +5
-0+} + +static void remove_internal_group (struct mddev *mddev) +{ + sysfs_unmerge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +7
-0+} + +static int bitmap_register_groups (struct mddev *mddev) +{ +int ret; + + ret = sysfs_create_group(&mddev->kobj,
&common_bitmap_group);drivers/md/md-bitmap.c +5
-0+ +if (ret) +return ret; + + ret = sysfs_merge_group(&mddev->kobj,
&internal_bitmap_group);drivers/md/md-bitmap.c +2
-0+if (ret) + sysfs_remove_group(&mddev->kobj,
&common_bitmap_group);drivers/md/md-bitmap.c +7
-0+ +return ret; +} + +static void bitmap_unregister_groups (struct mddev *mddev) +{ + sysfs_unmerge_group(&mddev->kobj,Quoted 2 lines
&internal_bitmap_group);
+}Hi Su
Quoted 3 lines
bitmap_unregister_groupsshould also removecommon_bitmap_group, right?As you pasted before, mdadm:Grow.c:
int
Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
{
...if (array.state & (1 << MD_SB_BITMAP_PRESENT)) { if (s->btype == BitmapNone) { array.state &= ~(1 << MD_SB_BITMAP_PRESENT); if (md_set_array_info(fd, &array) != 0) { if (array.state & (1 << MD_SB_CLUSTERED)) pr_err("failed to remove clustered bitmap.\n"); else pr_err("failed to remove internal bitmap.\n"); return 1; } return 0; } pr_err("bitmap already present on %s\n", devname); return 1; }...
}Quoted 9 lines
In case of growing from internal to none,
bitmap_unregister_groups()will
be called, ifcommon_bitmap_groupis removed, bitmap/location won't exist.update_array_info()is a common function used by many call paths. Also
llbitmap is involved here. I don't want to make situation and code more
complicated like adding more codes inupdate_array_info().
Quoted 5 lines
The above mdadm codes use bitmap/location. In your patch, you already pass false to
md_bitmap_destroyinlocation_store. So removingcommon_bitmap_groupinbitmap_unregister_groupscan't affect the above case.
Right. Got confusion between md_set_array_info() and update_array_info().
Quoted 4 lines
But after reading your below comments. It's good to me that
bitmap_unregister_groupsdoesn't removecommon_bitmap_group.
Thanks.
--
Su
Quoted 216 lines
Quoted 51 lines
Quoted 40 lines
Quoted 4 lines
+
static structbitmap_operationsbitmap_ops= {You already changed
md_bitmap_grouptointernal_bitmap_group. It's
better to changebitmap_opstointernal_bitmap_ops?Quoted 31 lines
.head = {
.type =MD_BITMAP,drivers/md/md-bitmap.c +22
-1@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = { .set_pages = bitmap_set_pages, .free = md_bitmap_free, - .group = &md_bitmap_group, + .register_groups = bitmap_register_groups, + .unregister_groups = bitmap_unregister_groups, +}; + +static int none_bitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &common_bitmap_group); +} + +static struct bitmap_operations none_bitmap_ops = { + .head = { + .type =MD_BITMAP , + .id =ID_BITMAP_NONE , + .name ="none" , + }, + + .enabled = dummy_bitmap_enabled, + .create = dummy_bitmap_create, + .destroy = bitmap_destroy, + .load = bitmap_load, + .get_stats = bitmap_get_stats, + .free = md_bitmap_free,Quoted 9 lines
drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112
Quoted 4 lines
drivers/md/md-bitmap.c +3
-0+ + .register_groups = none_bitmap_register_groups, + .unregister_groups = NULL,How does bitmap/location can be deleted if array is created with no
bitmap? Can themdadm --stopcommand get stuck?No. It wont get stuck.
Quoted 10 lines
I guess here your concern is the timing of removing
common_bitmap_group.
The life cycle is same as beforefb8cc3b0d9db. At the time llbitmap is
not introduced, because there is no need to switch bitmap ops, so all
attrs of bitmap are removed inkobject_put()inmddev_delayed_delete()
queued bymddev_put(). This is now wherecommon_bitmap_groupis being removed.Thanks for the explanation.
Best Regards
XiaoQuoted 150 lines
--
SuQuoted 146 lines
Best Regards
XiaoQuoted 143 lines
drivers/md/md-bitmap.c +8
-1};int md_bitmap_init (void ) { +int ret; + md_bitmap_wq =alloc_workqueue ("md_bitmap" ,WQ_MEM_RECLAIM |WQ_UNBOUND ,0 );if (!md_bitmap_wq)return -ENOMEM ; -return register_md_submodule (&bitmap_ops.head); + ret =register_md_submodule (&bitmap_ops.head); +if (ret) +return ret; + +return register_md_submodule (&none_bitmap_ops.head); }void md_bitmap_exit (void ) {destroy_workqueue (md_bitmap_wq);unregister_md_submodule (&bitmap_ops.head); +unregister_md_submodule (&none_bitmap_ops.head); }drivers/md/md-bitmap.h +3
-0@@ -125,6 +125,9 @@ struct bitmap_operations {void (*set_pages)(void *data,unsigned long pages);void (*free)(void *data); +int (*register_groups)(struct mddev *mddev); +void (*unregister_groups)(struct mddev *mddev); +struct attribute_group *group; };drivers/md/md-llbitmap.c +10
-0@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = { .attrs = md_llbitmap_attrs, }; +static int llbitmap_register_groups (struct mddev *mddev) +{ +return sysfs_create_group (&mddev->kobj, &md_llbitmap_group); +} + +static void llbitmap_unregister_groups (struct mddev *mddev) +{ +sysfs_remove_group (&mddev->kobj, &md_llbitmap_group); +} +static struct
bitmap_operationsllbitmap_ops= {
.head = {
.type =MD_BITMAP,drivers/md/md-llbitmap.c +2
-0@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = { .dirty_bits = llbitmap_dirty_bits, .write_all = llbitmap_write_all, + .register_groups = llbitmap_register_groups, + .unregister_groups = llbitmap_unregister_groups, .group = &md_llbitmap_group,};
drivers/md/md.c +1
-2@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)struct bitmap_operations *old = mddev->bitmap_ops;struct md_submodule_head *head; -if (mddev->bitmap_id ==ID_BITMAP_NONE || - (old && old->head.id == mddev->bitmap_id)) +if (old && old->head.id == mddev->bitmap_id)return true;xa_lock(&md_submodule);drivers/md/md.c +1
-2@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs) mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (create_sysfs && !mddev_is_dm (mddev) && mddev->bitmap_ops->group) { -if (sysfs_create_group (&mddev->kobj, mddev->bitmap_ops->group)) + if (create_sysfs && !mddev_is_dm(mddev) &&
mddev->bitmap_ops->register_groups) {
+ if
(mddev->bitmap_ops->register_groups(mddev))
pr_warn("md: cannot register extra bitmap attributes for %s\n",
mdname(mddev));
elsedrivers/md/md.c +0
-2@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)static void mddev_clear_bitmap_ops (struct mddev *mddev,bool remove_sysfs) { if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops && - mddev->bitmap_ops->group) - sysfs_remove_group(&mddev->kobj,
mddev->bitmap_ops->group);drivers/md/md.c +5
-2+ mddev->bitmap_ops->unregister_groups) + mddev->bitmap_ops->unregister_groups (mddev); mddev->bitmap_ops =NULL ; }@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev) (unsigned long long )pers->size (mddev, 0, 0) / 2); err = -EINVAL ; } -if (err ==0 && pers->sync_request && - (mddev->bitmap_info.file || mddev->bitmap_info.offset)) { +if (err ==0 && pers->sync_request) { +if (mddev->bitmap_info.offset ==0 ) + mddev->bitmap_id =ID_BITMAP_NONE ;err = md_bitmap_create(mddev, true); if (err) pr_warn("%s: failed to create bitmap (%d)\n",--
2.53.0
PATCH v2 4/5 md: skip ID_BITMAP_NONE when show available bitmap types
In reply to Su YueAs none_bitmap_ops is introduced, ID_BITMAP_NONE should be skipped while iterating md submodules, otherwise:
$ cat /sys/block/md0/md/bitmap_type [none] bitmap llbitmap [none]
--- drivers/md/md.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/md/md.c +2 -0
@@ -4269,6 +4269,8 @@ bitmap_type_show(struct mddev *mddev, char *page)xa_for_each (&md_submodule, i, head) {if (head->type !=MD_BITMAP )continue ; +if (head->id ==ID_BITMAP_NONE ) +continue ;if (mddev->bitmap_id == head->id) len +=sprintf (page + len,"[%s] " , head->name);
Signature · 1 line
2.53.0
Re: [PATCH v2 4/5] md: skip ID_BITMAP_NONE when show available bitmap types
Quoted 16 lines · Su Yue
As
none_bitmap_opsis introduced,ID_BITMAP_NONEshould be skipped while iterating md submodules, otherwise:$ cat /sys/block/md0/md/bitmap_type [none] bitmap llbitmap [none]Signed-off-by Su Yue <[email protected]>--- drivers/md/md.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/md.c +0
-0
drivers/md/md.c bitmap_type_show(struct mddev *mddev, char *page) · line 4276
Quoted 8 lines
drivers/md/md.c +2
-0@@ -4269,6 +4269,8 @@ bitmap_type_show(struct mddev *mddev, char *page)xa_for_each (&md_submodule, i, head) {if (head->type !=MD_BITMAP )continue ; +if (head->id ==ID_BITMAP_NONE ) +continue ;if (mddev->bitmap_id == head->id) len +=sprintf (page + len,"[%s] " , head->name);
If we indeed add ID_BITMAP_NONE as patch 3, would it be better to delete this piece of code instead?
if (mddev->bitmap_id == ID_BITMAP_NONE)
len += sprintf(page + len, "[none] ");
else
len += sprintf(page + len, "none ");Signature · 2 lines
Thanks, Nan
Re: [PATCH v2 4/5] md: skip ID_BITMAP_NONE when show available bitmap types
Quoted 17 lines · Li Nan
Quoted 16 lines · Su Yue
As
none_bitmap_opsis introduced,ID_BITMAP_NONEshould be skipped while iterating md submodules, otherwise:$ cat /sys/block/md0/md/bitmap_type [none] bitmap llbitmap [none] Signed-off-by: Su Yue <[email protected]> --- drivers/md/md.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/md.c +0
-0@@ -4269,6 +4269,8 @@ bitmap_type_show(struct mddev *mddev, char *page)xa_for_each (&md_submodule, i, head) {if (head->type !=MD_BITMAP )
Quoted 9 lines
drivers/md/md.c bitmap_type_show(struct mddev *mddev, char *page) · line 4276
Quoted 5 lines
drivers/md/md.c +2
-0continue ; +if (head->id ==ID_BITMAP_NONE ) +continue ;if (mddev->bitmap_id == head->id) len +=sprintf (page + len,"[%s] " , head->name);If we indeed add
ID_BITMAP_NONEas patch 3, would it be better to delete this piece of code instead?
I dont' think so. There is one case CONFIG_MD_BITMAP is not enabled which means register_md_submodule(&none_bitmap_ops.head) is not called but in mddev_init():
int mddev_init(struct mddev *mddev) |
{ |
int err = 0;
|
|
if (!IS_ENABLED(CONFIG_MD_BITMAP))
|
mddev->bitmap_id = ID_BITMAP_NONE;...
}
--
Su
Quoted 5 lines
if (mddev->bitmap_id == ID_BITMAP_NONE)
len += sprintf(page + len, "[none] ");
else
len += sprintf(page + len, "none ");PATCH v2 5/5 md/md-bitmap: remove member group from bitmap_operations
In reply to Su YueThe member is unused now so remove it.
--- drivers/md/md-bitmap.h | 2 -- drivers/md/md-llbitmap.c | 1 - 2 files changed, 3 deletions(-)
drivers/md/md-bitmap.h +0 -2
@@ -127,8 +127,6 @@ struct bitmap_operations {int (*register_groups)(struct mddev *mddev);void (*unregister_groups)(struct mddev *mddev); - -struct attribute_group *group; };/* the bitmap API */
drivers/md/md-llbitmap.c +0 -1
@@ -1609,7 +1609,6 @@ static struct bitmap_operations llbitmap_ops = { .register_groups = llbitmap_register_groups, .unregister_groups = llbitmap_unregister_groups, - .group = &md_llbitmap_group, };int md_llbitmap_init (void )
Signature · 1 line
2.53.0
Quoted 14 lines · Su Yue
Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.
The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.
v2:
Add a dummy bitmap operations per Kuai's suggestion.To Yu Kuai:
A NULL group can't used forinternal_bitmap_groupsince
the entries frominternal_bitmap_attrsshould be under bitmap.
So instead ofsysfs_create_groups(),sysfs_create_group()andsysfs_merge_group()are still needed /sigh.
Quoted 10 lines
Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy inlocation_storemd/md-bitmap: add an extra sysfs argument tomd_bitmap_createand
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skipID_BITMAP_NONEwhen show available bitmap types md/md-bitmap: remove member group frombitmap_operations
Missing part should be added in v3:
drivers/md/md.c +0 -0
@@ -6450,9 +6450,6 @@ static int start_dirty_degraded;
int md_bitmap_create(struct mddev *mddev, bool create_sysfs)
{
- if (mddev->bitmap_id == ID_BITMAP_NONE)
- return -EINVAL;
-
}
Quoted 7 lines
drivers/md/md-bitmap.c | 121 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 +- drivers/md/md-llbitmap.c | 13 ++++- drivers/md/md.c | 55 +++++++++--------- drivers/md/md.h | 2 + 5 files changed, 155 insertions(+), 39 deletions(-)
Hi,
Quoted 5 lines · Su Yue
Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.
The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.
Quoted 8 lines
v2:
Add a dummy bitmap operations per Kuai's suggestion.To Yu Kuai:
A NULL group can't used forinternal_bitmap_groupsince
the entries frominternal_bitmap_attrsshould be under bitmap.
So instead ofsysfs_create_groups(),sysfs_create_group()andsysfs_merge_group()are still needed /sigh.
Thanks for the set and sorry for the delay.
TBO, this set is still a bit complex than expected. It's true sysfs_create_groups() can't be used for the groups with same name. However, there is still another api sysfs_update_groups() that can be used in this case.
I just finish a local version and confirm following 3 commits should be enough:
1) factor out bitmap sysfs creation from mddev_set_bitmap_ops(), and create sysfs entries after bitmap is created.
2) split bitmap/location out as the common bitmap groups, and convert bitmap filed groups to struct attribute_group **groups;
3) add a separate bitmap_ops for none bitmap, location_store() can be similar to patch 3 in your set, just create/remove additional internal_bitmap_group.
Quoted 15 lines
Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy inlocation_storemd/md-bitmap: add an extra sysfs argument tomd_bitmap_createand
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skipID_BITMAP_NONEwhen show available bitmap types md/md-bitmap: remove member group frombitmap_operationsdrivers/md/md-bitmap.c | 121 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 +- drivers/md/md-llbitmap.c | 13 ++++- drivers/md/md.c | 55 +++++++++--------- drivers/md/md.h | 2 + 5 files changed, 155 insertions(+), 39 deletions(-)
Signature · 2 lines
Thansk, Kuai
Quoted 17 lines · Yu Kuai
Hi,
Quoted 5 lines · Su Yue
Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.
The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.
Quoted 8 lines
v2:
Add a dummy bitmap operations per Kuai's suggestion.To Yu Kuai:
A NULL group can't used forinternal_bitmap_groupsince
the entries frominternal_bitmap_attrsshould be under bitmap.
So instead ofsysfs_create_groups(),sysfs_create_group()andsysfs_merge_group()are still needed /sigh.
Quoted 6 lines
Thanks for the set and sorry for the delay.
TBO, this set is still a bit complex than expected. It's true
sysfs_create_groups()can't be used for the groups with same name. However, there is still another apisysfs_update_groups()that can be used in this case.
IIUC, sysfs_update_groups() can't be used because it first removes then re-add files see linux/fs/sysfs/group.c:create_files() for details.
Quoted 3 lines
I just finish a local version and confirm following 3 commits should be enough:
If you would like to send your own patchset. I am happy to give up my patchset then review yours :) The only thing I want to fix the issue. Thanks.
--
Su
Quoted 26 lines
1) factor out bitmap sysfs creation from
mddev_set_bitmap_ops(), and create sysfs entries after bitmap is created.
2) split bitmap/location out as the common bitmap groups, and convert bitmap filed groups to structattribute_group**groups;
3) add a separatebitmap_opsfor none bitmap,location_store()can be similar to patch 3 in your set, just create/remove additionalinternal_bitmap_group.Quoted 19 lines
Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy inlocation_store
md/md-bitmap: add an extra sysfs argument tomd_bitmap_createand
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skipID_BITMAP_NONEwhen show available bitmap types md/md-bitmap: remove member group frombitmap_operationsdrivers/md/md-bitmap.c | 121 ++++++++++++++++++++++++++++++++++++--- drivers/md/md-bitmap.h | 3 +- drivers/md/md-llbitmap.c | 13 ++++- drivers/md/md.c | 55 +++++++++--------- drivers/md/md.h | 2 + 5 files changed, 155 insertions(+), 39 deletions(-)