Skip to contentSkip to conversation

md: bitmap grow fixes

PATCH In review Su Yue 5 participants 22 messages Apr 7 – Apr 21 MD RAID

lore
Share & tools
lore

Su Yue

author
#0 lore raw

v2 cover md: bitmap grow fixes

Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.

The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.

v2:
Add a dummy bitmap operations per Kuai's suggestion.

To Yu Kuai:
A NULL group can't used for internal_bitmap_group since
the entries from internal_bitmap_attrs should be under bitmap.
So instead of sysfs_create_groups(), sysfs_create_group() and sysfs_merge_group() are still needed /sigh.

Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy in location_store md/md-bitmap: add an extra sysfs argument to md_bitmap_create and
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skip ID_BITMAP_NONE when show available bitmap types md/md-bitmap: remove member group from bitmap_operations

 drivers/md/md-bitmap.c   | 121 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +-
 drivers/md/md-llbitmap.c |  13 ++++-
 drivers/md/md.c          |  55 +++++++++---------
 drivers/md/md.h          |   2 +
 5 files changed, 155 insertions(+), 39 deletions(-)
Signature · 1 line
2.53.0
author
#1 lore raw

PATCH v2 1/5 md/md-bitmap: call md_bitmap_create,destroy in location_store

If bitmap/location is present, mdadm will call update_array_info() while growing bitmap from none to internal via location_store(). md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger kernel NULL pointer dereference.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
 drivers/md/md-bitmap.c | 11 ++++++++---
 drivers/md/md.c        |  4 ++--
 drivers/md/md.h        |  2 ++
 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +8 -3
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
 			goto out;
 		}
 
-		bitmap_destroy(mddev);
+		md_bitmap_destroy(mddev);
 		mddev->bitmap_info.offset = 0;
 		if (mddev->bitmap_info.file) {
 			struct file *f = mddev->bitmap_info.file;
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
 				goto out;
 			}
 
+			/*
+			 * lockless bitmap shoudle have set bitmap_id
+			 * using bitmap_type, so always ID_BITMAP.
+			 */
+			mddev->bitmap_id = ID_BITMAP;
 			mddev->bitmap_info.offset = offset;
-			rv = bitmap_create(mddev);
+			rv = md_bitmap_create(mddev);
 			if (rv)
 				goto out;
 
 			rv = bitmap_load(mddev);
 			if (rv) {
 				mddev->bitmap_info.offset = 0;
-				bitmap_destroy(mddev);
+				md_bitmap_destroy(mddev);
 				goto out;
 			}
 		}
drivers/md/md.c +2 -2
index 3ce6f9e9d38e..8b1ecc370ad6 100644
@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)
 
 static int start_dirty_degraded;
 
-static int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev)
 {
 	if (mddev->bitmap_id == ID_BITMAP_NONE)
 		return -EINVAL;
@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev *mddev)
 	return mddev->bitmap_ops->create(mddev);
 }
 
-static void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev)
 {
 	if (!md_bitmap_registered(mddev))
 		return;
drivers/md/md.h +2 -0
index ac84289664cd..ed69244af00d 100644
@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head *msh);
+int md_bitmap_create(struct mddev *mddev);
+void md_bitmap_destroy(struct mddev *mddev);
 
 extern struct md_thread *md_register_thread(
 	void (*run)(struct md_thread *thread),
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store

Quoted 33 lines · Su Yue

If bitmap/location is present, mdadm will call update_array_info() while growing bitmap from none to internal via location_store(). md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger kernel NULL pointer dereference.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
  drivers/md/md-bitmap.c | 11 ++++++++---
  drivers/md/md.c        |  4 ++--
  drivers/md/md.h        |  2 ++
  3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +3 -1
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
  			goto out;
  		}
 
-		bitmap_destroy(mddev);
+		md_bitmap_destroy(mddev);
  		mddev->bitmap_info.offset = 0;
  		if (mddev->bitmap_info.file) {
  			struct file *f = mddev->bitmap_info.file;
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
  				goto out;
  			}
 
+			/*
+			 * lockless bitmap shoudle have set bitmap_id

drivers/md/md-bitmap.c location_store(struct mddev *mddev, const char *buf, size_t len) · line 2666

Quoted 10 lines
drivers/md/md-bitmap.c +4 -1
+			 * using bitmap_type, so always ID_BITMAP.
+			 */
+			mddev->bitmap_id = ID_BITMAP;
  			mddev->bitmap_info.offset = offset;
-			rv = bitmap_create(mddev);
+			rv = md_bitmap_create(mddev);
  			if (rv)
  				goto out;
 
  			rv = bitmap_load(mddev);

mddev->bitmap_ops->load() should also be used here.

Signature · 2 lines
Thanks,
Nan
#3 lore raw

Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store

Quoted 36 lines · Li Nan
Quoted 35 lines · Su Yue

If bitmap/location is present, mdadm will call update_array_info() while growing bitmap from none to internal via location_store(). md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger kernel NULL pointer dereference.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until

creating bitmap")

---
  drivers/md/md-bitmap.c | 11 ++++++++---
  drivers/md/md.c        |  4 ++--
  drivers/md/md.h        |  2 ++
  3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +5 -0
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const 
 char *buf, size_t len)
  			goto out;
  		}
  -		bitmap_destroy(mddev);
+		md_bitmap_destroy(mddev);
  		mddev->bitmap_info.offset = 0;
  		if (mddev->bitmap_info.file) {
  			struct file *f = mddev->bitmap_info.file;
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, 
 const char *buf, size_t len)
  				goto out;
  			}
  +			/*
+			 * lockless bitmap shoudle have set bitmap_id
+			 * using bitmap_type, so always ID_BITMAP.
+			 */
+			mddev->bitmap_id = ID_BITMAP;
Quoted 8 lines

drivers/md/md-bitmap.c location_store(struct mddev *mddev, const char *buf, size_t len) · line 2666

Quoted 6 lines
drivers/md/md-bitmap.c +1 -1
  			mddev->bitmap_info.offset = offset;
-			rv = bitmap_create(mddev);
+			rv = md_bitmap_create(mddev);
  			if (rv)
  				goto out;
    			rv = bitmap_load(mddev);

mddev->bitmap_ops->load() should also be used here.

/NOD.
location_store() is only used for ID_BITMAP_NONE and ID_BITMAP, so mddev->bitmap_ops->load() is always bitmap_load().
But for code consistency, mddev->bitmap_ops->load() is better, will fix it.

--
Su

#4 lore raw

Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store

In reply to Su Yue
Quoted 6 lines · Su Yue

If bitmap/location is present, mdadm will call update_array_info() while growing bitmap from none to internal via location_store(). md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger kernel NULL pointer dereference.

Hi Su Yue

How can bitmap/location be present when bitmap is none? Could you provide the test commands that reproduce this problem?

mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --bitmap=none (There is not bitmap/location, because bitmap directory is not created) mdadm /dev/md0 --grow --bitmap=internal
Grow.c md_set_array_info runs
451 array.state |= (1 << MD_SB_BITMAP_PRESENT);
452 rv = md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv = md_bitmap_create(mddev);
8126 if (!rv)
8127 rv = mddev->bitmap_ops->load(mddev);

Best Regards
Xiao

Quoted 83 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
 drivers/md/md-bitmap.c | 11 ++++++++---
 drivers/md/md.c        |  4 ++--
 drivers/md/md.h        |  2 ++
 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +8 -3
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
                        goto out;
                }
 
-               bitmap_destroy(mddev);
+               md_bitmap_destroy(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {
                        struct file *f = mddev->bitmap_info.file;
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
                                goto out;
                        }
 
+                       /*
+                        * lockless bitmap shoudle have set bitmap_id
+                        * using bitmap_type, so always ID_BITMAP.
+                        */
+                       mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = bitmap_create(mddev);
+                       rv = md_bitmap_create(mddev);
                        if (rv)
                                goto out;
 
                        rv = bitmap_load(mddev);
                        if (rv) {
                                mddev->bitmap_info.offset = 0;
-                               bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev);
                                goto out;
                        }
                }
drivers/md/md.c +2 -2
index 3ce6f9e9d38e..8b1ecc370ad6 100644
@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct timer_list *t)
 
 static int start_dirty_degraded;
 
-static int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev)
 {
        if (mddev->bitmap_id == ID_BITMAP_NONE)
                return -EINVAL;
@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev *mddev)
        return mddev->bitmap_ops->create(mddev);
 }
 
-static void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev)
 {
        if (!md_bitmap_registered(mddev))
                return;
drivers/md/md.h +2 -1
index ac84289664cd..ed69244af00d 100644
@@ -895,6 +895,8 @@ static inline void safe_put_page(struct page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head *msh);
+int md_bitmap_create(struct mddev *mddev);
+void md_bitmap_destroy(struct mddev *mddev);
 
 extern struct md_thread *md_register_thread(
        void (*run)(struct md_thread *thread),
--

2.53.0

#5 lore raw

Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store

Quoted 9 lines · Xiao Ni
Quoted 6 lines · Su Yue

If bitmap/location is present, mdadm will call update_array_info() while growing bitmap from none to internal via location_store(). md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger kernel NULL pointer dereference.

Quoted 5 lines

Hi Su Yue

How can bitmap/location be present when bitmap is none? Could you provide the test commands that reproduce this problem?

Sorry for the misleading commit message. It can only be reproduced patch 3 is appiled. I adjusted the sequence of this patch for easy review because md_bitmap_create,destroy are touched in patch1,2 and 3. Also if put the patch after 3rd patch,
it will break ability to bisect.

# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal 
 --raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc /dev/vdd
# mdadm --grow /dev/md0 --bitmap=none
# mdadm --grow /dev/md0 --bitmap=internal # step 3
# mdadm --grow /dev/md0 --bitmap=none # step 4
[1]    2325 killed     mdadm --grow /dev/md0 --bitmap=none

When step 3 is called,
md_bitmap_destroy() is called in update_array_info() to set NULL mddev->bitmap_ops then in step 4 kernel Oops is triggered.

I am willing to amend commit message or move it after patch 3 if you would like.

--
Su

Quoted 106 lines

mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --bitmap=none (There is not bitmap/location, because bitmap directory is not created) mdadm /dev/md0 --grow --bitmap=internal
Grow.c md_set_array_info runs
451 array.state |= (1 << MD_SB_BITMAP_PRESENT);
452 rv = md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv = md_bitmap_create(mddev);
8126 if (!rv)
8127 rv = mddev->bitmap_ops->load(mddev);

Best Regards
Xiao

Quoted 91 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
 drivers/md/md-bitmap.c | 11 ++++++++---
 drivers/md/md.c        |  4 ++--
 drivers/md/md.h        |  2 ++
 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +1 -1
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const 
 char *buf, size_t len)
                        goto out;
                }
 
-               bitmap_destroy(mddev);
+               md_bitmap_destroy(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {

struct file *f =
mddev->bitmap_info.file;

drivers/md/md-bitmap.c +2 -0
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev, 
 const char *buf, size_t len)
                                goto out;
                        }
 
+                       /*
+                        * lockless bitmap shoudle have set 

bitmap_id
+ * using bitmap_type, so always
ID_BITMAP.

drivers/md/md-bitmap.c +4 -2
+                        */
+                       mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = bitmap_create(mddev);
+                       rv = md_bitmap_create(mddev);
                        if (rv)
                                goto out;
 
                        rv = bitmap_load(mddev);
                        if (rv) {
                                mddev->bitmap_info.offset = 0;
-                               bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev);
                                goto out;
                        }
                }
drivers/md/md.c +1 -1
index 3ce6f9e9d38e..8b1ecc370ad6 100644
@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct 
 timer_list *t)
 
 static int start_dirty_degraded;
 
-static int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev)
 {
        if (mddev->bitmap_id == ID_BITMAP_NONE)
                return -EINVAL;
drivers/md/md.c +1 -1
@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev 
 *mddev)
        return mddev->bitmap_ops->create(mddev);
 }
 
-static void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev)
 {
        if (!md_bitmap_registered(mddev))
                return;
drivers/md/md.h +2 -0
index ac84289664cd..ed69244af00d 100644
@@ -895,6 +895,8 @@ static inline void safe_put_page(struct 
 page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head *msh);
+int md_bitmap_create(struct mddev *mddev);
+void md_bitmap_destroy(struct mddev *mddev);
 
 extern struct md_thread *md_register_thread(

void (*run)(struct md_thread *thread),
--
2.53.0

#6 lore raw

Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store

Quoted 36 lines · Su Yue
Quoted 12 lines · Xiao Ni
Quoted 9 lines · Su Yue

If bitmap/location is present, mdadm will call update_array_info()
while growing bitmap from none to internal via location_store().
md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger
kernel NULL pointer dereference.

Quoted 6 lines

Hi Su Yue

How can bitmap/location be present when bitmap is none? Could you
provide the test commands that reproduce this problem?

Sorry for the misleading commit message. It can only be reproduced patch 3 is appiled.
I adjusted the sequence of this patch for easy review because md_bitmap_create,destroy
are touched in patch1,2 and 3. Also if put the patch after 3rd patch,
it will break ability to bisect.

# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal
  --raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc /dev/vdd
# mdadm --grow /dev/md0 --bitmap=none
# mdadm --grow /dev/md0 --bitmap=internal # step 3
# mdadm --grow /dev/md0 --bitmap=none # step 4
[1]    2325 killed     mdadm --grow /dev/md0 --bitmap=none
Quoted 8 lines

When step 3 is called,
md_bitmap_destroy() is called in update_array_info() to set NULL mddev->bitmap_ops
then in step 4 kernel Oops is triggered.

I am willing to amend commit message or move it after patch 3 if you would like.

Hi Su

Thanks for the detail explanation. After reading patch3, I totoally understand. The sequence is good to me. And yes, it's better to explain that this is needed after patch3.

Best Regards
Xiao

Quoted 113 lines

--
Su

Quoted 108 lines

mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --bitmap=none (There
is not bitmap/location, because bitmap directory is not created) mdadm /dev/md0 --grow --bitmap=internal
Grow.c md_set_array_info runs
451 array.state |= (1 << MD_SB_BITMAP_PRESENT);
452 rv = md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv = md_bitmap_create(mddev);
8126 if (!rv)
8127 rv = mddev->bitmap_ops->load(mddev);

Best Regards
Xiao

Quoted 92 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of

bitmap_ops until creating bitmap")

---
 drivers/md/md-bitmap.c | 11 ++++++++---
 drivers/md/md.c        |  4 ++--
 drivers/md/md.h        |  2 ++
 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +1 -1
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const
 char *buf, size_t len)
                        goto out;
                }
 
-               bitmap_destroy(mddev);
+               md_bitmap_destroy(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {

struct file *f =
mddev->bitmap_info.file;

drivers/md/md-bitmap.c +2 -0
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev,
 const char *buf, size_t len)
                                goto out;
                        }
 
+                       /*
+                        * lockless bitmap shoudle have set

bitmap_id
+ * using bitmap_type, so always
ID_BITMAP.

drivers/md/md-bitmap.c +4 -2
+                        */
+                       mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = bitmap_create(mddev);
+                       rv = md_bitmap_create(mddev);
                        if (rv)
                                goto out;
 
                        rv = bitmap_load(mddev);
                        if (rv) {
                                mddev->bitmap_info.offset = 0;
-                               bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev);
                                goto out;
                        }
                }
drivers/md/md.c +1 -1
index 3ce6f9e9d38e..8b1ecc370ad6 100644
@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct
 timer_list *t)
 
 static int start_dirty_degraded;
 
-static int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev)
 {
        if (mddev->bitmap_id == ID_BITMAP_NONE)
                return -EINVAL;
drivers/md/md.c +1 -1
@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev
 *mddev)
        return mddev->bitmap_ops->create(mddev);
 }
 
-static void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev)
 {
        if (!md_bitmap_registered(mddev))
                return;
drivers/md/md.h +2 -0
index ac84289664cd..ed69244af00d 100644
@@ -895,6 +895,8 @@ static inline void safe_put_page(struct
 page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head *msh);
+int md_bitmap_create(struct mddev *mddev);
+void md_bitmap_destroy(struct mddev *mddev);
 
 extern struct md_thread *md_register_thread(

void (*run)(struct md_thread *thread),
--
2.53.0

#7 lore raw

Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store

Quoted 44 lines · Xiao Ni
Quoted 43 lines · Su Yue
Quoted 12 lines · Xiao Ni
Quoted 9 lines · Su Yue

If bitmap/location is present, mdadm will call update_array_info()
while growing bitmap from none to internal via location_store().
md_bitmap_create() is needed to set mddev->bitmap_ops otherwise mddev->bitmap_ops->get_stats() in update_array_info() will trigger
kernel NULL pointer dereference.

Quoted 6 lines

Hi Su Yue

How can bitmap/location be present when bitmap is none? Could you
provide the test commands that reproduce this problem?

Sorry for the misleading commit message. It can only be reproduced patch 3 is appiled.
I adjusted the sequence of this patch for easy review because md_bitmap_create,destroy
are touched in patch1,2 and 3. Also if put the patch after 3rd patch,
it will break ability to bisect.

# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal
  --raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc 
  /dev/vdd
# mdadm --grow /dev/md0 --bitmap=none
# mdadm --grow /dev/md0 --bitmap=internal # step 3
# mdadm --grow /dev/md0 --bitmap=none # step 4
[1]    2325 killed     mdadm --grow /dev/md0 --bitmap=none

When step 3 is called,
md_bitmap_destroy() is called in update_array_info() to set NULL mddev->bitmap_ops
then in step 4 kernel Oops is triggered.

Quoted 9 lines
Quoted 2 lines

I am willing to amend commit message or move it after patch 3 if you would like.

Hi Su

Thanks for the detail explanation. After reading patch3, I totoally understand. The sequence is good to me. And yes, it's better to explain that this is needed after patch3.

Sure. I will do it in next version.

--
Su

Quoted 121 lines

Best Regards
Xiao

Quoted 118 lines

--
Su

Quoted 113 lines

mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --bitmap=none
(There
is not bitmap/location, because bitmap directory is not created) mdadm /dev/md0 --grow --bitmap=internal
Grow.c md_set_array_info runs
451 array.state |= (1 << MD_SB_BITMAP_PRESENT);
452 rv = md_set_array_info(fd, &array);
In kernel space, it runs
8125 rv = md_bitmap_create(mddev);
8126 if (!rv)
8127 rv = mddev->bitmap_ops->load(mddev);

Best Regards
Xiao

Quoted 96 lines
Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of

bitmap_ops until creating bitmap")

---
 drivers/md/md-bitmap.c | 11 ++++++++---
 drivers/md/md.c        |  4 ++--
 drivers/md/md.h        |  2 ++
 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/md/md-bitmap.c +0 -0
index 83378c033c72..2f24aae05552 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, 

const
char *buf, size_t len)

                        goto out;
                }
drivers/md/md-bitmap.c +3 -1
-               bitmap_destroy(mddev);
+               md_bitmap_destroy(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {
                        struct file *f =
                        mddev->bitmap_info.file;
@@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev,
 const char *buf, size_t len)
                                goto out;
                        }
 
+                       /*
+                        * lockless bitmap shoudle have set

bitmap_id
+ * using bitmap_type, so always
ID_BITMAP.

drivers/md/md-bitmap.c +4 -2
+                        */
+                       mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = bitmap_create(mddev);
+                       rv = md_bitmap_create(mddev);
                        if (rv)
                                goto out;
 
                        rv = bitmap_load(mddev);
                        if (rv) {
                                mddev->bitmap_info.offset = 
                                0;
-                               bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev);
                                goto out;
                        }
                }
drivers/md/md.c +1 -1
index 3ce6f9e9d38e..8b1ecc370ad6 100644
@@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct
 timer_list *t)
 
 static int start_dirty_degraded;
 
-static int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev)
 {
        if (mddev->bitmap_id == ID_BITMAP_NONE)
                return -EINVAL;
drivers/md/md.c +0 -0
@@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct 

mddev
*mddev)

        return mddev->bitmap_ops->create(mddev);

}

drivers/md/md.c +1 -1
-static void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev)
 {
        if (!md_bitmap_registered(mddev))
                return;
drivers/md/md.h +2 -0
index ac84289664cd..ed69244af00d 100644
@@ -895,6 +895,8 @@ static inline void safe_put_page(struct
 page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head 
 *msh);
+int md_bitmap_create(struct mddev *mddev);
+void md_bitmap_destroy(struct mddev *mddev);
 

extern struct md_thread *md_register_thread(
void (*run)(struct md_thread *thread),
--
2.53.0

author
#8 lore raw

PATCH v2 2/5 md/md-bitmap: add an extra sysfs argument to md_bitmap_create and destroy

In reply to Su Yue

For further use, no functional change.

---
 drivers/md/md-bitmap.c |  6 +++---
 drivers/md/md.c        | 36 ++++++++++++++++++------------------
 drivers/md/md.h        |  4 ++--
 3 files changed, 23 insertions(+), 23 deletions(-)
drivers/md/md-bitmap.c +3 -3
index 2f24aae05552..ac06c9647bf0 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
 			goto out;
 		}
 
-		md_bitmap_destroy(mddev);
+		md_bitmap_destroy(mddev, true);
 		mddev->bitmap_info.offset = 0;
 		if (mddev->bitmap_info.file) {
 			struct file *f = mddev->bitmap_info.file;
@@ -2659,14 +2659,14 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
 			 */
 			mddev->bitmap_id = ID_BITMAP;
 			mddev->bitmap_info.offset = offset;
-			rv = md_bitmap_create(mddev);
+			rv = md_bitmap_create(mddev, true);
 			if (rv)
 				goto out;
 
 			rv = bitmap_load(mddev);
 			if (rv) {
 				mddev->bitmap_info.offset = 0;
-				md_bitmap_destroy(mddev);
+				md_bitmap_destroy(mddev, true);
 				goto out;
 			}
 		}
drivers/md/md.c +18 -18
index 8b1ecc370ad6..d3c8f77b4fe3 100644
@@ -678,7 +678,7 @@ static void active_io_release(struct percpu_ref *ref)
 
 static void no_op(struct percpu_ref *r) {}
 
-static bool mddev_set_bitmap_ops(struct mddev *mddev)
+static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
 {
 	struct bitmap_operations *old = mddev->bitmap_ops;
 	struct md_submodule_head *head;
@@ -703,7 +703,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev)
 	mddev->bitmap_ops = (void *)head;
 	xa_unlock(&md_submodule);
 
-	if (!mddev_is_dm(mddev) && mddev->bitmap_ops->group) {
+	if (create_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops->group) {
 		if (sysfs_create_group(&mddev->kobj, mddev->bitmap_ops->group))
 			pr_warn("md: cannot register extra bitmap attributes for %s\n",
 				mdname(mddev));
@@ -721,9 +721,9 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev)
 	return false;
 }
 
-static void mddev_clear_bitmap_ops(struct mddev *mddev)
+static void mddev_clear_bitmap_ops(struct mddev *mddev, bool remove_sysfs)
 {
-	if (!mddev_is_dm(mddev) && mddev->bitmap_ops &&
+	if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops &&
 	    mddev->bitmap_ops->group)
 		sysfs_remove_group(&mddev->kobj, mddev->bitmap_ops->group);
 
@@ -6447,24 +6447,24 @@ static void md_safemode_timeout(struct timer_list *t)
 
 static int start_dirty_degraded;
 
-int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev, bool create_sysfs)
 {
 	if (mddev->bitmap_id == ID_BITMAP_NONE)
 		return -EINVAL;
 
-	if (!mddev_set_bitmap_ops(mddev))
+	if (!mddev_set_bitmap_ops(mddev, create_sysfs))
 		return -ENOENT;
 
 	return mddev->bitmap_ops->create(mddev);
 }
 
-void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev, bool remove_sysfs)
 {
 	if (!md_bitmap_registered(mddev))
 		return;
 
 	mddev->bitmap_ops->destroy(mddev);
-	mddev_clear_bitmap_ops(mddev);
+	mddev_clear_bitmap_ops(mddev, remove_sysfs);
 }
 
 int md_run(struct mddev *mddev)
@@ -6612,7 +6612,7 @@ int md_run(struct mddev *mddev)
 	}
 	if (err == 0 && pers->sync_request &&
 	    (mddev->bitmap_info.file || mddev->bitmap_info.offset)) {
-		err = md_bitmap_create(mddev);
+		err = md_bitmap_create(mddev, true);
 		if (err)
 			pr_warn("%s: failed to create bitmap (%d)\n",
 				mdname(mddev), err);
@@ -6685,7 +6685,7 @@ int md_run(struct mddev *mddev)
 		pers->free(mddev, mddev->private);
 	mddev->private = NULL;
 	put_pers(pers);
-	md_bitmap_destroy(mddev);
+	md_bitmap_destroy(mddev, true);
 	return err;
 }
 EXPORT_SYMBOL_GPL(md_run);
@@ -6702,7 +6702,7 @@ int do_md_run(struct mddev *mddev)
 	if (md_bitmap_registered(mddev)) {
 		err = mddev->bitmap_ops->load(mddev);
 		if (err) {
-			md_bitmap_destroy(mddev);
+			md_bitmap_destroy(mddev, true);
 			goto out;
 		}
 	}
@@ -6903,7 +6903,7 @@ static void __md_stop(struct mddev *mddev)
 {
 	struct md_personality *pers = mddev->pers;
 
-	md_bitmap_destroy(mddev);
+	md_bitmap_destroy(mddev, true);
 	mddev_detach(mddev);
 	spin_lock(&mddev->lock);
 	mddev->pers = NULL;
@@ -7680,16 +7680,16 @@ static int set_bitmap_file(struct mddev *mddev, int fd)
 	err = 0;
 	if (mddev->pers) {
 		if (fd >= 0) {
-			err = md_bitmap_create(mddev);
+			err = md_bitmap_create(mddev, true);
 			if (!err)
 				err = mddev->bitmap_ops->load(mddev);
 
 			if (err) {
-				md_bitmap_destroy(mddev);
+				md_bitmap_destroy(mddev, true);
 				fd = -1;
 			}
 		} else if (fd < 0) {
-			md_bitmap_destroy(mddev);
+			md_bitmap_destroy(mddev, true);
 		}
 	}
 
@@ -7996,12 +7996,12 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
 				mddev->bitmap_info.default_offset;
 			mddev->bitmap_info.space =
 				mddev->bitmap_info.default_space;
-			rv = md_bitmap_create(mddev);
+			rv = md_bitmap_create(mddev, true);
 			if (!rv)
 				rv = mddev->bitmap_ops->load(mddev);
 
 			if (rv)
-				md_bitmap_destroy(mddev);
+				md_bitmap_destroy(mddev, true);
 		} else {
 			struct md_bitmap_stats stats;
 
@@ -8027,7 +8027,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
 				put_cluster_ops(mddev);
 				mddev->safemode_delay = DEFAULT_SAFEMODE_DELAY;
 			}
-			md_bitmap_destroy(mddev);
+			md_bitmap_destroy(mddev, true);
 			mddev->bitmap_info.offset = 0;
 		}
 	}
drivers/md/md.h +2 -2
index ed69244af00d..4aaba3d7015c 100644
@@ -895,8 +895,8 @@ static inline void safe_put_page(struct page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head *msh);
-int md_bitmap_create(struct mddev *mddev);
-void md_bitmap_destroy(struct mddev *mddev);
+int md_bitmap_create(struct mddev *mddev, bool create_sysfs);
+void md_bitmap_destroy(struct mddev *mddev, bool remove_sysfs);
 
 extern struct md_thread *md_register_thread(
 	void (*run)(struct md_thread *thread),
Signature · 1 line
2.53.0
Reviewed-by on v2 2/5
#9 lore raw

Re: [PATCH v2 2/5] md/md-bitmap: add an extra sysfs argument to md_bitmap_create and destroy

Quoted 199 lines · Su Yue

For further use, no functional change.

---
 drivers/md/md-bitmap.c |  6 +++---
 drivers/md/md.c        | 36 ++++++++++++++++++------------------
 drivers/md/md.h        |  4 ++--
 3 files changed, 23 insertions(+), 23 deletions(-)
drivers/md/md-bitmap.c +3 -3
index 2f24aae05552..ac06c9647bf0 100644
@@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
                        goto out;
                }
 
-               md_bitmap_destroy(mddev);
+               md_bitmap_destroy(mddev, true);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {
                        struct file *f = mddev->bitmap_info.file;
@@ -2659,14 +2659,14 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
                         */
                        mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = md_bitmap_create(mddev);
+                       rv = md_bitmap_create(mddev, true);
                        if (rv)
                                goto out;
 
                        rv = bitmap_load(mddev);
                        if (rv) {
                                mddev->bitmap_info.offset = 0;
-                               md_bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev, true);
                                goto out;
                        }
                }
drivers/md/md.c +18 -18
index 8b1ecc370ad6..d3c8f77b4fe3 100644
@@ -678,7 +678,7 @@ static void active_io_release(struct percpu_ref *ref)
 
 static void no_op(struct percpu_ref *r) {}
 
-static bool mddev_set_bitmap_ops(struct mddev *mddev)
+static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
 {
        struct bitmap_operations *old = mddev->bitmap_ops;
        struct md_submodule_head *head;
@@ -703,7 +703,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev)
        mddev->bitmap_ops = (void *)head;
        xa_unlock(&md_submodule);
 
-       if (!mddev_is_dm(mddev) && mddev->bitmap_ops->group) {
+       if (create_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops->group) {
                if (sysfs_create_group(&mddev->kobj, mddev->bitmap_ops->group))
                        pr_warn("md: cannot register extra bitmap attributes for %s\n",
                                mdname(mddev));
@@ -721,9 +721,9 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev)
        return false;
 }
 
-static void mddev_clear_bitmap_ops(struct mddev *mddev)
+static void mddev_clear_bitmap_ops(struct mddev *mddev, bool remove_sysfs)
 {
-       if (!mddev_is_dm(mddev) && mddev->bitmap_ops &&
+       if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops &&
            mddev->bitmap_ops->group)
                sysfs_remove_group(&mddev->kobj, mddev->bitmap_ops->group);
 
@@ -6447,24 +6447,24 @@ static void md_safemode_timeout(struct timer_list *t)
 
 static int start_dirty_degraded;
 
-int md_bitmap_create(struct mddev *mddev)
+int md_bitmap_create(struct mddev *mddev, bool create_sysfs)
 {
        if (mddev->bitmap_id == ID_BITMAP_NONE)
                return -EINVAL;
 
-       if (!mddev_set_bitmap_ops(mddev))
+       if (!mddev_set_bitmap_ops(mddev, create_sysfs))
                return -ENOENT;
 
        return mddev->bitmap_ops->create(mddev);
 }
 
-void md_bitmap_destroy(struct mddev *mddev)
+void md_bitmap_destroy(struct mddev *mddev, bool remove_sysfs)
 {
        if (!md_bitmap_registered(mddev))
                return;
 
        mddev->bitmap_ops->destroy(mddev);
-       mddev_clear_bitmap_ops(mddev);
+       mddev_clear_bitmap_ops(mddev, remove_sysfs);
 }
 
 int md_run(struct mddev *mddev)
@@ -6612,7 +6612,7 @@ int md_run(struct mddev *mddev)
        }
        if (err == 0 && pers->sync_request &&
            (mddev->bitmap_info.file || mddev->bitmap_info.offset)) {
-               err = md_bitmap_create(mddev);
+               err = md_bitmap_create(mddev, true);
                if (err)
                        pr_warn("%s: failed to create bitmap (%d)\n",
                                mdname(mddev), err);
@@ -6685,7 +6685,7 @@ int md_run(struct mddev *mddev)
                pers->free(mddev, mddev->private);
        mddev->private = NULL;
        put_pers(pers);
-       md_bitmap_destroy(mddev);
+       md_bitmap_destroy(mddev, true);
        return err;
 }
 EXPORT_SYMBOL_GPL(md_run);
@@ -6702,7 +6702,7 @@ int do_md_run(struct mddev *mddev)
        if (md_bitmap_registered(mddev)) {
                err = mddev->bitmap_ops->load(mddev);
                if (err) {
-                       md_bitmap_destroy(mddev);
+                       md_bitmap_destroy(mddev, true);
                        goto out;
                }
        }
@@ -6903,7 +6903,7 @@ static void __md_stop(struct mddev *mddev)
 {
        struct md_personality *pers = mddev->pers;
 
-       md_bitmap_destroy(mddev);
+       md_bitmap_destroy(mddev, true);
        mddev_detach(mddev);
        spin_lock(&mddev->lock);
        mddev->pers = NULL;
@@ -7680,16 +7680,16 @@ static int set_bitmap_file(struct mddev *mddev, int fd)
        err = 0;
        if (mddev->pers) {
                if (fd >= 0) {
-                       err = md_bitmap_create(mddev);
+                       err = md_bitmap_create(mddev, true);
                        if (!err)
                                err = mddev->bitmap_ops->load(mddev);
 
                        if (err) {
-                               md_bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev, true);
                                fd = -1;
                        }
                } else if (fd < 0) {
-                       md_bitmap_destroy(mddev);
+                       md_bitmap_destroy(mddev, true);
                }
        }
 
@@ -7996,12 +7996,12 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
                                mddev->bitmap_info.default_offset;
                        mddev->bitmap_info.space =
                                mddev->bitmap_info.default_space;
-                       rv = md_bitmap_create(mddev);
+                       rv = md_bitmap_create(mddev, true);
                        if (!rv)
                                rv = mddev->bitmap_ops->load(mddev);
 
                        if (rv)
-                               md_bitmap_destroy(mddev);
+                               md_bitmap_destroy(mddev, true);
                } else {
                        struct md_bitmap_stats stats;
 
@@ -8027,7 +8027,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
                                put_cluster_ops(mddev);
                                mddev->safemode_delay = DEFAULT_SAFEMODE_DELAY;
                        }
-                       md_bitmap_destroy(mddev);
+                       md_bitmap_destroy(mddev, true);
                        mddev->bitmap_info.offset = 0;
                }
        }
drivers/md/md.h +2 -3
index ed69244af00d..4aaba3d7015c 100644
@@ -895,8 +895,8 @@ static inline void safe_put_page(struct page *p)
 
 int register_md_submodule(struct md_submodule_head *msh);
 void unregister_md_submodule(struct md_submodule_head *msh);
-int md_bitmap_create(struct mddev *mddev);
-void md_bitmap_destroy(struct mddev *mddev);
+int md_bitmap_create(struct mddev *mddev, bool create_sysfs);
+void md_bitmap_destroy(struct mddev *mddev, bool remove_sysfs);
 
 extern struct md_thread *md_register_thread(
        void (*run)(struct md_thread *thread),
--
Quoted 2 lines

2.53.0

Looks good to me.

author
#10 lore raw

PATCH v2 3/5 md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset

In reply to Su Yue

Before commit fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap") if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.

After the commit, if bitmap is none, bitmap/location doesn't exist anymore. It breaks 'grow' behavior of a md array of madam with MD_FEATURE_BITMAP_OFFSET. Take level=mirror and metadata=1.2 as an example:

$ mdadm  --create  /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \
  --metadata=1.2 /dev/vdd /dev/vde
$ mdadm --grow /dev/md0 --bitmap=internal
$ cat /sys/block/md0/md/bitmap/location
Before:+8
After: +2

While growing bitmap from none to internal, clustered and llbitmap, mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first. 1)If bitmap/location exists, it sets bitmap/location after getinfo_super(). 2)If bitmap/location doesn't exist, mdadm just calls md_set_array_info() then mddev->bitmap_info.default_offset will be used.
Situation can be worse if growing none to clustered, bitmap offset of the node calling madm --grow will be changed but the other node are reading bitmap sb from the old location.

Here introducing a dummy bitmap_operations for ID_BITMAP_NONE to restore sysfs file bitmap/location for ID_BITMAP_NONE and ID_BITMAP.
location_store() now calls md_bitmap_(create|destroy) with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
 drivers/md/md-bitmap.c   | 116 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +
 drivers/md/md-llbitmap.c |  12 ++++
 drivers/md/md.c          |  16 +++---
 4 files changed, 130 insertions(+), 17 deletions(-)
drivers/md/md-bitmap.c +107 -9
index ac06c9647bf0..a8a176428c61 100644
@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush)
 	       bitmap->storage.filemap != NULL;
 }
 
+static bool dummy_bitmap_enabled(void *data, bool flush)
+{
+	return false;
+}
+
 /*
  * check a page and, if necessary, allocate it (or hijack it if the alloc fails)
  *
@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)
 	return 0;
 }
 
+static int dummy_bitmap_create(struct mddev *mddev)
+{
+	return 0;
+}
+
 static int bitmap_load(struct mddev *mddev)
 {
 	int err = 0;
@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char *page)
 	return len;
 }
 
+static int create_internal_group(struct mddev *mddev);
+static void remove_internal_group(struct mddev *mddev);
+
 static ssize_t
 location_store(struct mddev *mddev, const char *buf, size_t len)
 {
@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
 			goto out;
 		}
 
-		md_bitmap_destroy(mddev, true);
+		md_bitmap_destroy(mddev, false);
+		remove_internal_group(mddev);
 		mddev->bitmap_info.offset = 0;
 		if (mddev->bitmap_info.file) {
 			struct file *f = mddev->bitmap_info.file;
@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
 			 */
 			mddev->bitmap_id = ID_BITMAP;
 			mddev->bitmap_info.offset = offset;
-			rv = md_bitmap_create(mddev, true);
+			rv = md_bitmap_create(mddev, false);
 			if (rv)
 				goto out;
 
+			rv = create_internal_group(mddev);
+			if (rv) {
+				mddev->bitmap_info.offset = 0;
+				bitmap_destroy(mddev);
+				goto out;
+			}
+
 			rv = bitmap_load(mddev);
 			if (rv) {
+				remove_internal_group(mddev);
 				mddev->bitmap_info.offset = 0;
-				md_bitmap_destroy(mddev, true);
+				md_bitmap_destroy(mddev, false);
 				goto out;
 			}
 		}
@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry max_backlog_used =
 __ATTR(max_backlog_used, S_IRUGO | S_IWUSR,
        behind_writes_used_show, behind_writes_used_reset);
 
-static struct attribute *md_bitmap_attrs[] = {
-	&bitmap_location.attr,
+static struct attribute *internal_bitmap_attrs[] = {
 	&bitmap_space.attr,
 	&bitmap_timeout.attr,
 	&bitmap_backlog.attr,
@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = {
 	NULL
 };
 
-static struct attribute_group md_bitmap_group = {
+static struct attribute_group internal_bitmap_group = {
 	.name = "bitmap",
-	.attrs = md_bitmap_attrs,
+	.attrs = internal_bitmap_attrs,
 };
 
+/* Only necessary attrs for compatibility */
+static struct attribute *common_bitmap_attrs[] = {
+	&bitmap_location.attr,
+	NULL
+};
+
+static const struct attribute_group common_bitmap_group = {
+	.name = "bitmap",
+	.attrs = common_bitmap_attrs,
+};
+
+static int create_internal_group(struct mddev *mddev)
+{
+	/*
+	 * md_bitmap_group and md_bitmap_common_group are using same name
+	 * 'bitmap'.
+	 */
+	return sysfs_merge_group(&mddev->kobj, &internal_bitmap_group);
+}
+
+static void remove_internal_group(struct mddev *mddev)
+{
+	sysfs_unmerge_group(&mddev->kobj, &internal_bitmap_group);
+}
+
+static int bitmap_register_groups(struct mddev *mddev)
+{
+	int ret;
+
+	ret = sysfs_create_group(&mddev->kobj, &common_bitmap_group);
+
+	if (ret)
+		return ret;
+
+	ret = sysfs_merge_group(&mddev->kobj, &internal_bitmap_group);
+	if (ret)
+		sysfs_remove_group(&mddev->kobj, &common_bitmap_group);
+
+	return ret;
+}
+
+static void bitmap_unregister_groups(struct mddev *mddev)
+{
+	sysfs_unmerge_group(&mddev->kobj, &internal_bitmap_group);
+}
+
 static struct bitmap_operations bitmap_ops = {
 	.head = {
 		.type	= MD_BITMAP,
@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = {
 	.set_pages		= bitmap_set_pages,
 	.free			= md_bitmap_free,
 
-	.group			= &md_bitmap_group,
+	.register_groups	= bitmap_register_groups,
+	.unregister_groups	= bitmap_unregister_groups,
+};
+
+static int none_bitmap_register_groups(struct mddev *mddev)
+{
+	return sysfs_create_group(&mddev->kobj, &common_bitmap_group);
+}
+
+static struct bitmap_operations none_bitmap_ops = {
+	.head = {
+		.type	= MD_BITMAP,
+		.id	= ID_BITMAP_NONE,
+		.name	= "none",
+	},
+
+	.enabled		= dummy_bitmap_enabled,
+	.create			= dummy_bitmap_create,
+	.destroy		= bitmap_destroy,
+	.load			= bitmap_load,
+	.get_stats		= bitmap_get_stats,
+	.free			= md_bitmap_free,
+
+	.register_groups	= none_bitmap_register_groups,
+	.unregister_groups	= NULL,
 };
 
 int md_bitmap_init(void)
 {
+	int ret;
+
 	md_bitmap_wq = alloc_workqueue("md_bitmap", WQ_MEM_RECLAIM | WQ_UNBOUND,
 				       0);
 	if (!md_bitmap_wq)
 		return -ENOMEM;
 
-	return register_md_submodule(&bitmap_ops.head);
+	ret = register_md_submodule(&bitmap_ops.head);
+	if (ret)
+		return ret;
+
+	return register_md_submodule(&none_bitmap_ops.head);
 }
 
 void md_bitmap_exit(void)
 {
 	destroy_workqueue(md_bitmap_wq);
 	unregister_md_submodule(&bitmap_ops.head);
+	unregister_md_submodule(&none_bitmap_ops.head);
 }
drivers/md/md-bitmap.h +3 -0
index b42a28fa83a0..10bc6854798c 100644
@@ -125,6 +125,9 @@ struct bitmap_operations {
 	void (*set_pages)(void *data, unsigned long pages);
 	void (*free)(void *data);
 
+	int (*register_groups)(struct mddev *mddev);
+	void (*unregister_groups)(struct mddev *mddev);
+
 	struct attribute_group *group;
 };
 
drivers/md/md-llbitmap.c +12 -0
index bf398d7476b3..9b3ea4f1d268 100644
@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = {
 	.attrs = md_llbitmap_attrs,
 };
 
+static int llbitmap_register_groups(struct mddev *mddev)
+{
+	return sysfs_create_group(&mddev->kobj, &md_llbitmap_group);
+}
+
+static void llbitmap_unregister_groups(struct mddev *mddev)
+{
+	sysfs_remove_group(&mddev->kobj, &md_llbitmap_group);
+}
+
 static struct bitmap_operations llbitmap_ops = {
 	.head = {
 		.type	= MD_BITMAP,
@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = {
 	.dirty_bits		= llbitmap_dirty_bits,
 	.write_all		= llbitmap_write_all,
 
+	.register_groups	= llbitmap_register_groups,
+	.unregister_groups	= llbitmap_unregister_groups,
 	.group			= &md_llbitmap_group,
 };
 
drivers/md/md.c +8 -8
index d3c8f77b4fe3..55a95b227b83 100644
@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
 	struct bitmap_operations *old = mddev->bitmap_ops;
 	struct md_submodule_head *head;
 
-	if (mddev->bitmap_id == ID_BITMAP_NONE ||
-	    (old && old->head.id == mddev->bitmap_id))
+	if (old && old->head.id == mddev->bitmap_id)
 		return true;
 
 	xa_lock(&md_submodule);
@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
 	mddev->bitmap_ops = (void *)head;
 	xa_unlock(&md_submodule);
 
-	if (create_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops->group) {
-		if (sysfs_create_group(&mddev->kobj, mddev->bitmap_ops->group))
+	if (create_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops->register_groups) {
+		if (mddev->bitmap_ops->register_groups(mddev))
 			pr_warn("md: cannot register extra bitmap attributes for %s\n",
 				mdname(mddev));
 		else
@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
 static void mddev_clear_bitmap_ops(struct mddev *mddev, bool remove_sysfs)
 {
 	if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops &&
-	    mddev->bitmap_ops->group)
-		sysfs_remove_group(&mddev->kobj, mddev->bitmap_ops->group);
+	    mddev->bitmap_ops->unregister_groups)
+		mddev->bitmap_ops->unregister_groups(mddev);
 
 	mddev->bitmap_ops = NULL;
 }
@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
 			(unsigned long long)pers->size(mddev, 0, 0) / 2);
 		err = -EINVAL;
 	}
-	if (err == 0 && pers->sync_request &&
-	    (mddev->bitmap_info.file || mddev->bitmap_info.offset)) {
+	if (err == 0 && pers->sync_request) {
+		if (mddev->bitmap_info.offset == 0)
+			mddev->bitmap_id = ID_BITMAP_NONE;
 		err = md_bitmap_create(mddev, true);
 		if (err)
 			pr_warn("%s: failed to create bitmap (%d)\n",
Signature · 1 line
2.53.0
#11 lore raw

Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset

Quoted 169 lines · Su Yue

Before commit fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap") if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.

After the commit, if bitmap is none, bitmap/location doesn't exist anymore. It breaks 'grow' behavior of a md array of madam with MD_FEATURE_BITMAP_OFFSET. Take level=mirror and metadata=1.2 as an example:

$ mdadm  --create  /dev/md0 -f --bitmap=none --raid-devices=2 --level=mirror \
  --metadata=1.2 /dev/vdd /dev/vde
$ mdadm --grow /dev/md0 --bitmap=internal
$ cat /sys/block/md0/md/bitmap/location
Before:+8
After: +2

While growing bitmap from none to internal, clustered and llbitmap, mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first. 1)If bitmap/location exists, it sets bitmap/location after getinfo_super(). 2)If bitmap/location doesn't exist, mdadm just calls md_set_array_info() then mddev->bitmap_info.default_offset will be used.
Situation can be worse if growing none to clustered, bitmap offset of the node calling madm --grow will be changed but the other node are reading bitmap sb from the old location.

Here introducing a dummy bitmap_operations for ID_BITMAP_NONE to restore sysfs file bitmap/location for ID_BITMAP_NONE and ID_BITMAP.
location_store() now calls md_bitmap_(create|destroy) with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
 drivers/md/md-bitmap.c   | 116 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +
 drivers/md/md-llbitmap.c |  12 ++++
 drivers/md/md.c          |  16 +++---
 4 files changed, 130 insertions(+), 17 deletions(-)
drivers/md/md-bitmap.c +63 -7
index ac06c9647bf0..a8a176428c61 100644
@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, bool flush)
               bitmap->storage.filemap != NULL;
 }
 
+static bool dummy_bitmap_enabled(void *data, bool flush)
+{
+       return false;
+}
+
 /*
  * check a page and, if necessary, allocate it (or hijack it if the alloc fails)
  *
@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev *mddev)
        return 0;
 }
 
+static int dummy_bitmap_create(struct mddev *mddev)
+{
+       return 0;
+}
+
 static int bitmap_load(struct mddev *mddev)
 {
        int err = 0;
@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char *page)
        return len;
 }
 
+static int create_internal_group(struct mddev *mddev);
+static void remove_internal_group(struct mddev *mddev);
+
 static ssize_t
 location_store(struct mddev *mddev, const char *buf, size_t len)
 {
@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
                        goto out;
                }
 
-               md_bitmap_destroy(mddev, true);
+               md_bitmap_destroy(mddev, false);
+               remove_internal_group(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {
                        struct file *f = mddev->bitmap_info.file;
@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
                         */
                        mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = md_bitmap_create(mddev, true);
+                       rv = md_bitmap_create(mddev, false);
                        if (rv)
                                goto out;
 
+                       rv = create_internal_group(mddev);
+                       if (rv) {
+                               mddev->bitmap_info.offset = 0;
+                               bitmap_destroy(mddev);
+                               goto out;
+                       }
+
                        rv = bitmap_load(mddev);
                        if (rv) {
+                               remove_internal_group(mddev);
                                mddev->bitmap_info.offset = 0;
-                               md_bitmap_destroy(mddev, true);
+                               md_bitmap_destroy(mddev, false);
                                goto out;
                        }
                }
@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry max_backlog_used =
 __ATTR(max_backlog_used, S_IRUGO | S_IWUSR,
        behind_writes_used_show, behind_writes_used_reset);
 
-static struct attribute *md_bitmap_attrs[] = {
-       &bitmap_location.attr,
+static struct attribute *internal_bitmap_attrs[] = {
        &bitmap_space.attr,
        &bitmap_timeout.attr,
        &bitmap_backlog.attr,
@@ -2972,11 +2993,57 @@ static struct attribute *md_bitmap_attrs[] = {
        NULL
 };
 
-static struct attribute_group md_bitmap_group = {
+static struct attribute_group internal_bitmap_group = {
        .name = "bitmap",
-       .attrs = md_bitmap_attrs,
+       .attrs = internal_bitmap_attrs,
 };
 
+/* Only necessary attrs for compatibility */
+static struct attribute *common_bitmap_attrs[] = {
+       &bitmap_location.attr,
+       NULL
+};
+
+static const struct attribute_group common_bitmap_group = {
+       .name = "bitmap",
+       .attrs = common_bitmap_attrs,
+};
+
+static int create_internal_group(struct mddev *mddev)
+{
+       /*
+        * md_bitmap_group and md_bitmap_common_group are using same name
+        * 'bitmap'.
+        */
+       return sysfs_merge_group(&mddev->kobj, &internal_bitmap_group);
+}
+
+static void remove_internal_group(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj, &internal_bitmap_group);
+}
+
+static int bitmap_register_groups(struct mddev *mddev)
+{
+       int ret;
+
+       ret = sysfs_create_group(&mddev->kobj, &common_bitmap_group);
+
+       if (ret)
+               return ret;
+
+       ret = sysfs_merge_group(&mddev->kobj, &internal_bitmap_group);

drivers/md/md-bitmap.c static struct attribute *md_bitmap_attrs[] = { · line 3045

Quoted 10 lines
drivers/md/md-bitmap.c +10 -0
+       if (ret)
+               sysfs_remove_group(&mddev->kobj, &common_bitmap_group);
+
+       return ret;
+}
+
+static void bitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj, &internal_bitmap_group);
+}

Hi Su

bitmap_unregister_groups should also remove common_bitmap_group, right?

drivers/md/md-bitmap.c static struct attribute *md_bitmap_attrs[] = { · line 3047

Quoted 2 lines
drivers/md/md-bitmap.c +1 -0
+
 static struct bitmap_operations bitmap_ops = {

You already changed md_bitmap_group to internal_bitmap_group. It's better to change bitmap_ops to internal_bitmap_ops?

Quoted 22 lines
drivers/md/md-bitmap.c +15 -1
        .head = {
                .type   = MD_BITMAP,
@@ -3018,21 +3085,52 @@ static struct bitmap_operations bitmap_ops = {
        .set_pages              = bitmap_set_pages,
        .free                   = md_bitmap_free,
 
-       .group                  = &md_bitmap_group,
+       .register_groups        = bitmap_register_groups,
+       .unregister_groups      = bitmap_unregister_groups,
+};
+
+static int none_bitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj, &common_bitmap_group);
+}
+
+static struct bitmap_operations none_bitmap_ops = {
+       .head = {
+               .type   = MD_BITMAP,
+               .id     = ID_BITMAP_NONE,
+               .name   = "none",
+       },

drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112

Quoted 10 lines
drivers/md/md-bitmap.c +10 -0
+
+       .enabled                = dummy_bitmap_enabled,
+       .create                 = dummy_bitmap_create,
+       .destroy                = bitmap_destroy,
+       .load                   = bitmap_load,
+       .get_stats              = bitmap_get_stats,
+       .free                   = md_bitmap_free,
+
+       .register_groups        = none_bitmap_register_groups,
+       .unregister_groups      = NULL,

How does bitmap/location can be deleted if array is created with no bitmap? Can the mdadm --stop command get stuck?

Best Regards
Xiao

Quoted 120 lines
drivers/md/md-bitmap.c +8 -1
 };
 
 int md_bitmap_init(void)
 {
+       int ret;
+
        md_bitmap_wq = alloc_workqueue("md_bitmap", WQ_MEM_RECLAIM | WQ_UNBOUND,
                                       0);
        if (!md_bitmap_wq)
                return -ENOMEM;
 
-       return register_md_submodule(&bitmap_ops.head);
+       ret = register_md_submodule(&bitmap_ops.head);
+       if (ret)
+               return ret;
+
+       return register_md_submodule(&none_bitmap_ops.head);
 }
 
 void md_bitmap_exit(void)
 {
        destroy_workqueue(md_bitmap_wq);
        unregister_md_submodule(&bitmap_ops.head);
+       unregister_md_submodule(&none_bitmap_ops.head);
 }
drivers/md/md-bitmap.h +3 -0
index b42a28fa83a0..10bc6854798c 100644
@@ -125,6 +125,9 @@ struct bitmap_operations {
        void (*set_pages)(void *data, unsigned long pages);
        void (*free)(void *data);
 
+       int (*register_groups)(struct mddev *mddev);
+       void (*unregister_groups)(struct mddev *mddev);
+
        struct attribute_group *group;
 };
 
drivers/md/md-llbitmap.c +12 -0
index bf398d7476b3..9b3ea4f1d268 100644
@@ -1561,6 +1561,16 @@ static struct attribute_group md_llbitmap_group = {
        .attrs = md_llbitmap_attrs,
 };
 
+static int llbitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj, &md_llbitmap_group);
+}
+
+static void llbitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_remove_group(&mddev->kobj, &md_llbitmap_group);
+}
+
 static struct bitmap_operations llbitmap_ops = {
        .head = {
                .type   = MD_BITMAP,
@@ -1597,6 +1607,8 @@ static struct bitmap_operations llbitmap_ops = {
        .dirty_bits             = llbitmap_dirty_bits,
        .write_all              = llbitmap_write_all,
 
+       .register_groups        = llbitmap_register_groups,
+       .unregister_groups      = llbitmap_unregister_groups,
        .group                  = &md_llbitmap_group,
 };
 
drivers/md/md.c +8 -9
index d3c8f77b4fe3..55a95b227b83 100644
@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
        struct bitmap_operations *old = mddev->bitmap_ops;
        struct md_submodule_head *head;
 
-       if (mddev->bitmap_id == ID_BITMAP_NONE ||
-           (old && old->head.id == mddev->bitmap_id))
+       if (old && old->head.id == mddev->bitmap_id)
                return true;
 
        xa_lock(&md_submodule);
@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
        mddev->bitmap_ops = (void *)head;
        xa_unlock(&md_submodule);
 
-       if (create_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops->group) {
-               if (sysfs_create_group(&mddev->kobj, mddev->bitmap_ops->group))
+       if (create_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops->register_groups) {
+               if (mddev->bitmap_ops->register_groups(mddev))
                        pr_warn("md: cannot register extra bitmap attributes for %s\n",
                                mdname(mddev));
                else
@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct mddev *mddev, bool create_sysfs)
 static void mddev_clear_bitmap_ops(struct mddev *mddev, bool remove_sysfs)
 {
        if (remove_sysfs && !mddev_is_dm(mddev) && mddev->bitmap_ops &&
-           mddev->bitmap_ops->group)
-               sysfs_remove_group(&mddev->kobj, mddev->bitmap_ops->group);
+           mddev->bitmap_ops->unregister_groups)
+               mddev->bitmap_ops->unregister_groups(mddev);
 
        mddev->bitmap_ops = NULL;
 }
@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
                        (unsigned long long)pers->size(mddev, 0, 0) / 2);
                err = -EINVAL;
        }
-       if (err == 0 && pers->sync_request &&
-           (mddev->bitmap_info.file || mddev->bitmap_info.offset)) {
+       if (err == 0 && pers->sync_request) {
+               if (mddev->bitmap_info.offset == 0)
+                       mddev->bitmap_id = ID_BITMAP_NONE;
                err = md_bitmap_create(mddev, true);
                if (err)
                        pr_warn("%s: failed to create bitmap (%d)\n",
--

2.53.0

#12 lore raw

Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset

Quoted 203 lines · Xiao Ni
Quoted 197 lines · Su Yue

Before commit fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap") if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.

After the commit, if bitmap is none, bitmap/location doesn't exist anymore. It breaks 'grow' behavior of a md array of madam with MD_FEATURE_BITMAP_OFFSET. Take level=mirror and metadata=1.2 as an example:

$ mdadm  --create  /dev/md0 -f --bitmap=none --raid-devices=2 
--level=mirror \
  --metadata=1.2 /dev/vdd /dev/vde
$ mdadm --grow /dev/md0 --bitmap=internal
$ cat /sys/block/md0/md/bitmap/location
Before:+8
After: +2

While growing bitmap from none to internal, clustered and llbitmap, mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first. 1)If bitmap/location exists, it sets bitmap/location after getinfo_super(). 2)If bitmap/location doesn't exist, mdadm just calls md_set_array_info() then mddev->bitmap_info.default_offset will be used.
Situation can be worse if growing none to clustered, bitmap offset of the node calling madm --grow will be changed but the other node are reading bitmap sb from the old location.

Here introducing a dummy bitmap_operations for ID_BITMAP_NONE to restore sysfs file bitmap/location for ID_BITMAP_NONE and ID_BITMAP.
location_store() now calls md_bitmap_(create|destroy) with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
---
 drivers/md/md-bitmap.c   | 116 
 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +
 drivers/md/md-llbitmap.c |  12 ++++
 drivers/md/md.c          |  16 +++---
 4 files changed, 130 insertions(+), 17 deletions(-)
drivers/md/md-bitmap.c +5 -0
index ac06c9647bf0..a8a176428c61 100644
@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data, 
 bool flush)
               bitmap->storage.filemap != NULL;
 }
 
+static bool dummy_bitmap_enabled(void *data, bool flush)
+{
+       return false;
+}
+
 /*
  * check a page and, if necessary, allocate it (or hijack it 

if the alloc fails)
*

drivers/md/md-bitmap.c +5 -0
@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev 
 *mddev)
        return 0;
 }
 
+static int dummy_bitmap_create(struct mddev *mddev)
+{
+       return 0;
+}
+
 static int bitmap_load(struct mddev *mddev)
 {
        int err = 0;
drivers/md/md-bitmap.c +3 -0
@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char 
 *page)
        return len;
 }
 
+static int create_internal_group(struct mddev *mddev);
+static void remove_internal_group(struct mddev *mddev);
+
 static ssize_t
 location_store(struct mddev *mddev, const char *buf, size_t 

len)
{

drivers/md/md-bitmap.c +2 -1
@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const 
 char *buf, size_t len)
                        goto out;
                }
 
-               md_bitmap_destroy(mddev, true);
+               md_bitmap_destroy(mddev, false);
+               remove_internal_group(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {

struct file *f =
mddev->bitmap_info.file;

drivers/md/md-bitmap.c +10 -2
@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev, 
 const char *buf, size_t len)
                         */
                        mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = md_bitmap_create(mddev, true);
+                       rv = md_bitmap_create(mddev, false);
                        if (rv)
                                goto out;
 
+                       rv = create_internal_group(mddev);
+                       if (rv) {
+                               mddev->bitmap_info.offset = 0;
+                               bitmap_destroy(mddev);
+                               goto out;
+                       }
+
                        rv = bitmap_load(mddev);
                        if (rv) {
+                               remove_internal_group(mddev);
                                mddev->bitmap_info.offset = 0;
-                               md_bitmap_destroy(mddev, true);
+                               md_bitmap_destroy(mddev, 
 false);
                                goto out;
                        }
                }
drivers/md/md-bitmap.c +0 -0
@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry 

max_backlog_used =
__ATTR(max_backlog_used, S_IRUGO | S_IWUSR,

        behind_writes_used_show, behind_writes_used_reset);
drivers/md/md-bitmap.c +18 -4
-static struct attribute *md_bitmap_attrs[] = {
-       &bitmap_location.attr,
+static struct attribute *internal_bitmap_attrs[] = {
        &bitmap_space.attr,
        &bitmap_timeout.attr,
        &bitmap_backlog.attr,
@@ -2972,11 +2993,57 @@ static struct attribute 
 *md_bitmap_attrs[] = {
        NULL
 };
 
-static struct attribute_group md_bitmap_group = {
+static struct attribute_group internal_bitmap_group = {
        .name = "bitmap",
-       .attrs = md_bitmap_attrs,
+       .attrs = internal_bitmap_attrs,
 };
 
+/* Only necessary attrs for compatibility */
+static struct attribute *common_bitmap_attrs[] = {
+       &bitmap_location.attr,
+       NULL
+};
+
+static const struct attribute_group common_bitmap_group = {
+       .name = "bitmap",
+       .attrs = common_bitmap_attrs,
+};
+
+static int create_internal_group(struct mddev *mddev)
+{
+       /*
+        * md_bitmap_group and md_bitmap_common_group are using 

same name

drivers/md/md-bitmap.c +3 -0
+        * 'bitmap'.
+        */
+       return sysfs_merge_group(&mddev->kobj, 

&internal_bitmap_group);

drivers/md/md-bitmap.c +5 -0
+}
+
+static void remove_internal_group(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj, 

&internal_bitmap_group);

drivers/md/md-bitmap.c +7 -0
+}
+
+static int bitmap_register_groups(struct mddev *mddev)
+{
+       int ret;
+
+       ret = sysfs_create_group(&mddev->kobj, 

&common_bitmap_group);

drivers/md/md-bitmap.c +5 -0
+
+       if (ret)
+               return ret;
+
+       ret = sysfs_merge_group(&mddev->kobj, 

&internal_bitmap_group);

drivers/md/md-bitmap.c +2 -0
+       if (ret)
+               sysfs_remove_group(&mddev->kobj, 

&common_bitmap_group);

drivers/md/md-bitmap.c +7 -0
+
+       return ret;
+}
+
+static void bitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj, 
Quoted 2 lines

&internal_bitmap_group);
+}

Hi Su

Quoted 2 lines

bitmap_unregister_groups should also remove common_bitmap_group, right?

As you pasted before, mdadm:Grow.c:

int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s) {
...

	if (array.state & (1 << MD_SB_BITMAP_PRESENT)) {
		if (s->btype == BitmapNone) {
			array.state &= ~(1 << MD_SB_BITMAP_PRESENT);
			if (md_set_array_info(fd, &array) != 0) {
				if (array.state & (1 << MD_SB_CLUSTERED))
					pr_err("failed to remove clustered 
					bitmap.\n");
				else
					pr_err("failed to remove internal bitmap.\n");
				return 1;
			}
			return 0;
		}
		pr_err("bitmap already present on %s\n", devname);
		return 1;
	}

...
}

In case of growing from internal to none, bitmap_unregister_groups() will be called, if common_bitmap_group is removed, bitmap/location won't exist. update_array_info() is a common function used by many call paths. Also llbitmap is involved here. I don't want to make situation and code more complicated like adding more codes in update_array_info().

Quoted 35 lines
Quoted 2 lines

+
static struct bitmap_operations bitmap_ops = {

You already changed md_bitmap_group to internal_bitmap_group. It's better to change bitmap_ops to internal_bitmap_ops?

Quoted 29 lines

.head = {
.type = MD_BITMAP,

drivers/md/md-bitmap.c +20 -1
@@ -3018,21 +3085,52 @@ static struct bitmap_operations 
 bitmap_ops = {
        .set_pages              = bitmap_set_pages,
        .free                   = md_bitmap_free,
 
-       .group                  = &md_bitmap_group,
+       .register_groups        = bitmap_register_groups,
+       .unregister_groups      = bitmap_unregister_groups,
+};
+
+static int none_bitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj, 
 &common_bitmap_group);
+}
+
+static struct bitmap_operations none_bitmap_ops = {
+       .head = {
+               .type   = MD_BITMAP,
+               .id     = ID_BITMAP_NONE,
+               .name   = "none",
+       },
+
+       .enabled                = dummy_bitmap_enabled,
+       .create                 = dummy_bitmap_create,
+       .destroy                = bitmap_destroy,
+       .load                   = bitmap_load,
Quoted 9 lines

drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112

Quoted 5 lines
drivers/md/md-bitmap.c +5 -0
+       .get_stats              = bitmap_get_stats,
+       .free                   = md_bitmap_free,
+
+       .register_groups        = none_bitmap_register_groups,
+       .unregister_groups      = NULL,

How does bitmap/location can be deleted if array is created with no bitmap? Can the mdadm --stop command get stuck?

No. It wont get stuck.

I guess here your concern is the timing of removing common_bitmap_group. The life cycle is same as before fb8cc3b0d9db. At the time llbitmap is not introduced, because there is no need to switch bitmap ops, so all attrs of bitmap are removed in kobject_put() in mddev_delayed_delete() queued by mddev_put(). This is now where common_bitmap_group is being removed.

--
Su

Quoted 140 lines

Best Regards
Xiao

Quoted 137 lines
drivers/md/md-bitmap.c +8 -1
 };
 
 int md_bitmap_init(void)
 {
+       int ret;
+
        md_bitmap_wq = alloc_workqueue("md_bitmap", 
        WQ_MEM_RECLAIM | WQ_UNBOUND,
                                       0);
        if (!md_bitmap_wq)
                return -ENOMEM;
 
-       return register_md_submodule(&bitmap_ops.head);
+       ret = register_md_submodule(&bitmap_ops.head);
+       if (ret)
+               return ret;
+
+       return register_md_submodule(&none_bitmap_ops.head);
 }
 
 void md_bitmap_exit(void)
 {
        destroy_workqueue(md_bitmap_wq);
        unregister_md_submodule(&bitmap_ops.head);
+       unregister_md_submodule(&none_bitmap_ops.head);
 }
drivers/md/md-bitmap.h +3 -0
index b42a28fa83a0..10bc6854798c 100644
@@ -125,6 +125,9 @@ struct bitmap_operations {
        void (*set_pages)(void *data, unsigned long pages);
        void (*free)(void *data);
 
+       int (*register_groups)(struct mddev *mddev);
+       void (*unregister_groups)(struct mddev *mddev);
+
        struct attribute_group *group;
 };
 
drivers/md/md-llbitmap.c +10 -0
index bf398d7476b3..9b3ea4f1d268 100644
@@ -1561,6 +1561,16 @@ static struct attribute_group 
 md_llbitmap_group = {
        .attrs = md_llbitmap_attrs,
 };
 
+static int llbitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj, 
 &md_llbitmap_group);
+}
+
+static void llbitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_remove_group(&mddev->kobj, &md_llbitmap_group);
+}
+
 static struct bitmap_operations llbitmap_ops = {

.head = {
.type = MD_BITMAP,

drivers/md/md-llbitmap.c +2 -0
@@ -1597,6 +1607,8 @@ static struct bitmap_operations 
 llbitmap_ops = {
        .dirty_bits             = llbitmap_dirty_bits,
        .write_all              = llbitmap_write_all,
 
+       .register_groups        = llbitmap_register_groups,
+       .unregister_groups      = llbitmap_unregister_groups,
        .group                  = &md_llbitmap_group,
 };
drivers/md/md.c +1 -2
index d3c8f77b4fe3..55a95b227b83 100644
@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct 
 mddev *mddev, bool create_sysfs)
        struct bitmap_operations *old = mddev->bitmap_ops;
        struct md_submodule_head *head;
 
-       if (mddev->bitmap_id == ID_BITMAP_NONE ||
-           (old && old->head.id == mddev->bitmap_id))
+       if (old && old->head.id == mddev->bitmap_id)
                return true;
 
        xa_lock(&md_submodule);
drivers/md/md.c +1 -2
@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct 
 mddev *mddev, bool create_sysfs)
        mddev->bitmap_ops = (void *)head;
        xa_unlock(&md_submodule);
 
-       if (create_sysfs && !mddev_is_dm(mddev) && 
 mddev->bitmap_ops->group) {
-               if (sysfs_create_group(&mddev->kobj, 
 mddev->bitmap_ops->group))
+       if (create_sysfs && !mddev_is_dm(mddev) && 

mddev->bitmap_ops->register_groups) {

drivers/md/md.c +1 -2
+               if (mddev->bitmap_ops->register_groups(mddev))
                        pr_warn("md: cannot register extra 
                        bitmap attributes for %s\n",
                                mdname(mddev));
                else
@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct 
 mddev *mddev, bool create_sysfs)
 static void mddev_clear_bitmap_ops(struct mddev *mddev, bool 
 remove_sysfs)
 {
        if (remove_sysfs && !mddev_is_dm(mddev) && 
        mddev->bitmap_ops &&
-           mddev->bitmap_ops->group)
-               sysfs_remove_group(&mddev->kobj, 

mddev->bitmap_ops->group);

drivers/md/md.c +5 -2
+           mddev->bitmap_ops->unregister_groups)
+               mddev->bitmap_ops->unregister_groups(mddev);
 
        mddev->bitmap_ops = NULL;
 }
@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
                        (unsigned long long)pers->size(mddev, 
                        0, 0) / 2);
                err = -EINVAL;
        }
-       if (err == 0 && pers->sync_request &&
-           (mddev->bitmap_info.file || 
 mddev->bitmap_info.offset)) {
+       if (err == 0 && pers->sync_request) {
+               if (mddev->bitmap_info.offset == 0)
+                       mddev->bitmap_id = ID_BITMAP_NONE;
                err = md_bitmap_create(mddev, true);

if (err)
pr_warn("%s: failed to create bitmap (%d)\n",
--
2.53.0

#13 lore raw

Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset

Quoted 249 lines · Su Yue
Quoted 217 lines · Xiao Ni
Quoted 211 lines · Su Yue

Before commit fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.

After the commit, if bitmap is none, bitmap/location doesn't exist anymore.
It breaks 'grow' behavior of a md array of madam with MD_FEATURE_BITMAP_OFFSET.
Take level=mirror and metadata=1.2 as an example:

$ mdadm  --create  /dev/md0 -f --bitmap=none --raid-devices=2
--level=mirror \
  --metadata=1.2 /dev/vdd /dev/vde
$ mdadm --grow /dev/md0 --bitmap=internal
$ cat /sys/block/md0/md/bitmap/location
Before:+8
After: +2

While growing bitmap from none to internal, clustered and llbitmap,
mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first.
1)If bitmap/location exists, it sets bitmap/location after getinfo_super().
2)If bitmap/location doesn't exist, mdadm just calls md_set_array_info() then mddev->bitmap_info.default_offset will be used. Situation can be worse if growing none to clustered, bitmap offset of the node
calling madm --grow will be changed but the other node are reading bitmap sb from
the old location.

Here introducing a dummy bitmap_operations for ID_BITMAP_NONE to restore sysfs
file bitmap/location for ID_BITMAP_NONE and ID_BITMAP. location_store() now calls md_bitmap_(create|destroy) with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of

bitmap_ops until creating bitmap")

---
 drivers/md/md-bitmap.c   | 116
 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +
 drivers/md/md-llbitmap.c |  12 ++++
 drivers/md/md.c          |  16 +++---
 4 files changed, 130 insertions(+), 17 deletions(-)
drivers/md/md-bitmap.c +5 -0
index ac06c9647bf0..a8a176428c61 100644
@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data,
 bool flush)
               bitmap->storage.filemap != NULL;
 }
 
+static bool dummy_bitmap_enabled(void *data, bool flush)
+{
+       return false;
+}
+
 /*
  * check a page and, if necessary, allocate it (or hijack it

if the alloc fails)
*

drivers/md/md-bitmap.c +5 -0
@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev
 *mddev)
        return 0;
 }
 
+static int dummy_bitmap_create(struct mddev *mddev)
+{
+       return 0;
+}
+
 static int bitmap_load(struct mddev *mddev)
 {
        int err = 0;
drivers/md/md-bitmap.c +3 -0
@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char
 *page)
        return len;
 }
 
+static int create_internal_group(struct mddev *mddev);
+static void remove_internal_group(struct mddev *mddev);
+
 static ssize_t
 location_store(struct mddev *mddev, const char *buf, size_t

len)
{

drivers/md/md-bitmap.c +2 -1
@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const
 char *buf, size_t len)
                        goto out;
                }
 
-               md_bitmap_destroy(mddev, true);
+               md_bitmap_destroy(mddev, false);
+               remove_internal_group(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {

struct file *f =
mddev->bitmap_info.file;

drivers/md/md-bitmap.c +10 -2
@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,
 const char *buf, size_t len)
                         */
                        mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = md_bitmap_create(mddev, true);
+                       rv = md_bitmap_create(mddev, false);
                        if (rv)
                                goto out;
 
+                       rv = create_internal_group(mddev);
+                       if (rv) {
+                               mddev->bitmap_info.offset = 0;
+                               bitmap_destroy(mddev);
+                               goto out;
+                       }
+
                        rv = bitmap_load(mddev);
                        if (rv) {
+                               remove_internal_group(mddev);
                                mddev->bitmap_info.offset = 0;
-                               md_bitmap_destroy(mddev, true);
+                               md_bitmap_destroy(mddev,
 false);
                                goto out;
                        }
                }
drivers/md/md-bitmap.c +0 -0
@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry

max_backlog_used =
__ATTR(max_backlog_used, S_IRUGO | S_IWUSR,

        behind_writes_used_show, behind_writes_used_reset);
drivers/md/md-bitmap.c +18 -4
-static struct attribute *md_bitmap_attrs[] = {
-       &bitmap_location.attr,
+static struct attribute *internal_bitmap_attrs[] = {
        &bitmap_space.attr,
        &bitmap_timeout.attr,
        &bitmap_backlog.attr,
@@ -2972,11 +2993,57 @@ static struct attribute
 *md_bitmap_attrs[] = {
        NULL
 };
 
-static struct attribute_group md_bitmap_group = {
+static struct attribute_group internal_bitmap_group = {
        .name = "bitmap",
-       .attrs = md_bitmap_attrs,
+       .attrs = internal_bitmap_attrs,
 };
 
+/* Only necessary attrs for compatibility */
+static struct attribute *common_bitmap_attrs[] = {
+       &bitmap_location.attr,
+       NULL
+};
+
+static const struct attribute_group common_bitmap_group = {
+       .name = "bitmap",
+       .attrs = common_bitmap_attrs,
+};
+
+static int create_internal_group(struct mddev *mddev)
+{
+       /*
+        * md_bitmap_group and md_bitmap_common_group are using

same name

drivers/md/md-bitmap.c +3 -0
+        * 'bitmap'.
+        */
+       return sysfs_merge_group(&mddev->kobj,

&internal_bitmap_group);

drivers/md/md-bitmap.c +5 -0
+}
+
+static void remove_internal_group(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj,

&internal_bitmap_group);

drivers/md/md-bitmap.c +7 -0
+}
+
+static int bitmap_register_groups(struct mddev *mddev)
+{
+       int ret;
+
+       ret = sysfs_create_group(&mddev->kobj,

&common_bitmap_group);

drivers/md/md-bitmap.c +5 -0
+
+       if (ret)
+               return ret;
+
+       ret = sysfs_merge_group(&mddev->kobj,

&internal_bitmap_group);

drivers/md/md-bitmap.c +2 -0
+       if (ret)
+               sysfs_remove_group(&mddev->kobj,

&common_bitmap_group);

drivers/md/md-bitmap.c +7 -0
+
+       return ret;
+}
+
+static void bitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj,
Quoted 2 lines

&internal_bitmap_group);
+}

Hi Su

Quoted 3 lines

bitmap_unregister_groups should also remove common_bitmap_group, right?

As you pasted before, mdadm:Grow.c:

int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
{
...

        if (array.state & (1 << MD_SB_BITMAP_PRESENT)) {
                if (s->btype == BitmapNone) {
                        array.state &= ~(1 << MD_SB_BITMAP_PRESENT);
                        if (md_set_array_info(fd, &array) != 0) {
                                if (array.state & (1 << MD_SB_CLUSTERED))
                                        pr_err("failed to remove clustered
                                        bitmap.\n");
                                else
                                        pr_err("failed to remove internal bitmap.\n");
                                return 1;
                        }
                        return 0;
                }
                pr_err("bitmap already present on %s\n", devname);
                return 1;
        }

...
}

Quoted 9 lines

In case of growing from internal to none, bitmap_unregister_groups() will
be called, if common_bitmap_group is removed, bitmap/location won't exist.
update_array_info() is a common function used by many call paths. Also
llbitmap is involved here. I don't want to make situation and code more
complicated like adding more codes in update_array_info().

The above mdadm codes use bitmap/location. In your patch, you already pass false to md_bitmap_destroy in location_store. So removing common_bitmap_group in bitmap_unregister_groups can't affect the above case.

But after reading your below comments. It's good to me that bitmap_unregister_groups doesn't remove common_bitmap_group.

Quoted 50 lines
Quoted 39 lines
Quoted 4 lines

+
static struct bitmap_operations bitmap_ops = {

You already changed md_bitmap_group to internal_bitmap_group. It's
better to change bitmap_ops to internal_bitmap_ops?

Quoted 30 lines

.head = {
.type = MD_BITMAP,

drivers/md/md-bitmap.c +21 -1
@@ -3018,21 +3085,52 @@ static struct bitmap_operations
 bitmap_ops = {
        .set_pages              = bitmap_set_pages,
        .free                   = md_bitmap_free,
 
-       .group                  = &md_bitmap_group,
+       .register_groups        = bitmap_register_groups,
+       .unregister_groups      = bitmap_unregister_groups,
+};
+
+static int none_bitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj,
 &common_bitmap_group);
+}
+
+static struct bitmap_operations none_bitmap_ops = {
+       .head = {
+               .type   = MD_BITMAP,
+               .id     = ID_BITMAP_NONE,
+               .name   = "none",
+       },
+
+       .enabled                = dummy_bitmap_enabled,
+       .create                 = dummy_bitmap_create,
+       .destroy                = bitmap_destroy,
+       .load                   = bitmap_load,
+       .get_stats              = bitmap_get_stats,
Quoted 9 lines

drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112

Quoted 4 lines
drivers/md/md-bitmap.c +4 -0
+       .free                   = md_bitmap_free,
+
+       .register_groups        = none_bitmap_register_groups,
+       .unregister_groups      = NULL,

How does bitmap/location can be deleted if array is created with no
bitmap? Can the mdadm --stop command get stuck?

No. It wont get stuck.

Quoted 10 lines

I guess here your concern is the timing of removing common_bitmap_group.
The life cycle is same as before fb8cc3b0d9db. At the time llbitmap is
not introduced, because there is no need to switch bitmap ops, so all
attrs of bitmap are removed in kobject_put() in mddev_delayed_delete()
queued by mddev_put(). This is now where common_bitmap_group is being removed.

Thanks for the explanation.

Best Regards
Xiao

Quoted 145 lines

--
Su

Quoted 141 lines

Best Regards
Xiao

Quoted 138 lines
drivers/md/md-bitmap.c +8 -1
 };
 
 int md_bitmap_init(void)
 {
+       int ret;
+
        md_bitmap_wq = alloc_workqueue("md_bitmap",
        WQ_MEM_RECLAIM | WQ_UNBOUND,
                                       0);
        if (!md_bitmap_wq)
                return -ENOMEM;
 
-       return register_md_submodule(&bitmap_ops.head);
+       ret = register_md_submodule(&bitmap_ops.head);
+       if (ret)
+               return ret;
+
+       return register_md_submodule(&none_bitmap_ops.head);
 }
 
 void md_bitmap_exit(void)
 {
        destroy_workqueue(md_bitmap_wq);
        unregister_md_submodule(&bitmap_ops.head);
+       unregister_md_submodule(&none_bitmap_ops.head);
 }
drivers/md/md-bitmap.h +3 -0
index b42a28fa83a0..10bc6854798c 100644
@@ -125,6 +125,9 @@ struct bitmap_operations {
        void (*set_pages)(void *data, unsigned long pages);
        void (*free)(void *data);
 
+       int (*register_groups)(struct mddev *mddev);
+       void (*unregister_groups)(struct mddev *mddev);
+
        struct attribute_group *group;
 };
 
drivers/md/md-llbitmap.c +10 -0
index bf398d7476b3..9b3ea4f1d268 100644
@@ -1561,6 +1561,16 @@ static struct attribute_group
 md_llbitmap_group = {
        .attrs = md_llbitmap_attrs,
 };
 
+static int llbitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj,
 &md_llbitmap_group);
+}
+
+static void llbitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_remove_group(&mddev->kobj, &md_llbitmap_group);
+}
+
 static struct bitmap_operations llbitmap_ops = {

.head = {
.type = MD_BITMAP,

drivers/md/md-llbitmap.c +2 -0
@@ -1597,6 +1607,8 @@ static struct bitmap_operations
 llbitmap_ops = {
        .dirty_bits             = llbitmap_dirty_bits,
        .write_all              = llbitmap_write_all,
 
+       .register_groups        = llbitmap_register_groups,
+       .unregister_groups      = llbitmap_unregister_groups,
        .group                  = &md_llbitmap_group,
 };
drivers/md/md.c +1 -2
index d3c8f77b4fe3..55a95b227b83 100644
@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct
 mddev *mddev, bool create_sysfs)
        struct bitmap_operations *old = mddev->bitmap_ops;
        struct md_submodule_head *head;
 
-       if (mddev->bitmap_id == ID_BITMAP_NONE ||
-           (old && old->head.id == mddev->bitmap_id))
+       if (old && old->head.id == mddev->bitmap_id)
                return true;
 
        xa_lock(&md_submodule);
drivers/md/md.c +1 -2
@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct
 mddev *mddev, bool create_sysfs)
        mddev->bitmap_ops = (void *)head;
        xa_unlock(&md_submodule);
 
-       if (create_sysfs && !mddev_is_dm(mddev) &&
 mddev->bitmap_ops->group) {
-               if (sysfs_create_group(&mddev->kobj,
 mddev->bitmap_ops->group))
+       if (create_sysfs && !mddev_is_dm(mddev) &&

mddev->bitmap_ops->register_groups) {

drivers/md/md.c +1 -2
+               if (mddev->bitmap_ops->register_groups(mddev))
                        pr_warn("md: cannot register extra
                        bitmap attributes for %s\n",
                                mdname(mddev));
                else
@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct
 mddev *mddev, bool create_sysfs)
 static void mddev_clear_bitmap_ops(struct mddev *mddev, bool
 remove_sysfs)
 {
        if (remove_sysfs && !mddev_is_dm(mddev) &&
        mddev->bitmap_ops &&
-           mddev->bitmap_ops->group)
-               sysfs_remove_group(&mddev->kobj,

mddev->bitmap_ops->group);

drivers/md/md.c +5 -2
+           mddev->bitmap_ops->unregister_groups)
+               mddev->bitmap_ops->unregister_groups(mddev);
 
        mddev->bitmap_ops = NULL;
 }
@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
                        (unsigned long long)pers->size(mddev,
                        0, 0) / 2);
                err = -EINVAL;
        }
-       if (err == 0 && pers->sync_request &&
-           (mddev->bitmap_info.file ||
 mddev->bitmap_info.offset)) {
+       if (err == 0 && pers->sync_request) {
+               if (mddev->bitmap_info.offset == 0)
+                       mddev->bitmap_id = ID_BITMAP_NONE;
                err = md_bitmap_create(mddev, true);

if (err)
pr_warn("%s: failed to create bitmap (%d)\n",
--
2.53.0

#14 lore raw

Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset

Quoted 276 lines · Xiao Ni
Quoted 265 lines · Su Yue
Quoted 226 lines · Xiao Ni
Quoted 220 lines · Su Yue

Before commit fb8cc3b0d9db ("md/md-bitmap: delay registration of bitmap_ops until creating bitmap")
if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and clustered have
the sysfs file bitmap/location.

After the commit, if bitmap is none, bitmap/location doesn't exist anymore.
It breaks 'grow' behavior of a md array of madam with MD_FEATURE_BITMAP_OFFSET.
Take level=mirror and metadata=1.2 as an example:

$ mdadm  --create  /dev/md0 -f --bitmap=none 
--raid-devices=2
--level=mirror \
  --metadata=1.2 /dev/vdd /dev/vde
$ mdadm --grow /dev/md0 --bitmap=internal
$ cat /sys/block/md0/md/bitmap/location
Before:+8
After: +2

While growing bitmap from none to internal, clustered and llbitmap,
mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location first.
1)If bitmap/location exists, it sets bitmap/location after getinfo_super().
2)If bitmap/location doesn't exist, mdadm just calls md_set_array_info() then mddev->bitmap_info.default_offset will be used. Situation can be worse if growing none to clustered, bitmap offset of the node
calling madm --grow will be changed but the other node are reading bitmap sb from
the old location.

Here introducing a dummy bitmap_operations for ID_BITMAP_NONE to restore sysfs
file bitmap/location for ID_BITMAP_NONE and ID_BITMAP. location_store() now calls md_bitmap_(create|destroy) with false sysfs parameter then (add,remove)_internal_bitmap() will be called to manage sysfs entries.

Fixes fb8cc3b0d9db ("md/md-bitmap: delay registration of

bitmap_ops until creating bitmap")

---
 drivers/md/md-bitmap.c   | 116
 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +
 drivers/md/md-llbitmap.c |  12 ++++
 drivers/md/md.c          |  16 +++---
 4 files changed, 130 insertions(+), 17 deletions(-)
drivers/md/md-bitmap.c +5 -0
index ac06c9647bf0..a8a176428c61 100644
@@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data,
 bool flush)
               bitmap->storage.filemap != NULL;
 }
 
+static bool dummy_bitmap_enabled(void *data, bool flush)
+{
+       return false;
+}
+
 /*
  * check a page and, if necessary, allocate it (or hijack 

it
if the alloc fails)
*

drivers/md/md-bitmap.c +5 -0
@@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev
 *mddev)
        return 0;
 }
 
+static int dummy_bitmap_create(struct mddev *mddev)
+{
+       return 0;
+}
+
 static int bitmap_load(struct mddev *mddev)
 {
        int err = 0;
drivers/md/md-bitmap.c +0 -0
@@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, 

char
*page)

        return len;

}

drivers/md/md-bitmap.c +3 -0
+static int create_internal_group(struct mddev *mddev);
+static void remove_internal_group(struct mddev *mddev);
+
 static ssize_t
 location_store(struct mddev *mddev, const char *buf, size_t
 len)
 {
@@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, 

const
char *buf, size_t len)

                        goto out;
                }
drivers/md/md-bitmap.c +11 -3
-               md_bitmap_destroy(mddev, true);
+               md_bitmap_destroy(mddev, false);
+               remove_internal_group(mddev);
                mddev->bitmap_info.offset = 0;
                if (mddev->bitmap_info.file) {
                        struct file *f =
                        mddev->bitmap_info.file;
@@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,
 const char *buf, size_t len)
                         */
                        mddev->bitmap_id = ID_BITMAP;
                        mddev->bitmap_info.offset = offset;
-                       rv = md_bitmap_create(mddev, true);
+                       rv = md_bitmap_create(mddev, false);
                        if (rv)
                                goto out;
 
+                       rv = create_internal_group(mddev);
+                       if (rv) {
+                               mddev->bitmap_info.offset = 
 0;
+                               bitmap_destroy(mddev);
+                               goto out;
+                       }
+
                        rv = bitmap_load(mddev);
                        if (rv) {
+ 
 remove_internal_group(mddev);
                                mddev->bitmap_info.offset = 
                                0;
-                               md_bitmap_destroy(mddev, 

true);
+ md_bitmap_destroy(mddev,

false);
                                goto out;
                        }
                }
drivers/md/md-bitmap.c +0 -0
@@ -2960,8 +2982,7 @@ static struct md_sysfs_entry

max_backlog_used =
__ATTR(max_backlog_used, S_IRUGO | S_IWUSR,

        behind_writes_used_show, behind_writes_used_reset);
drivers/md/md-bitmap.c +18 -4
-static struct attribute *md_bitmap_attrs[] = {
-       &bitmap_location.attr,
+static struct attribute *internal_bitmap_attrs[] = {
        &bitmap_space.attr,
        &bitmap_timeout.attr,
        &bitmap_backlog.attr,
@@ -2972,11 +2993,57 @@ static struct attribute
 *md_bitmap_attrs[] = {
        NULL
 };
 
-static struct attribute_group md_bitmap_group = {
+static struct attribute_group internal_bitmap_group = {
        .name = "bitmap",
-       .attrs = md_bitmap_attrs,
+       .attrs = internal_bitmap_attrs,
 };
 
+/* Only necessary attrs for compatibility */
+static struct attribute *common_bitmap_attrs[] = {
+       &bitmap_location.attr,
+       NULL
+};
+
+static const struct attribute_group common_bitmap_group = {
+       .name = "bitmap",
+       .attrs = common_bitmap_attrs,
+};
+
+static int create_internal_group(struct mddev *mddev)
+{
+       /*
+        * md_bitmap_group and md_bitmap_common_group are 

using
same name

drivers/md/md-bitmap.c +3 -0
+        * 'bitmap'.
+        */
+       return sysfs_merge_group(&mddev->kobj,

&internal_bitmap_group);

drivers/md/md-bitmap.c +5 -0
+}
+
+static void remove_internal_group(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj,

&internal_bitmap_group);

drivers/md/md-bitmap.c +7 -0
+}
+
+static int bitmap_register_groups(struct mddev *mddev)
+{
+       int ret;
+
+       ret = sysfs_create_group(&mddev->kobj,

&common_bitmap_group);

drivers/md/md-bitmap.c +5 -0
+
+       if (ret)
+               return ret;
+
+       ret = sysfs_merge_group(&mddev->kobj,

&internal_bitmap_group);

drivers/md/md-bitmap.c +2 -0
+       if (ret)
+               sysfs_remove_group(&mddev->kobj,

&common_bitmap_group);

drivers/md/md-bitmap.c +7 -0
+
+       return ret;
+}
+
+static void bitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_unmerge_group(&mddev->kobj,
Quoted 2 lines

&internal_bitmap_group);
+}

Hi Su

Quoted 3 lines

bitmap_unregister_groups should also remove common_bitmap_group, right?

As you pasted before, mdadm:Grow.c:

int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
{
...

        if (array.state & (1 << MD_SB_BITMAP_PRESENT)) {
                if (s->btype == BitmapNone) {
                        array.state &= ~(1 << 
                        MD_SB_BITMAP_PRESENT);
                        if (md_set_array_info(fd, &array) != 0) 
                        {
                                if (array.state & (1 << 
                                MD_SB_CLUSTERED))
                                        pr_err("failed to 
                                        remove clustered
                                        bitmap.\n");
                                else
                                        pr_err("failed to 
                                        remove internal 
                                        bitmap.\n");
                                return 1;
                        }
                        return 0;
                }
                pr_err("bitmap already present on %s\n", 
                devname);
                return 1;
        }

...
}

Quoted 9 lines

In case of growing from internal to none, bitmap_unregister_groups() will
be called, if common_bitmap_group is removed, bitmap/location won't exist.
update_array_info() is a common function used by many call paths. Also
llbitmap is involved here. I don't want to make situation and code more
complicated like adding more codes in update_array_info().

Quoted 5 lines

The above mdadm codes use bitmap/location. In your patch, you already pass false to md_bitmap_destroy in location_store. So removing common_bitmap_group in bitmap_unregister_groups can't affect the above case.

Right. Got confusion between md_set_array_info() and update_array_info().

Quoted 4 lines

But after reading your below comments. It's good to me that bitmap_unregister_groups doesn't remove common_bitmap_group.

Thanks.

--
Su

Quoted 216 lines
Quoted 51 lines
Quoted 40 lines
Quoted 4 lines

+
static struct bitmap_operations bitmap_ops = {

You already changed md_bitmap_group to internal_bitmap_group. It's
better to change bitmap_ops to internal_bitmap_ops?

Quoted 31 lines

.head = {
.type = MD_BITMAP,

drivers/md/md-bitmap.c +22 -1
@@ -3018,21 +3085,52 @@ static struct bitmap_operations
 bitmap_ops = {
        .set_pages              = bitmap_set_pages,
        .free                   = md_bitmap_free,
 
-       .group                  = &md_bitmap_group,
+       .register_groups        = bitmap_register_groups,
+       .unregister_groups      = bitmap_unregister_groups,
+};
+
+static int none_bitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj,
 &common_bitmap_group);
+}
+
+static struct bitmap_operations none_bitmap_ops = {
+       .head = {
+               .type   = MD_BITMAP,
+               .id     = ID_BITMAP_NONE,
+               .name   = "none",
+       },
+
+       .enabled                = dummy_bitmap_enabled,
+       .create                 = dummy_bitmap_create,
+       .destroy                = bitmap_destroy,
+       .load                   = bitmap_load,
+       .get_stats              = bitmap_get_stats,
+       .free                   = md_bitmap_free,
Quoted 9 lines

drivers/md/md-bitmap.c static struct bitmap_operations bitmap_ops = { · line 3112

Quoted 4 lines
drivers/md/md-bitmap.c +3 -0
+
+       .register_groups        = 
 none_bitmap_register_groups,
+       .unregister_groups      = NULL,

How does bitmap/location can be deleted if array is created with no
bitmap? Can the mdadm --stop command get stuck?

No. It wont get stuck.

Quoted 10 lines

I guess here your concern is the timing of removing common_bitmap_group.
The life cycle is same as before fb8cc3b0d9db. At the time llbitmap is
not introduced, because there is no need to switch bitmap ops, so all
attrs of bitmap are removed in kobject_put() in mddev_delayed_delete()
queued by mddev_put(). This is now where common_bitmap_group is being removed.

Thanks for the explanation.

Best Regards
Xiao

Quoted 150 lines

--
Su

Quoted 146 lines

Best Regards
Xiao

Quoted 143 lines
drivers/md/md-bitmap.c +8 -1
 };
 
 int md_bitmap_init(void)
 {
+       int ret;
+
        md_bitmap_wq = alloc_workqueue("md_bitmap",
        WQ_MEM_RECLAIM | WQ_UNBOUND,
                                       0);
        if (!md_bitmap_wq)
                return -ENOMEM;
 
-       return register_md_submodule(&bitmap_ops.head);
+       ret = register_md_submodule(&bitmap_ops.head);
+       if (ret)
+               return ret;
+
+       return register_md_submodule(&none_bitmap_ops.head);
 }
 
 void md_bitmap_exit(void)
 {
        destroy_workqueue(md_bitmap_wq);
        unregister_md_submodule(&bitmap_ops.head);
+       unregister_md_submodule(&none_bitmap_ops.head);
 }
drivers/md/md-bitmap.h +3 -0
index b42a28fa83a0..10bc6854798c 100644
@@ -125,6 +125,9 @@ struct bitmap_operations {
        void (*set_pages)(void *data, unsigned long pages);
        void (*free)(void *data);
 
+       int (*register_groups)(struct mddev *mddev);
+       void (*unregister_groups)(struct mddev *mddev);
+
        struct attribute_group *group;
 };
 
drivers/md/md-llbitmap.c +10 -0
index bf398d7476b3..9b3ea4f1d268 100644
@@ -1561,6 +1561,16 @@ static struct attribute_group
 md_llbitmap_group = {
        .attrs = md_llbitmap_attrs,
 };
 
+static int llbitmap_register_groups(struct mddev *mddev)
+{
+       return sysfs_create_group(&mddev->kobj,
 &md_llbitmap_group);
+}
+
+static void llbitmap_unregister_groups(struct mddev *mddev)
+{
+       sysfs_remove_group(&mddev->kobj, 
 &md_llbitmap_group);
+}
+

static struct bitmap_operations llbitmap_ops = {
.head = {
.type = MD_BITMAP,

drivers/md/md-llbitmap.c +2 -0
@@ -1597,6 +1607,8 @@ static struct bitmap_operations
 llbitmap_ops = {
        .dirty_bits             = llbitmap_dirty_bits,
        .write_all              = llbitmap_write_all,
 
+       .register_groups        = llbitmap_register_groups,
+       .unregister_groups      = 
 llbitmap_unregister_groups,
        .group                  = &md_llbitmap_group,

};

drivers/md/md.c +1 -2
index d3c8f77b4fe3..55a95b227b83 100644
@@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct
 mddev *mddev, bool create_sysfs)
        struct bitmap_operations *old = mddev->bitmap_ops;
        struct md_submodule_head *head;
 
-       if (mddev->bitmap_id == ID_BITMAP_NONE ||
-           (old && old->head.id == mddev->bitmap_id))
+       if (old && old->head.id == mddev->bitmap_id)
                return true;
 
        xa_lock(&md_submodule);
drivers/md/md.c +1 -2
@@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct
 mddev *mddev, bool create_sysfs)
        mddev->bitmap_ops = (void *)head;
        xa_unlock(&md_submodule);
 
-       if (create_sysfs && !mddev_is_dm(mddev) &&
 mddev->bitmap_ops->group) {
-               if (sysfs_create_group(&mddev->kobj,
 mddev->bitmap_ops->group))
+       if (create_sysfs && !mddev_is_dm(mddev) &&

mddev->bitmap_ops->register_groups) {
+ if
(mddev->bitmap_ops->register_groups(mddev))
pr_warn("md: cannot register extra bitmap attributes for %s\n",
mdname(mddev));
else

drivers/md/md.c +0 -2
@@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct
 mddev *mddev, bool create_sysfs)
 static void mddev_clear_bitmap_ops(struct mddev *mddev, 
 bool
 remove_sysfs)
 {
        if (remove_sysfs && !mddev_is_dm(mddev) &&
        mddev->bitmap_ops &&
-           mddev->bitmap_ops->group)
-               sysfs_remove_group(&mddev->kobj,

mddev->bitmap_ops->group);

drivers/md/md.c +5 -2
+           mddev->bitmap_ops->unregister_groups)
+               mddev->bitmap_ops->unregister_groups(mddev);
 
        mddev->bitmap_ops = NULL;
 }
@@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
                        (unsigned long 
                        long)pers->size(mddev,
                        0, 0) / 2);
                err = -EINVAL;
        }
-       if (err == 0 && pers->sync_request &&
-           (mddev->bitmap_info.file ||
 mddev->bitmap_info.offset)) {
+       if (err == 0 && pers->sync_request) {
+               if (mddev->bitmap_info.offset == 0)
+                       mddev->bitmap_id = ID_BITMAP_NONE;
                err = md_bitmap_create(mddev, true);
                if (err)
                        pr_warn("%s: failed to create bitmap
                        (%d)\n",

--
2.53.0

author
#15 lore raw

PATCH v2 4/5 md: skip ID_BITMAP_NONE when show available bitmap types

In reply to Su Yue

As none_bitmap_ops is introduced, ID_BITMAP_NONE should be skipped while iterating md submodules, otherwise:

$ cat /sys/block/md0/md/bitmap_type
[none] bitmap llbitmap [none]
---
 drivers/md/md.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/md/md.c +2 -0
index 55a95b227b83..20a953676319 100644
@@ -4269,6 +4269,8 @@ bitmap_type_show(struct mddev *mddev, char *page)
 	xa_for_each(&md_submodule, i, head) {
 		if (head->type != MD_BITMAP)
 			continue;
+		if (head->id == ID_BITMAP_NONE)
+			continue;
 
 		if (mddev->bitmap_id == head->id)
 			len += sprintf(page + len, "[%s] ", head->name);
Signature · 1 line
2.53.0
#16 lore raw

Re: [PATCH v2 4/5] md: skip ID_BITMAP_NONE when show available bitmap types

Quoted 16 lines · Su Yue

As none_bitmap_ops is introduced, ID_BITMAP_NONE should be skipped while iterating md submodules, otherwise:

$ cat /sys/block/md0/md/bitmap_type
[none] bitmap llbitmap [none]
---
  drivers/md/md.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/md/md.c +0 -0
index 55a95b227b83..20a953676319 100644

drivers/md/md.c bitmap_type_show(struct mddev *mddev, char *page) · line 4276

Quoted 8 lines
drivers/md/md.c +2 -0
@@ -4269,6 +4269,8 @@ bitmap_type_show(struct mddev *mddev, char *page)
  	xa_for_each(&md_submodule, i, head) {
  		if (head->type != MD_BITMAP)
  			continue;
+		if (head->id == ID_BITMAP_NONE)
+			continue;
 
  		if (mddev->bitmap_id == head->id)
  			len += sprintf(page + len, "[%s] ", head->name);

If we indeed add ID_BITMAP_NONE as patch 3, would it be better to delete this piece of code instead?

        if (mddev->bitmap_id == ID_BITMAP_NONE)
                len += sprintf(page + len, "[none] ");
        else
                len += sprintf(page + len, "none ");
Signature · 2 lines
Thanks,
Nan
#17 lore raw

Re: [PATCH v2 4/5] md: skip ID_BITMAP_NONE when show available bitmap types

Quoted 17 lines · Li Nan
Quoted 16 lines · Su Yue

As none_bitmap_ops is introduced, ID_BITMAP_NONE should be skipped while iterating md submodules, otherwise:

$ cat /sys/block/md0/md/bitmap_type
[none] bitmap llbitmap [none]
Signed-off-by: Su Yue <[email protected]>
---
  drivers/md/md.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/md/md.c +0 -0
index 55a95b227b83..20a953676319 100644
@@ -4269,6 +4269,8 @@ bitmap_type_show(struct mddev *mddev, 
 char *page)
  	xa_for_each(&md_submodule, i, head) {
  		if (head->type != MD_BITMAP)
Quoted 9 lines

drivers/md/md.c bitmap_type_show(struct mddev *mddev, char *page) · line 4276

Quoted 5 lines
drivers/md/md.c +2 -0
  			continue;
+		if (head->id == ID_BITMAP_NONE)
+			continue;
    		if (mddev->bitmap_id == head->id)
  			len += sprintf(page + len, "[%s] ", head->name);

If we indeed add ID_BITMAP_NONE as patch 3, would it be better to delete this piece of code instead?

I dont' think so. There is one case CONFIG_MD_BITMAP is not enabled which means register_md_submodule(&none_bitmap_ops.head) is not called but in mddev_init():

int mddev_init(struct mddev *mddev) |
{ |

        int err = 0; 
        |
                                                                                |
        if (!IS_ENABLED(CONFIG_MD_BITMAP)) 
        |
                mddev->bitmap_id = ID_BITMAP_NONE;

...
}

--
Su

Quoted 5 lines
        if (mddev->bitmap_id == ID_BITMAP_NONE)
                len += sprintf(page + len, "[none] ");
        else
                len += sprintf(page + len, "none ");
author
#18 lore raw

PATCH v2 5/5 md/md-bitmap: remove member group from bitmap_operations

In reply to Su Yue

The member is unused now so remove it.

---
 drivers/md/md-bitmap.h   | 2 --
 drivers/md/md-llbitmap.c | 1 -
 2 files changed, 3 deletions(-)
drivers/md/md-bitmap.h +0 -2
index 10bc6854798c..7fba97a09b66 100644
@@ -127,8 +127,6 @@ struct bitmap_operations {
 
 	int (*register_groups)(struct mddev *mddev);
 	void (*unregister_groups)(struct mddev *mddev);
-
-	struct attribute_group *group;
 };
 
 /* the bitmap API */
drivers/md/md-llbitmap.c +0 -1
index 9b3ea4f1d268..4fb9003cdc49 100644
@@ -1609,7 +1609,6 @@ static struct bitmap_operations llbitmap_ops = {
 
 	.register_groups	= llbitmap_register_groups,
 	.unregister_groups	= llbitmap_unregister_groups,
-	.group			= &md_llbitmap_group,
 };
 
 int md_llbitmap_init(void)
Signature · 1 line
2.53.0
#19 lore raw
In reply to Su Yue
Quoted 14 lines · Su Yue

Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.

The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.

v2:
Add a dummy bitmap operations per Kuai's suggestion.

To Yu Kuai:
A NULL group can't used for internal_bitmap_group since
the entries from internal_bitmap_attrs should be under bitmap.
So instead of sysfs_create_groups(), sysfs_create_group() and sysfs_merge_group() are still needed /sigh.

Quoted 10 lines

Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy in location_store md/md-bitmap: add an extra sysfs argument to md_bitmap_create and
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skip ID_BITMAP_NONE when show available bitmap types md/md-bitmap: remove member group from bitmap_operations

Missing part should be added in v3:

drivers/md/md.c +0 -0
index 20a953676319..453b89941db2 100644
@@ -6450,9 +6450,6 @@ static int start_dirty_degraded;
 

int md_bitmap_create(struct mddev *mddev, bool create_sysfs)
{
- if (mddev->bitmap_id == ID_BITMAP_NONE)
- return -EINVAL;
-
}

Quoted 7 lines
 drivers/md/md-bitmap.c   | 121 
 ++++++++++++++++++++++++++++++++++++---
 drivers/md/md-bitmap.h   |   3 +-
 drivers/md/md-llbitmap.c |  13 ++++-
 drivers/md/md.c          |  55 +++++++++---------
 drivers/md/md.h          |   2 +
 5 files changed, 155 insertions(+), 39 deletions(-)
#20 lore raw
In reply to Su Yue

Hi,

Quoted 5 lines · Su Yue

Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.

The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.

Quoted 8 lines

v2:
Add a dummy bitmap operations per Kuai's suggestion.

To Yu Kuai:
A NULL group can't used for internal_bitmap_group since
the entries from internal_bitmap_attrs should be under bitmap.
So instead of sysfs_create_groups(), sysfs_create_group() and sysfs_merge_group() are still needed /sigh.

Thanks for the set and sorry for the delay.

TBO, this set is still a bit complex than expected. It's true sysfs_create_groups() can't be used for the groups with same name. However, there is still another api sysfs_update_groups() that can be used in this case.

I just finish a local version and confirm following 3 commits should be enough:

1) factor out bitmap sysfs creation from mddev_set_bitmap_ops(), and create sysfs entries after bitmap is created.
2) split bitmap/location out as the common bitmap groups, and convert bitmap filed groups to struct attribute_group **groups;
3) add a separate bitmap_ops for none bitmap, location_store() can be similar to patch 3 in your set, just create/remove additional internal_bitmap_group.

Quoted 15 lines

Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy in location_store md/md-bitmap: add an extra sysfs argument to md_bitmap_create and
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skip ID_BITMAP_NONE when show available bitmap types md/md-bitmap: remove member group from bitmap_operations

  drivers/md/md-bitmap.c   | 121 ++++++++++++++++++++++++++++++++++++---
  drivers/md/md-bitmap.h   |   3 +-
  drivers/md/md-llbitmap.c |  13 ++++-
  drivers/md/md.c          |  55 +++++++++---------
  drivers/md/md.h          |   2 +
  5 files changed, 155 insertions(+), 39 deletions(-)
Signature · 2 lines
Thansk,
Kuai
#21 lore raw
Quoted 17 lines · Yu Kuai

Hi,

Quoted 5 lines · Su Yue

Hi, This v2 series is to fixes bugs exposed by mdadm/clustermd-autotest. The bugs are not cluster md only but also bitmap related.

The series based on v7.0-rc7 passes tests with mdadm v4.6 without regression.

Quoted 8 lines

v2:
Add a dummy bitmap operations per Kuai's suggestion.

To Yu Kuai:
A NULL group can't used for internal_bitmap_group since
the entries from internal_bitmap_attrs should be under bitmap.
So instead of sysfs_create_groups(), sysfs_create_group() and sysfs_merge_group() are still needed /sigh.

Quoted 6 lines

Thanks for the set and sorry for the delay.

TBO, this set is still a bit complex than expected. It's true sysfs_create_groups() can't be used for the groups with same name. However, there is still another api sysfs_update_groups() that can be used in this case.

IIUC, sysfs_update_groups() can't be used because it first removes then re-add files see linux/fs/sysfs/group.c:create_files() for details.

Quoted 3 lines

I just finish a local version and confirm following 3 commits should be enough:

If you would like to send your own patchset. I am happy to give up my patchset then review yours :) The only thing I want to fix the issue. Thanks.

--
Su

Quoted 26 lines

1) factor out bitmap sysfs creation from mddev_set_bitmap_ops(), and create sysfs entries after bitmap is created.
2) split bitmap/location out as the common bitmap groups, and convert bitmap filed groups to struct attribute_group **groups;
3) add a separate bitmap_ops for none bitmap, location_store() can be similar to patch 3 in your set, just create/remove additional internal_bitmap_group.

Quoted 19 lines

Su Yue (5):
md/md-bitmap: call md_bitmap_create,destroy in location_store
md/md-bitmap: add an extra sysfs argument to md_bitmap_create and
destroy
md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
md: skip ID_BITMAP_NONE when show available bitmap types md/md-bitmap: remove member group from bitmap_operations

  drivers/md/md-bitmap.c   | 121 
  ++++++++++++++++++++++++++++++++++++---
  drivers/md/md-bitmap.h   |   3 +-
  drivers/md/md-llbitmap.c |  13 ++++-
  drivers/md/md.c          |  55 +++++++++---------
  drivers/md/md.h          |   2 +
  5 files changed, 155 insertions(+), 39 deletions(-)