dm raid: ensure metadata IO matches device block size.

PATCH In review NeilBrown 3 participants 7 messages Oct 15, 2014 – Oct 16, 2014 Device mapper

Share & tools

NeilBrown

author

2014-10-15 01:19 UTC #0 lore raw

PATCH 1/1 dm raid: ensure metadata IO matches device block size.

dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.

If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.

Reported-by "Liuhua Wang" <[email protected]>

Signed-off-by NeilBrown <[email protected]>

---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.

Thanks,
NeilBrown

drivers/md/dm-raid.c +2 -1

index 4880b69e2e9e..31bdd73bc368 100644
@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
 	uint64_t events_sb, events_refsb;
 
 	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 
 	ret = read_disk_sb(rdev, rdev->sb_size);
 	if (ret)

Mike Snitzer

2014-10-15 02:55 UTC #1 lore raw

Quoted 26 lines · NeilBrown

dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.
If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.
Reported-by "Liuhua Wang" <[email protected]>
Signed-off-by NeilBrown <[email protected]>
---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.
Thanks,
NeilBrown
drivers/md/dm-raid.c +0 -0
index 4880b69e2e9e..31bdd73bc368 100644

drivers/md/dm-raid.c static int super_load(struct md_rdev *rdev, struct md_rdev *refdev) · line 865

Quoted 9 lines

drivers/md/dm-raid.c +2 -1

@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
 	uint64_t events_sb, events_refsb;
 
 	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 
 	ret = read_disk_sb(rdev, rdev->sb_size);
 	if (ret)

Wouldn't it be better to use bdev_physical_block_size()?

Even on a 4K device that emulates 512b logical sectors it is better to use the physical block size (4K).

NeilBrown

author

2014-10-15 03:40 UTC #2 lore raw

Quoted 36 lines · On Tue, 14 Oct 2014 22:55:50 -0400 Mike Snitzer <[email protected]> wrote:

Quoted 33 lines · NeilBrown
dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.
If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.
Reported-by "Liuhua Wang" <[email protected]>
Signed-off-by NeilBrown <[email protected]>
---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.
Thanks,
NeilBrown
drivers/md/dm-raid.c +2 -1
index 4880b69e2e9e..31bdd73bc368 100644
@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
 	uint64_t events_sb, events_refsb;
 
 	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 

Quoted 7 lines

drivers/md/dm-raid.c static int super_load(struct md_rdev *rdev, struct md_rdev *refdev) · line 865
Quoted 2 lines
drivers/md/dm-raid.c +0 -0
 	ret = read_disk_sb(rdev, rdev->sb_size);
 	if (ret)
Wouldn't it be better to use bdev_physical_block_size()?
Even on a 4K device that emulates 512b logical sectors it is better to use the physical block size (4K).

_logical_ is the smallest value for which the IO actually works. And the goal of the change is to make it work.

I don't object to using _physical_, but it isn't clear to me how I would justify that as "correct".

A big question in my mind is: how much space does LVM reserve in this device for the metadata? It seems reasonable to assume that it reserves at least 1 logical block. If the API guarantees that at least one physical block is reserved, then that would justify using _physical_.

A quick look at the code shows that the bitmap superblock is placed 4K after the start of the metadata.
So the code should probably fail if the rounded-up sb_size exceeds 4K. Mind you, that would exceed PAGE_SIZE too which would cause other problems.

Maybe use _physical_ unless that exceeds 4K, then try _logical_, then fail if even that > 4K ??

Thanks,
NeilBrown

Mike Snitzer

2014-10-15 13:13 UTC #3 lore raw

Quoted 50 lines · NeilBrown

Quoted 36 lines · On Tue, 14 Oct 2014 22:55:50 -0400 Mike Snitzer <[email protected]> wrote:
Quoted 33 lines · NeilBrown
dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.
If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.
Reported-by "Liuhua Wang" <[email protected]>
Signed-off-by NeilBrown <[email protected]>
---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.
Thanks,
NeilBrown
drivers/md/dm-raid.c +2 -1
index 4880b69e2e9e..31bdd73bc368 100644
@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
 	uint64_t events_sb, events_refsb;
 
 	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 
Quoted 7 lines
drivers/md/dm-raid.c static int super_load(struct md_rdev *rdev, struct md_rdev *refdev) · line 865
Quoted 2 lines
drivers/md/dm-raid.c +0 -0
 	ret = read_disk_sb(rdev, rdev->sb_size);
 	if (ret)
Wouldn't it be better to use bdev_physical_block_size()?
Even on a 4K device that emulates 512b logical sectors it is better to use the physical block size (4K).
_logical_ is the smallest value for which the IO actually works. And the goal of the change is to make it work.

Quoted 7 lines

I don't object to using _physical_, but it isn't clear to me how I would justify that as "correct".
A big question in my mind is: how much space does LVM reserve in this device for the metadata? It seems reasonable to assume that it reserves at least 1 logical block. If the API guarantees that at least one physical block is reserved, then that would justify using _physical_.

I'll have to check with Jon and/or Heinz on this point.

Quoted 2 lines

A quick look at the code shows that the bitmap superblock is placed 4K after the start of the metadata.

"the code" being the MD kernel code right? Any reason not to export a #define that reflects the space MD reserves and just have dm-raid use that?

Starting to feel like hardcoding 4K is the right thing to do given the current code.

NeilBrown

author

2014-10-15 21:00 UTC #4 lore raw

Quoted 61 lines · On Wed, 15 Oct 2014 09:13:08 -0400 Mike Snitzer <[email protected]> wrote:

Quoted 50 lines · NeilBrown
Quoted 36 lines · On Tue, 14 Oct 2014 22:55:50 -0400 Mike Snitzer <[email protected]> wrote:
Quoted 33 lines · NeilBrown
dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.
If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.
Reported-by "Liuhua Wang" <[email protected]>
Signed-off-by NeilBrown <[email protected]>
---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.
Thanks,
NeilBrown
drivers/md/dm-raid.c +2 -1
index 4880b69e2e9e..31bdd73bc368 100644
@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
 	uint64_t events_sb, events_refsb;
 
 	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 
Quoted 7 lines
drivers/md/dm-raid.c static int super_load(struct md_rdev *rdev, struct md_rdev *refdev) · line 865
Quoted 2 lines
drivers/md/dm-raid.c +0 -0
 	ret = read_disk_sb(rdev, rdev->sb_size);
 	if (ret)
Wouldn't it be better to use bdev_physical_block_size()?
Even on a 4K device that emulates 512b logical sectors it is better to use the physical block size (4K).
_logical_ is the smallest value for which the IO actually works. And the goal of the change is to make it work.
Quoted 7 lines
I don't object to using _physical_, but it isn't clear to me how I would justify that as "correct".
A big question in my mind is: how much space does LVM reserve in this device for the metadata? It seems reasonable to assume that it reserves at least 1 logical block. If the API guarantees that at least one physical block is reserved, then that would justify using _physical_.

Quoted 7 lines

I'll have to check with Jon and/or Heinz on this point.
Quoted 2 lines
A quick look at the code shows that the bitmap superblock is placed 4K after the start of the metadata.
"the code" being the MD kernel code right? Any reason not to export a #define that reflects the space MD reserves and just have dm-raid use that?

No, "the code" being

	mddev->bitmap_info.offset = 4096 >> 9; /* Enable bitmap creation */
	rdev->mddev->bitmap_info.default_offset = 4096 >> 9;

in super_validate in dm-raid.c.
i.e. dm-raid specific code.
md doesn't reserve space, it just uses what it is told to. Told either by mdadm via the md superblock or by dm-raid.

NeilBrown

Quoted 3 lines

Starting to feel like hardcoding 4K is the right thing to do given the current code.

Heinz Mauelshagen

2014-10-16 13:31 UTC #5 lore raw

Quoted 52 lines · NeilBrown

Quoted 50 lines · On Wed, 15 Oct 2014 09:13:08 -0400 Mike Snitzer <[email protected]> wrote:
Quoted 47 lines · NeilBrown
Quoted 35 lines · On Tue, 14 Oct 2014 22:55:50 -0400 Mike Snitzer <[email protected]> wrote:
Quoted 32 lines · NeilBrown
dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.
If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.
Reported-by "Liuhua Wang" <[email protected]>
Signed-off-by NeilBrown <[email protected]>
---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.
Thanks,
NeilBrown
drivers/md/dm-raid.c +2 -1
index 4880b69e2e9e..31bdd73bc368 100644
@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
  	uint64_t events_sb, events_refsb;
 
  	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 
Quoted 6 lines
drivers/md/dm-raid.c static int super_load(struct md_rdev *rdev, struct md_rdev *refdev) · line 865
Quoted 2 lines
drivers/md/dm-raid.c +0 -0
  	ret = read_disk_sb(rdev, rdev->sb_size);
  	if (ret)
Wouldn't it be better to use bdev_physical_block_size()?
Even on a 4K device that emulates 512b logical sectors it is better to use the physical block size (4K).
_logical_ is the smallest value for which the IO actually works. And the goal of the change is to make it work.

Quoted 7 lines

Quoted 7 lines
Quoted 7 lines
I don't object to using _physical_, but it isn't clear to me how I would justify that as "correct".
A big question in my mind is: how much space does LVM reserve in this device for the metadata? It seems reasonable to assume that it reserves at least 1 logical block. If the API guarantees that at least one physical block is reserved, then that would justify using _physical_.

dm-raid uses 512 bytes for its superblock including padding in the current code (with way less payload), followed by the bitmap at offset 4096 bytes.

With Neil's proposal, the padding will be avoided alltogether.

So logical looks fine to me, because physical may well be more than 4K in the future. Admittingly a longer time in the future though.

If that happens, we'd end up with the bitmap start at offset 4096 bytes in the same physical sector and would have to cope with it plus the issues involved with 4K page sizes.

In order to prevent this to occur unrecognized in dm/md later,
we should have an "if (rdev->sb_size > PAGE_SIZE) return -EINVAL;" after the setting of rdev->sb_size in dm-raid.c and appropriate checks in md.c.

BTW: even with additions to the dm-raid superblock I have to make in order to allow for reshaping etc.,
it's paylod is going to stay < 512 bytes.

Heinz

Quoted 26 lines

Quoted 6 lines
I'll have to check with Jon and/or Heinz on this point.
Quoted 2 lines
A quick look at the code shows that the bitmap superblock is placed 4K after the start of the metadata.
"the code" being the MD kernel code right? Any reason not to export a #define that reflects the space MD reserves and just have dm-raid use that?
No, "the code" being
	mddev->bitmap_info.offset = 4096 >> 9; /* Enable bitmap creation */
	rdev->mddev->bitmap_info.default_offset = 4096 >> 9;
in super_validate in dm-raid.c.
i.e. dm-raid specific code.
md doesn't reserve space, it just uses what it is told to. Told either by mdadm via the md superblock or by dm-raid.
NeilBrown
Quoted 2 lines
Starting to feel like hardcoding 4K is the right thing to do given the current code.
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

Mike Snitzer

2014-10-16 19:56 UTC #6 lore raw

Quoted 74 lines · Heinz Mauelshagen

Quoted 50 lines · NeilBrown
Quoted 48 lines · On Wed, 15 Oct 2014 09:13:08 -0400 Mike Snitzer <[email protected]> wrote:
Quoted 45 lines · NeilBrown
Quoted 37 lines · On Tue, 14 Oct 2014 22:55:50 -0400 Mike Snitzer <[email protected]> wrote:
Quoted 25 lines · NeilBrown
dm_raid_superblock is 512.
Reading or writing this on a 512-byte sector works fine. On a 4096-byte sector device, this fails.
If we round up rdev->sb_size to match the block size of the device, all IO will work correctly.
Reported-by "Liuhua Wang" <[email protected]>
Signed-off-by NeilBrown <[email protected]>
---
this issue has been discussed already a bit. See email thread
Subject: Re: [dm-devel] [PATCH] fix mirror device creation with lvcreate failed
I think this is the best fix. It handles boths read and writes, and (I think) at the best level.
Thanks,
NeilBrown
drivers/md/dm-raid.c +0 -0
index 4880b69e2e9e..31bdd73bc368 100644
drivers/md/dm-raid.c static int super_load(struct md_rdev *rdev, struct md_rdev *refdev) · line 863
Quoted 7 lines
drivers/md/dm-raid.c +2 -1
@@ -858,7 +858,8 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)
 	uint64_t events_sb, events_refsb;
 	rdev->sb_start = 0;
-	rdev->sb_size = sizeof(*sb);
+	rdev->sb_size = roundup(sizeof(*sb),
+				bdev_logical_block_size(rdev->meta_bdev));
 	ret = read_disk_sb(rdev, rdev->sb_size);
 	if (ret)
Wouldn't it be better to use bdev_physical_block_size()?
Quoted 2 lines
Even on a 4K device that emulates 512b logical sectors it is better to use the physical block size (4K).
_logical_ is the smallest value for which the IO actually works. And the goal of the change is to make it work.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I don't object to using _physical_, but it isn't clear to me how I would justify that as "correct".
A big question in my mind is: how much space does LVM reserve in this device for the metadata? It seems reasonable to assume that it reserves at least 1 logical block. If the API guarantees that at least one physical block is reserved, then that would justify using _physical_.
dm-raid uses 512 bytes for its superblock including padding in the current code
(with way less payload), followed by the bitmap at offset 4096 bytes.
With Neil's proposal, the padding will be avoided alltogether.
So logical looks fine to me, because physical may well be more than 4K in the future.
Admittingly a longer time in the future though.
If that happens, we'd end up with the bitmap start at offset 4096 bytes in the same physical sector
and would have to cope with it plus the issues involved with 4K page sizes.

Quoted 8 lines

In order to prevent this to occur unrecognized in dm/md later, we should have an "if (rdev->sb_size > PAGE_SIZE) return -EINVAL;" after the setting
of rdev->sb_size in dm-raid.c and appropriate checks in md.c.
BTW: even with additions to the dm-raid superblock I have to make in order to allow for reshaping etc.,
it's paylod is going to stay < 512 bytes.

Heinz,

If you could propose a revised patch I'd appreciate it.

Thanks,
Mike