Skip to contentSkip to conversation

btrfs zoned fixlets

PATCH In review Christoph Hellwig 4 participants 17 messages Mar 24, 2022 – Apr 11, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi all,

this series fixes a minor and slightly less minor problem in the btrfs zoned device code. Note that for the second patch the comment might not be correct any more - AFAICS 5.18 added support for the dup profile for zoned devices, which means we do have a real issue now if different devices have different hardware limitations. I think we'll need some code to check that all zoned devices have the exact same hardware limits (max_sectors, max_segments, max_segment_size, queue_boundary, virt_boundary), but I don't know the code well enough to implement that myself

Found by code inspection as part of my bio cleanups.

#1 lore raw

PATCH 1/2 btrfs: remove the zoned/zone_size union in struct btrfs_fs_info

Reading a value from a different member of a union is not just a great way to obsfucate code, but also creates an aliasing violation. Switch btrfs_is_zoned to look at ->zone_size and remove the union.

---
 fs/btrfs/ctree.h | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
fs/btrfs/ctree.h +2 -5
index b7631b88426e3..0edcce6d2db64 100644
@@ -1045,10 +1045,7 @@ struct btrfs_fs_info {
 	 * Zone size > 0 when in ZONED mode, otherwise it's used for a check
 	 * if the mode is enabled
 	 */
-	union {
-		u64 zone_size;
-		u64 zoned;
-	};
+	u64 zone_size;
 
 	struct mutex zoned_meta_io_lock;
 	spinlock_t treelog_bg_lock;
@@ -3928,7 +3925,7 @@ static inline int btrfs_is_testing(struct btrfs_fs_info *fs_info)
 
 static inline bool btrfs_is_zoned(const struct btrfs_fs_info *fs_info)
 {
-	return fs_info->zoned != 0;
+	return fs_info->zone_size > 0;
 }
 
 static inline bool btrfs_is_data_reloc_root(const struct btrfs_root *root)
Signature · 1 line
2.30.2
#3 lore raw

Re: [PATCH 1/2] btrfs: remove the zoned/zone_size union in struct btrfs_fs_info

Quoted 1 line · Johannes Thumshirn

Fine by me, but it was explicitly requested to be this way IIRC.

This was before there was the btrfs_is_zoned helper that now wraps the condition.

Reviewed-by on v1 1/2
#4 lore raw

Re: [PATCH 1/2] btrfs: remove the zoned/zone_size union in struct btrfs_fs_info

In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

Reading a value from a different member of a union is not just a great way to obsfucate code, but also creates an aliasing violation. Switch btrfs_is_zoned to look at ->zone_size and remove the union.

Also, fine for me.

#5 lore raw

Re: [PATCH 1/2] btrfs: remove the zoned/zone_size union in struct btrfs_fs_info

In reply to Christoph Hellwig
Quoted 2 lines · Christoph Hellwig

Reading a value from a different member of a union is not just a great way to obsfucate code, but also creates an aliasing violation.

Is it a violation in this case? Both are of the same type, violation could be int/u64 and others but even in this case this is transparent to the compiler and union cast is the cleanest way to access same bytes in a structure. Anaywy now the helper btrfs_is_zoned is used everywhere which abstracts the condition, which was the original idea for the union.

#6 lore raw

PATCH 2/2 btrfs: fix and document the zoned device choice in alloc_new_bio

In reply to Christoph Hellwig

Zone Append bios only need a valid block device in struct bio, but not the device in the btrfs_bio. Use the information from btrfs_zoned_get_device to set up bi_bdev and fix zoned writes on multi-device file system with non-homogeneous capabilities and remove the pointless btrfs_bio.device assignment.

Add big fat comments explaining what is going on here.

---
 fs/btrfs/extent_io.c | 43 ++++++++++++++++++++++++++++---------------
 1 file changed, 28 insertions(+), 15 deletions(-)
fs/btrfs/extent_io.c +28 -15
index d78b3a2d04e3b..175a3f16c2da0 100644
@@ -3334,24 +3334,37 @@ static int alloc_new_bio(struct btrfs_inode *inode,
 	ret = calc_bio_boundaries(bio_ctrl, inode, file_offset);
 	if (ret < 0)
 		goto error;
-	if (wbc) {
-		struct block_device *bdev;
 
-		bdev = fs_info->fs_devices->latest_dev->bdev;
-		bio_set_dev(bio, bdev);
-		wbc_init_bio(wbc, bio);
-	}
-	if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
-		struct btrfs_device *device;
+	if (wbc) {
+		/*
+		 * For Zone append we need the correct block_device that we are
+		 * going to write to set in the bio to be able to respect the
+		 * hardware limitation.  Look it up here:
+		 */
+		if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+			struct btrfs_device *dev;
+
+			dev = btrfs_zoned_get_device(fs_info, disk_bytenr,
+						     fs_info->sectorsize);
+			if (IS_ERR(dev)) {
+				ret = PTR_ERR(dev);
+				goto error;
+			}
 
-		device = btrfs_zoned_get_device(fs_info, disk_bytenr,
-						fs_info->sectorsize);
-		if (IS_ERR(device)) {
-			ret = PTR_ERR(device);
-			goto error;
+			bio_set_dev(bio, dev->bdev);
+		} else {
+			/*
+			 * Otherwise pick the last added device to support
+			 * cgroup writeback.  For multi-device file systems this
+			 * means blk-cgroup policies have to always be set on the
+			 * last added/replaced device.  This is a bit odd but has
+			 * been like that for a long time.
+			 */
+			bio_set_dev(bio, fs_info->fs_devices->latest_dev->bdev);
 		}
-
-		btrfs_bio(bio)->device = device;
+		wbc_init_bio(wbc, bio);
+	} else {
+		ASSERT(bio_op(bio) != REQ_OP_ZONE_APPEND);
 	}
 	return 0;
 error:
Signature · 1 line
2.30.2
Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] btrfs: fix and document the zoned device choice in alloc_new_bio

Quoted 7 lines · Christoph Hellwig

Zone Append bios only need a valid block device in struct bio, but not the device in the btrfs_bio. Use the information from btrfs_zoned_get_device to set up bi_bdev and fix zoned writes on multi-device file system with non-homogeneous capabilities and remove the pointless btrfs_bio.device assignment.

Add big fat comments explaining what is going on here.

Looks like the old code worked by sheer luck, as we had wbc set and thus always assigned fs_info->fs_devices->latest_dev->bdev to the bio. Which would obviously not work on a multi device FS.

#8 lore raw

Re: [PATCH 2/2] btrfs: fix and document the zoned device choice in alloc_new_bio

Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig

Zone Append bios only need a valid block device in struct bio, but not the device in the btrfs_bio. Use the information from btrfs_zoned_get_device to set up bi_bdev and fix zoned writes on multi-device file system with non-homogeneous capabilities and remove the pointless btrfs_bio.device assignment.

Quoted 5 lines
Quoted 1 line

Add big fat comments explaining what is going on here.

Looks like the old code worked by sheer luck, as we had wbc set and thus always assigned fs_info->fs_devices->latest_dev->bdev to the bio. Which would obviously not work on a multi device FS.

No, it worked fine because the real bio is set just before writing the data somewhere deep in the io submit path in submit_stripe_bio().

That it has to be set here is because of the cgroup implementation that accesses it, see 429aebc0a9a0 ("btrfs: get bdev directly from fs_devices in submit_extent_page").

Which brings me to the question if Christoph's fix is correct because the comment for the wbc + zoned append is assuming something that's not true.

#9 lore raw

Re: [PATCH 2/2] btrfs: fix and document the zoned device choice in alloc_new_bio

Quoted 17 lines · David Sterba
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig

Zone Append bios only need a valid block device in struct bio, but not the device in the btrfs_bio. Use the information from btrfs_zoned_get_device to set up bi_bdev and fix zoned writes on multi-device file system with non-homogeneous capabilities and remove the pointless btrfs_bio.device assignment.

Quoted 5 lines
Quoted 1 line

Add big fat comments explaining what is going on here.

Looks like the old code worked by sheer luck, as we had wbc set and thus always assigned fs_info->fs_devices->latest_dev->bdev to the bio. Which would obviously not work on a multi device FS.

No, it worked fine because the real bio is set just before writing the data somewhere deep in the io submit path in submit_stripe_bio().

Quoted 7 lines

That it has to be set here is because of the cgroup implementation that accesses it, see 429aebc0a9a0 ("btrfs: get bdev directly from fs_devices in submit_extent_page").

Which brings me to the question if Christoph's fix is correct because the comment for the wbc + zoned append is assuming something that's not true.

While the real bio is setup in submit_stripe_bio(), we need to set the device destination for bio_add_zone_append_page() called in btrfs_bio_add_page(). The bio_add_zone_append_page() checks that the bio length is not exceeding max_zone_append_sectors() of the device, and checks other hardware restrictions.

#10 lore raw

Re: [PATCH 2/2] btrfs: fix and document the zoned device choice in alloc_new_bio

Quoted 22 lines · Naohiro Aota
Quoted 21 lines · David Sterba
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig

Zone Append bios only need a valid block device in struct bio, but not the device in the btrfs_bio. Use the information from btrfs_zoned_get_device to set up bi_bdev and fix zoned writes on multi-device file system with non-homogeneous capabilities and remove the pointless btrfs_bio.device assignment.

Quoted 5 lines
Quoted 1 line

Add big fat comments explaining what is going on here.

Looks like the old code worked by sheer luck, as we had wbc set and thus always assigned fs_info->fs_devices->latest_dev->bdev to the bio. Which would obviously not work on a multi device FS.

No, it worked fine because the real bio is set just before writing the data somewhere deep in the io submit path in submit_stripe_bio().

That it has to be set here is because of the cgroup implementation that accesses it, see 429aebc0a9a0 ("btrfs: get bdev directly from fs_devices in submit_extent_page").

Quoted 5 lines
Quoted 3 lines

Which brings me to the question if Christoph's fix is correct because the comment for the wbc + zoned append is assuming something that's not true.

While the real bio is setup in submit_stripe_bio(), we need to set the

Oh sorry I actually wanted to say that the real 'bdev' is set in submit_stripe_bio (ie. the one where the write is going to be done).

Quoted 4 lines

device destination for bio_add_zone_append_page() called in btrfs_bio_add_page(). The bio_add_zone_append_page() checks that the bio length is not exceeding max_zone_append_sectors() of the device, and checks other hardware restrictions.

Yeah, but can this still mean that it's checking potentially different devices with different hw restrictions? In alloc_new_bio() it's one and in submit_stripe_bio() it's a different one.

Before the cgroup writeback was added to bios, the only reason why bio_set_bdev required the block device is to check if it's the same one as before and drop some bit:

static inline void bio_set_dev(struct bio *bio, struct block_device *bdev)

{
	bio_clear_flag(bio, BIO_REMAPPED);
	if (bio->bi_bdev != bdev)
		bio_clear_flag(bio, BIO_THROTTLED);
	bio->bi_bdev = bdev;
	bio_associate_blkg(bio);		<-- this was not here
}

So the latest_dev was just a stub to satisfy the bio API requirements. Please note that its existence spans a long time and things have changed, I remember that Chris' answer to why we need the latest_dev was "to put something to the bios". Ie. we don't need it because we have to write same data to different block devices and distribute that in submit_stripe_bio(), while the bios have to be set much earlier expecting a block device.

I'm not sure we have a 1:1 match in what the APIs provide and expect and what btrfs wants to do. At this point multi-device support for zoned mode is not complete so we probably won't observe any problems with hardware with different restrictions.

Reviewed-by on v1 2/2
#11 lore raw

Re: [PATCH 2/2] btrfs: fix and document the zoned device choice in alloc_new_bio

In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig

Zone Append bios only need a valid block device in struct bio, but not the device in the btrfs_bio. Use the information from btrfs_zoned_get_device to set up bi_bdev and fix zoned writes on multi-device file system with non-homogeneous capabilities and remove the pointless btrfs_bio.device assignment.

Add big fat comments explaining what is going on here.

Looks good to me.

I'm wondering why I used the ->device here. Maybe, it was left over from pre-btrfs_bio_add_page().

#12 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig

Hi all,

this series fixes a minor and slightly less minor problem in the btrfs zoned device code. Note that for the second patch the comment might not be correct any more - AFAICS 5.18 added support for the dup profile for zoned devices, which means we do have a real issue now if different devices have different hardware limitations.

It's not a problem, as a) DUP is not spanning multiple devices, but only used on one device and b) DUP is (as of now) only used on meta-data which can't use zone append.

#13 lore raw
Quoted 9 lines · Johannes Thumshirn
Quoted 7 lines · Christoph Hellwig

Hi all,

this series fixes a minor and slightly less minor problem in the btrfs zoned device code. Note that for the second patch the comment might not be correct any more - AFAICS 5.18 added support for the dup profile for zoned devices, which means we do have a real issue now if different devices have different hardware limitations.

Quoted 3 lines

It's not a problem, as a) DUP is not spanning multiple devices, but only used on one device and b) DUP is (as of now) only used on meta-data which can't use zone append.

Ok, in that case the comment that I wrote based on the 5.17-ish code is still correct.

#15 lore raw
Quoted 3 lines · Christoph Hellwig

David,

do you plan to pick these up?

Yes I do but I had some questions after the initial review and don't know when I'll get back to it because of other patches being reviewed.

#16 lore raw
In reply to Christoph Hellwig
Quoted 12 lines · Christoph Hellwig

Hi all,

this series fixes a minor and slightly less minor problem in the btrfs zoned device code. Note that for the second patch the comment might not be correct any more - AFAICS 5.18 added support for the dup profile for zoned devices, which means we do have a real issue now if different devices have different hardware limitations. I think we'll need some code to check that all zoned devices have the exact same hardware limits (max_sectors, max_segments, max_segment_size, queue_boundary, virt_boundary), but I don't know the code well enough to implement that myself

Quoted 1 line

Found by code inspection as part of my bio cleanups.

Added to misc-next, thanks.