Skip to contentSkip to conversation

btrfs: the missing 4 patches to implement metadata write path

PATCH In review Qu Wenruo 3 participants 9 messages Apr 6, 2021 – Apr 23, 2021 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs: the missing 4 patches to implement metadata write path

When adding the comments for subpage metadata code, I inserted the comment patch into the wrong position, and then use that patch as a separator between data and metadata write path.

Thus the submitted metadata write path patchset lacks the real functions to submit subpage metadata write bio.

Qu Wenruo (4):
btrfs: introduce end_bio_subpage_eb_writepage() function
btrfs: introduce write_one_subpage_eb() function
btrfs: make lock_extent_buffer_for_io() to be subpage compatible
btrfs: introduce submit_eb_subpage() to submit a subpage metadata page

 fs/btrfs/extent_io.c | 293 ++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 263 insertions(+), 30 deletions(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH 1/4 btrfs: introduce end_bio_subpage_eb_writepage() function

The new function, end_bio_subpage_eb_writepage(), will handle the metadata writeback endio.

The major differences involved are:
- How to grab extent buffer
Now page::private is a pointer to btrfs_subpage, we can no longer grab extent buffer directly.
Thus we need to use the bv_offset to locate the extent buffer manually and iterate through the whole range.

- Use btrfs_subpage_end_writeback() caller This helper will handle the subpage writeback for us.

Since this function is executed under endio context, when grabbing extent buffers it can't grab eb->refs_lock as that lock is not designed to be grabbed under hardirq context.

So here introduce a helper, find_extent_buffer_nospinlock(), for such situation, and convert find_extent_buffer() to use that helper.

---
 fs/btrfs/extent_io.c | 135 +++++++++++++++++++++++++++++++++----------
 1 file changed, 106 insertions(+), 29 deletions(-)
fs/btrfs/extent_io.c +106 -29
index 9bebc6786b15..ea8ee925738a 100644
@@ -4080,13 +4080,97 @@ static void set_btree_ioerr(struct page *page, struct extent_buffer *eb)
 	}
 }
 
+/*
+ * This is the endio specific version which won't touch any unsafe spinlock
+ * in endio context.
+ */
+static struct extent_buffer *find_extent_buffer_nospinlock(
+		struct btrfs_fs_info *fs_info, u64 start)
+{
+	struct extent_buffer *eb;
+
+	rcu_read_lock();
+	eb = radix_tree_lookup(&fs_info->buffer_radix,
+			       start >> fs_info->sectorsize_bits);
+	if (eb && atomic_inc_not_zero(&eb->refs)) {
+		rcu_read_unlock();
+		return eb;
+	}
+	rcu_read_unlock();
+	return NULL;
+}
+/*
+ * The endio function for subpage extent buffer write.
+ *
+ * Unlike end_bio_extent_buffer_writepage(), we only call end_page_writeback()
+ * after all extent buffers in the page has finished their writeback.
+ */
+static void end_bio_subpage_eb_writepage(struct btrfs_fs_info *fs_info,
+					 struct bio *bio)
+{
+	struct bio_vec *bvec;
+	struct bvec_iter_all iter_all;
+
+	ASSERT(!bio_flagged(bio, BIO_CLONED));
+	bio_for_each_segment_all(bvec, bio, iter_all) {
+		struct page *page = bvec->bv_page;
+		u64 bvec_start = page_offset(page) + bvec->bv_offset;
+		u64 bvec_end = bvec_start + bvec->bv_len - 1;
+		u64 cur_bytenr = bvec_start;
+
+		ASSERT(IS_ALIGNED(bvec->bv_len, fs_info->nodesize));
+
+		/* Iterate through all extent buffers in the range */
+		while (cur_bytenr <= bvec_end) {
+			struct extent_buffer *eb;
+			int done;
+
+			/*
+			 * Here we can't use find_extent_buffer(), as it may
+			 * try to lock eb->refs_lock, which is not safe in endio
+			 * context.
+			 */
+			eb = find_extent_buffer_nospinlock(fs_info, cur_bytenr);
+			ASSERT(eb);
+
+			cur_bytenr = eb->start + eb->len;
+
+			ASSERT(test_bit(EXTENT_BUFFER_WRITEBACK, &eb->bflags));
+			done = atomic_dec_and_test(&eb->io_pages);
+			ASSERT(done);
+
+			if (bio->bi_status ||
+			    test_bit(EXTENT_BUFFER_WRITE_ERR, &eb->bflags)) {
+				ClearPageUptodate(page);
+				set_btree_ioerr(page, eb);
+			}
+
+			btrfs_subpage_clear_writeback(fs_info, page, eb->start,
+						      eb->len);
+			end_extent_buffer_writeback(eb);
+			/*
+			 * free_extent_buffer() will grab spinlock which is not
+			 * safe in endio context. Thus here we manually dec
+			 * the ref.
+			 */
+			atomic_dec(&eb->refs);
+		}
+	}
+	bio_put(bio);
+}
+
 static void end_bio_extent_buffer_writepage(struct bio *bio)
 {
+	struct btrfs_fs_info *fs_info;
 	struct bio_vec *bvec;
 	struct extent_buffer *eb;
 	int done;
 	struct bvec_iter_all iter_all;
 
+	fs_info = btrfs_sb(bio_first_page_all(bio)->mapping->host->i_sb);
+	if (fs_info->sectorsize < PAGE_SIZE)
+		return end_bio_subpage_eb_writepage(fs_info, bio);
+
 	ASSERT(!bio_flagged(bio, BIO_CLONED));
 	bio_for_each_segment_all(bvec, bio, iter_all) {
 		struct page *page = bvec->bv_page;
@@ -5467,36 +5551,29 @@ struct extent_buffer *find_extent_buffer(struct btrfs_fs_info *fs_info,
 {
 	struct extent_buffer *eb;
 
-	rcu_read_lock();
-	eb = radix_tree_lookup(&fs_info->buffer_radix,
-			       start >> fs_info->sectorsize_bits);
-	if (eb && atomic_inc_not_zero(&eb->refs)) {
-		rcu_read_unlock();
-		/*
-		 * Lock our eb's refs_lock to avoid races with
-		 * free_extent_buffer. When we get our eb it might be flagged
-		 * with EXTENT_BUFFER_STALE and another task running
-		 * free_extent_buffer might have seen that flag set,
-		 * eb->refs == 2, that the buffer isn't under IO (dirty and
-		 * writeback flags not set) and it's still in the tree (flag
-		 * EXTENT_BUFFER_TREE_REF set), therefore being in the process
-		 * of decrementing the extent buffer's reference count twice.
-		 * So here we could race and increment the eb's reference count,
-		 * clear its stale flag, mark it as dirty and drop our reference
-		 * before the other task finishes executing free_extent_buffer,
-		 * which would later result in an attempt to free an extent
-		 * buffer that is dirty.
-		 */
-		if (test_bit(EXTENT_BUFFER_STALE, &eb->bflags)) {
-			spin_lock(&eb->refs_lock);
-			spin_unlock(&eb->refs_lock);
-		}
-		mark_extent_buffer_accessed(eb, NULL);
-		return eb;
+	eb = find_extent_buffer_nospinlock(fs_info, start);
+	if (!eb)
+		return NULL;
+	/*
+	 * Lock our eb's refs_lock to avoid races with free_extent_buffer().
+	 * When we get our eb it might be flagged with EXTENT_BUFFER_STALE and
+	 * another task running free_extent_buffer() might have seen that flag
+	 * set, eb->refs == 2, that the buffer isn't under IO (dirty and
+	 * writeback flags not set) and it's still in the tree (flag
+	 * EXTENT_BUFFER_TREE_REF set), therefore being in the process
+	 * of decrementing the extent buffer's reference count twice.
+	 * So here we could race and increment the eb's reference count,
+	 * clear its stale flag, mark it as dirty and drop our reference
+	 * before the other task finishes executing free_extent_buffer,
+	 * which would later result in an attempt to free an extent
+	 * buffer that is dirty.
+	 */
+	if (test_bit(EXTENT_BUFFER_STALE, &eb->bflags)) {
+		spin_lock(&eb->refs_lock);
+		spin_unlock(&eb->refs_lock);
 	}
-	rcu_read_unlock();
-
-	return NULL;
+	mark_extent_buffer_accessed(eb, NULL);
+	return eb;
 }
 
 #ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH 2/4 btrfs: introduce write_one_subpage_eb() function

In reply to Qu Wenruo

The new function, write_one_subpage_eb(), as a subroutine for subpage metadata write, will handle the extent buffer bio submission.

The major differences between the new write_one_subpage_eb() and write_one_eb() is:
- No page locking
When entering write_one_subpage_eb() the page is no longer locked. We only lock the page for its status update, and unlock immediately. Now we completely rely on extent io tree locking.

- Extra bitmap update along with page status update
Now page dirty and writeback is controlled by btrfs_subpage::dirty_bitmap and btrfs_subpage::writeback_bitmap. They both follow the schema that any sector is dirty/writeback, then the full page get dirty/writeback.

- When to update the nr_written number
Now we take a short cut, if we have cleared the last dirty bit of the page, we update nr_written.
This is not completely perfect, but should emulate the old behavior good enough.

---
 fs/btrfs/extent_io.c | 55 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
fs/btrfs/extent_io.c +55 -0
index ea8ee925738a..9567e7b2b6cf 100644
@@ -4196,6 +4196,58 @@ static void end_bio_extent_buffer_writepage(struct bio *bio)
 	bio_put(bio);
 }
 
+/*
+ * Unlike the work in write_one_eb(), we rely completely on extent locking.
+ * Page locking is only utizlied at minimal to keep the VM code happy.
+ *
+ * Caller should still call write_one_eb() other than this function directly.
+ * As write_one_eb() has extra prepration before submitting the extent buffer.
+ */
+static int write_one_subpage_eb(struct extent_buffer *eb,
+				struct writeback_control *wbc,
+				struct extent_page_data *epd)
+{
+	struct btrfs_fs_info *fs_info = eb->fs_info;
+	struct page *page = eb->pages[0];
+	unsigned int write_flags = wbc_to_write_flags(wbc) | REQ_META;
+	bool no_dirty_ebs = false;
+	int ret;
+
+	/* clear_page_dirty_for_io() in subpage helper need page locked. */
+	lock_page(page);
+	btrfs_subpage_set_writeback(fs_info, page, eb->start, eb->len);
+
+	/* If we're the last dirty bit to update nr_written */
+	no_dirty_ebs = btrfs_subpage_clear_and_test_dirty(fs_info, page,
+							  eb->start, eb->len);
+	if (no_dirty_ebs)
+		clear_page_dirty_for_io(page);
+
+	ret = submit_extent_page(REQ_OP_WRITE | write_flags, wbc, page,
+			eb->start, eb->len, eb->start - page_offset(page),
+			&epd->bio, end_bio_extent_buffer_writepage, 0, 0, 0,
+			false);
+	if (ret) {
+		btrfs_subpage_clear_writeback(fs_info, page, eb->start,
+					      eb->len);
+		set_btree_ioerr(page, eb);
+		unlock_page(page);
+
+		if (atomic_dec_and_test(&eb->io_pages))
+			end_extent_buffer_writeback(eb);
+		return -EIO;
+	}
+	unlock_page(page);
+	/*
+	 * Submission finishes without problem, if no range of the page is
+	 * dirty anymore, we have submitted a page.
+	 * Update the nr_written in wbc.
+	 */
+	if (no_dirty_ebs)
+		update_nr_written(wbc, 1);
+	return ret;
+}
+
 static noinline_for_stack int write_one_eb(struct extent_buffer *eb,
 			struct writeback_control *wbc,
 			struct extent_page_data *epd)
@@ -4227,6 +4279,9 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb,
 		memzero_extent_buffer(eb, start, end - start);
 	}
 
+	if (eb->fs_info->sectorsize < PAGE_SIZE)
+		return write_one_subpage_eb(eb, wbc, epd);
+
 	for (i = 0; i < num_pages; i++) {
 		struct page *p = eb->pages[i];
 
Signature · 1 line
2.31.1
author
#3 lore raw

PATCH 3/4 btrfs: make lock_extent_buffer_for_io() to be subpage compatible

In reply to Qu Wenruo

For subpage metadata, we don't use page locking at all. So just skip the page locking part for subpage.

All the remaining routine can be reused.

---
 fs/btrfs/extent_io.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
fs/btrfs/extent_io.c +7 -1
index 9567e7b2b6cf..db40bc701a03 100644
@@ -3967,7 +3967,13 @@ static noinline_for_stack int lock_extent_buffer_for_io(struct extent_buffer *eb
 
 	btrfs_tree_unlock(eb);
 
-	if (!ret)
+	/*
+	 * Either we don't need to submit any tree block, or we're submitting
+	 * subpage.
+	 * Subpage metadata doesn't use page locking at all, so we can skip
+	 * the page locking.
+	 */
+	if (!ret || fs_info->sectorsize < PAGE_SIZE)
 		return ret;
 
 	num_pages = num_extent_pages(eb);
Signature · 1 line
2.31.1
author
#4 lore raw

PATCH 4/4 btrfs: introduce submit_eb_subpage() to submit a subpage metadata page

In reply to Qu Wenruo

The new function, submit_eb_subpage(), will submit all the dirty extent buffers in the page.

The major difference between submit_eb_page() and submit_eb_subpage() is:
- How to grab extent buffer
Now we use find_extent_buffer_nospinlock() other than using page::private.

All other different handling is already done in functions like lock_extent_buffer_for_io() and write_one_eb().

---
 fs/btrfs/extent_io.c | 95 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
fs/btrfs/extent_io.c +95 -0
index db40bc701a03..12321c06e212 100644
@@ -4323,6 +4323,98 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb,
 	return ret;
 }
 
+/*
+ * Submit one subpage btree page.
+ *
+ * The main difference between submit_eb_page() is:
+ * - Page locking
+ *   For subpage, we don't rely on page locking at all.
+ *
+ * - Flush write bio
+ *   We only flush bio if we may be unable to fit current extent buffers into
+ *   current bio.
+ *
+ * Return >=0 for the number of submitted extent buffers.
+ * Return <0 for fatal error.
+ */
+static int submit_eb_subpage(struct page *page,
+			     struct writeback_control *wbc,
+			     struct extent_page_data *epd)
+{
+	struct btrfs_fs_info *fs_info = btrfs_sb(page->mapping->host->i_sb);
+	int submitted = 0;
+	u64 page_start = page_offset(page);
+	int bit_start = 0;
+	int nbits = BTRFS_SUBPAGE_BITMAP_SIZE;
+	int sectors_per_node = fs_info->nodesize >> fs_info->sectorsize_bits;
+	int ret;
+
+	/* Lock and write each dirty extent buffers in the range */
+	while (bit_start < nbits) {
+		struct btrfs_subpage *subpage = (struct btrfs_subpage *)page->private;
+		struct extent_buffer *eb;
+		unsigned long flags;
+		u64 start;
+
+		/*
+		 * Take private lock to ensure the subpage won't be detached
+		 * halfway.
+		 */
+		spin_lock(&page->mapping->private_lock);
+		if (!PagePrivate(page)) {
+			spin_unlock(&page->mapping->private_lock);
+			break;
+		}
+		spin_lock_irqsave(&subpage->lock, flags);
+		if (!((1 << bit_start) & subpage->dirty_bitmap)) {
+			spin_unlock_irqrestore(&subpage->lock, flags);
+			spin_unlock(&page->mapping->private_lock);
+			bit_start++;
+			continue;
+		}
+
+		start = page_start + bit_start * fs_info->sectorsize;
+		bit_start += sectors_per_node;
+
+		/*
+		 * Here we just want to grab the eb without touching extra
+		 * spin locks. So here we call find_extent_buffer_nospinlock().
+		 */
+		eb = find_extent_buffer_nospinlock(fs_info, start);
+		spin_unlock_irqrestore(&subpage->lock, flags);
+		spin_unlock(&page->mapping->private_lock);
+
+		/*
+		 * The eb has already reached 0 refs thus find_extent_buffer()
+		 * doesn't return it. We don't need to write back such eb
+		 * anyway.
+		 */
+		if (!eb)
+			continue;
+
+		ret = lock_extent_buffer_for_io(eb, epd);
+		if (ret == 0) {
+			free_extent_buffer(eb);
+			continue;
+		}
+		if (ret < 0) {
+			free_extent_buffer(eb);
+			goto cleanup;
+		}
+		ret = write_one_eb(eb, wbc, epd);
+		free_extent_buffer(eb);
+		if (ret < 0)
+			goto cleanup;
+		submitted++;
+	}
+	return submitted;
+
+cleanup:
+	/* We hit error, end bio for the submitted extent buffers */
+	end_write_bio(epd, ret);
+	return ret;
+}
+
 /*
  * Submit all page(s) of one extent buffer.
  *
@@ -4355,6 +4447,9 @@ static int submit_eb_page(struct page *page, struct writeback_control *wbc,
 	if (!PagePrivate(page))
 		return 0;
 
+	if (btrfs_sb(page->mapping->host->i_sb)->sectorsize < PAGE_SIZE)
+		return submit_eb_subpage(page, wbc, epd);
+
 	spin_lock(&mapping->private_lock);
 	if (!PagePrivate(page)) {
 		spin_unlock(&mapping->private_lock);
Signature · 1 line
2.31.1
#5 lore raw
In reply to Qu Wenruo
Quoted 4 lines · Qu Wenruo

When adding the comments for subpage metadata code, I inserted the comment patch into the wrong position, and then use that patch as a separator between data and metadata write path.

Quoted 8 lines

Thus the submitted metadata write path patchset lacks the real functions to submit subpage metadata write bio.

Qu Wenruo (4):
btrfs: introduce end_bio_subpage_eb_writepage() function
btrfs: introduce write_one_subpage_eb() function
btrfs: make lock_extent_buffer_for_io() to be subpage compatible
btrfs: introduce submit_eb_subpage() to submit a subpage metadata page

For the record, the patches have been added to 5.13 queue a few days ago.

#6 lore raw
Quoted 8 lines · David Sterba
Quoted 7 lines · Qu Wenruo

When adding the comments for subpage metadata code, I inserted the comment patch into the wrong position, and then use that patch as a separator between data and metadata write path.

Thus the submitted metadata write path patchset lacks the real functions to submit subpage metadata write bio.

Quoted 9 lines
Quoted 5 lines

Qu Wenruo (4):
btrfs: introduce end_bio_subpage_eb_writepage() function
btrfs: introduce write_one_subpage_eb() function
btrfs: make lock_extent_buffer_for_io() to be subpage compatible
btrfs: introduce submit_eb_subpage() to submit a subpage metadata page

For the record, the patches have been added to 5.13 queue a few days ago.

Josef had some comments on them, most of them are just related to introducing a new subpage specific endio function other than reusing the existing one.

So I guess if I go that direction, I should just add new patches as a refactor?

Thanks,
Qu

#7 lore raw
Quoted 17 lines · Qu Wenruo
Quoted 14 lines · David Sterba
Quoted 12 lines · Qu Wenruo

When adding the comments for subpage metadata code, I inserted the comment patch into the wrong position, and then use that patch as a separator between data and metadata write path.

Thus the submitted metadata write path patchset lacks the real functions to submit subpage metadata write bio.

Qu Wenruo (4):
btrfs: introduce end_bio_subpage_eb_writepage() function
btrfs: introduce write_one_subpage_eb() function
btrfs: make lock_extent_buffer_for_io() to be subpage compatible
btrfs: introduce submit_eb_subpage() to submit a subpage metadata page

Quoted 7 lines
Quoted 3 lines

For the record, the patches have been added to 5.13 queue a few days ago.

Josef had some comments on them, most of them are just related to introducing a new subpage specific endio function other than reusing the existing one.

I haven't seen any comments for that patchset.

Quoted 2 lines

So I guess if I go that direction, I should just add new patches as a refactor?

Yes please.

#8 lore raw
Quoted 22 lines · David Sterba
Quoted 21 lines · Qu Wenruo
Quoted 17 lines · David Sterba
Quoted 4 lines · Qu Wenruo

When adding the comments for subpage metadata code, I inserted the comment patch into the wrong position, and then use that patch as a separator between data and metadata write path.

Quoted 8 lines

Thus the submitted metadata write path patchset lacks the real functions to submit subpage metadata write bio.

Qu Wenruo (4):
btrfs: introduce end_bio_subpage_eb_writepage() function
btrfs: introduce write_one_subpage_eb() function
btrfs: make lock_extent_buffer_for_io() to be subpage compatible
btrfs: introduce submit_eb_subpage() to submit a subpage metadata page

For the record, the patches have been added to 5.13 queue a few days ago.

Quoted 5 lines
Quoted 3 lines

Josef had some comments on them, most of them are just related to introducing a new subpage specific endio function other than reusing the existing one.

I haven't seen any comments for that patchset.

It's for the big subpage RW patchset, which includes the four missing patches.

Thanks,
Qu

Quoted 6 lines
Quoted 3 lines

So I guess if I go that direction, I should just add new patches as a refactor?

Yes please.