Skip to contentSkip to conversation

btrfs: zoned block device support

PATCH Superseded Naohiro Aota 7 participants 119 messages Oct 30, 2020 – Nov 10, 2020 Page 1 of 2

Subsystems (4)

v9 of 15 Superseded Latest is v15 →

lore
Share & tools
lore
#0 lore raw

v9 cover btrfs: zoned block device support

This series adds zoned block device support to btrfs.

This series is also available on github.
Kernel https://github.com/naota/linux/tree/btrfs-zoned-v9
Userland https://github.com/naota/btrfs-progs/tree/btrfs-zoned
xfstests https://github.com/naota/fstests/tree/btrfs-zoned

Userland tool depends on patched util-linux (libblkid and wipefs) to handle log-structured superblock. To ease the testing, pre-compiled static linked userland tools are available here:
https://wdc.app.box.com/s/fnhqsb3otrvgkstq66o6bvdw6tk525kp

Changes from v8:
- Direct-IO path now follow several hardware restrictions (other than max_zone_append_size) by using ZONE_APPEND support of iomap
- introduces union of fs_info->zone_size and fs_info->zoned [Johannes]
- and use btrfs_is_zoned(fs_info) in place of btrfs_fs_incompat(fs_info, ZONED)
- print if zoned is enabled or not when printing module info [Johannes]
- drop patch of disabling inode_cache on ZONED
- moved for_teelog flag to a proper location [Johannes]
- Code style fixes [Johannes]
- Add comment about adding physical layer things to ordered extent structure
- Pass file_offset explicitly to extract_ordered_extent() instead of determining it from bio
- Bug fixes
- write out fsync region so that the logical address of ordered extents and checksums are properly finalized
- free zone_info at umount time
- fix superblock log handling when entering zones[1] in the first time
- fixes double free of log-tree roots [Johannes]
- Drop erroneous ASSERT in do_allocation_zoned()

btrfs-progs and xfstests series will follow.

This version of ZONED btrfs switched from normal write command to zone append write command. You do not need to specify LBA (at the write pointer) to write for zone append write command. Instead, you only select a zone to write with its start LBA. Then the device (NVMe ZNS), or the emulation of zone append command in the sd driver in the case of SAS or SATA HDDs, automatically writes the data at the write pointer position and return the written LBA as a command reply.

The benefit of using the zone append write command is that write command issuing order does not matter. So, we can eliminate block group lock and utilize asynchronous checksum, which can reorder the IOs.

Eliminating the lock improves performance. In particular, on a workload with massive competing to the same zone [1], we observed 36% performance improvement compared to normal write.

[1] Fio running 16 jobs with 4KB random writes for 5 minutes

However, there are some limitations. We cannot use the non-SINGLE profile. Supporting non-SINGLE profile with zone append writing is not trivial. For example, in the DUP profile, we send a zone append writing IO to two zones on a device. The device reply with written LBAs for the IOs. If the offsets of the returned addresses from the beginning of the zone are different, then it results in different logical addresses.

For the same reason, we cannot issue multiple IOs for one ordered extent. Thus, the size of an ordered extent is limited under max_zone_append_size. This limitation will cause fragmentation and increased usage of metadata. In the future, we can add optimization to merge ordered extents after end_bio.

* Patch series description

A zoned block device consists of a number of zones. Zones are either conventional and accepting random writes or sequential and requiring that writes be issued in LBA order from each zone write pointer position. This patch series ensures that the sequential write constraint of sequential zones is respected while fundamentally not changing BtrFS block and I/O management for block stored in conventional zones.

To achieve this, the default chunk size of btrfs is changed on zoned block devices so that chunks are always aligned to a zone. Allocation of blocks within a chunk is changed so that the allocation is always sequential from the beginning of the chunks. To do so, an allocation pointer is added to block groups and used as the allocation hint. The allocation changes also ensure that blocks freed below the allocation pointer are ignored, resulting in sequential block allocation regardless of the chunk usage.

The zone of a chunk is reset to allow reuse of the zone only when the block group is being freed, that is, when all the chunks of the block group are unused.

For btrfs volumes composed of multiple zoned disks, a restriction is added to ensure that all disks have the same zone size. This restriction matches the existing constraint that all chunks in a block group must have the same size.

* Enabling tree-log

The tree-log feature does not work on ZONED mode as is. Blocks for a tree-log tree are allocated mixed with other metadata blocks, and btrfs writes and syncs the tree-log blocks to devices at the time of fsync(), which is different timing than a global transaction commit. As a result, both writing tree-log blocks and writing other metadata blocks become non-sequential writes which ZONED mode must avoid.

This series introduces a dedicated block group for tree-log blocks to create two metadata writing streams, one for tree-log blocks and the other for metadata blocks. As a result, each write stream can now be written to devices separately and sequentially.

* Log-structured superblock

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone.

This series implements superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second zone. The first zone will be reset once both zones are filled. We can determine the postion of the latest superblock by reading the write pointer information from a device.

* Patch series organization

Patches 1 and 2 are preparing patches for block and iomap layer.

Patch 3 introduces the ZONED incompatible feature flag to indicate that the btrfs volume was formatted for use on zoned block devices.

Patches 4 to 6 implement functions to gather information on the zones of the device (zones type, write pointer position, and max_zone_append_size).

Patches 7 to 10 disable features which are not compatible with the sequential write constraints of zoned block devices. These includes space_cache, NODATACOW, fallocate, and MIXED_BG.

Patch 11 implements the log-structured superblock writing.

Patches 12 and 13 tweak the device extent allocation for ZONED mode and add verification to check if a device extent is properly aligned to zones.

Patches 14 to 17 implements sequential block allocator for ZONED mode.

Patch 18 implement a zone reset for unused block groups.

Patches 19 to 30 implement the writing path for several types of IO (non-compressed data, direct IO, and metadata). These include re-dirtying once-freed metadata blocks to prevent write holes.

Patches 31 to 40 tweak some btrfs features work with ZONED mode. These include device-replace, relocation, repairing IO error, and tree-log.

Finally, patch 41 adds the ZONED feature to the list of supported features.

* Patch testing note

** Zone-aware util-linux

Since the log-structured superblock feature changed the location of superblock magic, the current util-linux (libblkid) cannot detect ZONED btrfs anymore. You need to apply a to-be posted patch to util-linux to make it "zone aware".

** Testing device

You need devices with zone append writing command support to run ZONED btrfs.

Other than real devices, null_blk supports zone append write command. You can use memory backed null_blk to run the test on it. Following script creates 12800 MB /dev/nullb0.

    sysfs=/sys/kernel/config/nullb/nullb0
    size=12800 # MB
    
    # drop nullb0
    if [[ -d $sysfs ]]; then
            echo 0 > "${sysfs}"/power
            rmdir $sysfs
    fi
    lsmod | grep -q null_blk && rmmod null_blk
    modprobe null_blk nr_devices=0
    
    mkdir "${sysfs}"
    
    echo "${size}" > "${sysfs}"/size
    echo 1 > "${sysfs}"/zoned
    echo 0 > "${sysfs}"/zone_nr_conv
    echo 1 > "${sysfs}"/memory_backed
    
    echo 1 > "${sysfs}"/power
    udevadm settle

Zoned SCSI devices such as SMR HDDs or scsi_debug also support the zone append command as an emulated command within the SCSI sd driver. This emulation is completely transparent to the user and provides the same semantic as a NVMe ZNS native drive support.

Also, there is a qemu patch available to enable NVMe ZNS device.

** xfstests

We ran xfstests on ZONED btrfs, and, if we omit some cases that are known to fail currently, all test cases pass.

Cases that can be ignored:
1) failing also with the regular btrfs on regular devices,
2) trying to test fallocate feature without testing with "_require_xfs_io_command "falloc"",
3) trying to test incompatible features for ZONED btrfs (e.g. RAID5/6)
4) trying to use incompatible setup for ZONED btrfs (e.g. dm-linear not aligned to zone boundary, swap)
5) trying to create a file system with too small size, (we require at least 9 zones to initiate a ZONED btrfs)
6) dropping original MKFS_OPTIONS ("-O zoned"), so it cannot create ZONED btrfs (btrfs/003)
7) having ENOSPC which incurred by larger metadata block group size

I will send a patch series for xfstests to handle these cases (2-6) properly.

Patched xfstests is available here:

https://github.com/naota/fstests/tree/btrfs-zoned

Also, you need to apply the following patch if you run xfstests with tcmu devices. xfstests btrfs/003 failed to "_devmgt_add" after "_devmgt_remove" without this patch.

https://marc.info/?l=linux-scsi&m=156498625421698&w=2

v8 https://lore.kernel.org/linux-btrfs/[email protected]/
v7 https://lore.kernel.org/linux-btrfs/[email protected]/
v6 https://lore.kernel.org/linux-btrfs/[email protected]/
v5 https://lore.kernel.org/linux-btrfs/[email protected]/
v4 https://lwn.net/Articles/797061/
v3 https://lore.kernel.org/linux-btrfs/[email protected]/
v2 https://lore.kernel.org/linux-btrfs/[email protected]/
v1 https://lore.kernel.org/linux-btrfs/[email protected]/

Changelog
v8
- Use bio_add_hw_page() to build up bio to honor hardware restrictions
- add bio_add_zone_append_page() as a wrapper of the function
- Split file extent on submitting bio
- If bio_add_zone_append_page() fails, split the file extent and send out bio
- so, we can ensure one bio == one file extent
- Fix build bot issues
- Rebased on misc-next
v7:
- Use zone append write command instead of normal write command
- Bio issuing order does not matter
- No need to use lock anymore
- Can use asynchronous checksum
- Removed RAID support for now
- Rename HMZONED to ZONED
- Split some patches
- Rebased on kdave/for-5.9-rc3 + iomap direct IO
v6:
- Use bitmap helpers (Johannes)
- Code cleanup (Johannes)
- Rebased on kdave/for-5.5
- Enable the tree-log feature.
- Treat conventional zones as sequential zones, so we can now allow mixed allocation of conventional zone and sequential write required zone to construct a block group.
- Implement log-structured superblock
- No need for one conventional zone at the beginning of a device.
- Fix deadlock of direct IO writing
- Fix building with !CONFIG_BLK_DEV_ZONED (Johannes)
- Fix leak of zone_info (Johannes)
v5:
- Rebased on kdave/for-5.5
- Enable the tree-log feature.
- Treat conventional zones as sequential zones, so we can now allow mixed allocation of conventional zone and sequential write required zone to construct a block group.
- Implement log-structured superblock
- No need for one conventional zone at the beginning of a device.
- Fix deadlock of direct IO writing
- Fix building with !CONFIG_BLK_DEV_ZONED (Johannes)
- Fix leak of zone_info (Johannes)
v4:
- Move memory allcation of zone informattion out of btrfs_get_dev_zones() (Anand)
- Add disabled features table in commit log (Anand)
- Ensure "max_chunk_size >= devs_min * data_stripes * zone_size"
v3:
- Serialize allocation and submit_bio instead of bio buffering in btrfs_map_bio().
-- Disable async checksum/submit in HMZONED mode
- Introduce helper functions and hmzoned.c/h (Josef, David)
- Add support for repairing IO failure
- Add support for NOCOW direct IO write (Josef)
- Disable preallocation entirely
-- Disable INODE_MAP_CACHE
-- relocation is reworked not to rely on preallocation in HMZONED mode
- Disable NODATACOW
-Disable MIXED_BG
- Device extent that cover super block position is banned (David)
v2:
- Add support for dev-replace
-- To support dev-replace, moved submit_buffer one layer up. It now
handles bio instead of btrfs_bio.
-- Mark unmirrored Block Group readonly only when there are writable
mirrored BGs. Necessary to handle degraded RAID.
- Expire worker use vanilla delayed_work instead of btrfs's async-thread
- Device extent allocator now ensure that region is on the same zone type.
- Add delayed allocation shrinking.
- Rename btrfs_drop_dev_zonetypes() to btrfs_destroy_dev_zonetypes
- Fix
-- Use SECTOR_SHIFT (Nikolay)
-- Use btrfs_err (Nikolay)

Johannes Thumshirn (1):
block: add bio_add_zone_append_page

Naohiro Aota (40):
iomap: support REQ_OP_ZONE_APPEND
btrfs: introduce ZONED feature flag
btrfs: Get zone information of zoned block devices
btrfs: Check and enable ZONED mode
btrfs: introduce max_zone_append_size
btrfs: disallow space_cache in ZONED mode
btrfs: disallow NODATACOW in ZONED mode
btrfs: disable fallocate in ZONED mode
btrfs: disallow mixed-bg in ZONED mode
btrfs: implement log-structured superblock for ZONED mode
btrfs: implement zoned chunk allocator
btrfs: verify device extent is aligned to zone
btrfs: load zone's alloction offset
btrfs: emulate write pointer for conventional zones
btrfs: track unusable bytes for zones
btrfs: do sequential extent allocation in ZONED mode
btrfs: reset zones of unused block groups
btrfs: redirty released extent buffers in ZONED mode
btrfs: extract page adding function
btrfs: use bio_add_zone_append_page for zoned btrfs
btrfs: handle REQ_OP_ZONE_APPEND as writing
btrfs: split ordered extent when bio is sent
btrfs: extend btrfs_rmap_block for specifying a device
btrfs: use ZONE_APPEND write for ZONED btrfs
btrfs: enable zone append writing for direct IO
btrfs: introduce dedicated data write path for ZONED mode
btrfs: serialize meta IOs on ZONED mode
btrfs: wait existing extents before truncating
btrfs: avoid async metadata checksum on ZONED mode
btrfs: mark block groups to copy for device-replace
btrfs: implement cloning for ZONED device-replace
btrfs: implement copying for ZONED device-replace
btrfs: support dev-replace in ZONED mode
btrfs: enable relocation in ZONED mode
btrfs: relocate block group to repair IO failure in ZONED
btrfs: split alloc_log_tree()
btrfs: extend zoned allocator to use dedicated tree-log block group
btrfs: serialize log transaction on ZONED mode
btrfs: reorder log node allocation
btrfs: enable to mount ZONED incompat flag

 block/bio.c                 |   36 +
 fs/btrfs/Makefile           |    1 +
 fs/btrfs/block-group.c      |   90 ++-
 fs/btrfs/block-group.h      |   13 +
 fs/btrfs/ctree.h            |   18 +-
 fs/btrfs/dev-replace.c      |  187 ++++++
 fs/btrfs/dev-replace.h      |    3 +
 fs/btrfs/disk-io.c          |   92 ++-
 fs/btrfs/disk-io.h          |    2 +
 fs/btrfs/extent-tree.c      |  206 +++++-
 fs/btrfs/extent_io.c        |  136 +++-
 fs/btrfs/extent_io.h        |    2 +
 fs/btrfs/file.c             |    6 +-
 fs/btrfs/free-space-cache.c |   58 ++
 fs/btrfs/free-space-cache.h |    4 +
 fs/btrfs/inode.c            |  164 ++++-
 fs/btrfs/ioctl.c            |   16 +
 fs/btrfs/ordered-data.c     |   79 +++
 fs/btrfs/ordered-data.h     |   10 +
 fs/btrfs/relocation.c       |   35 +-
 fs/btrfs/scrub.c            |  145 ++++
 fs/btrfs/space-info.c       |   13 +-
 fs/btrfs/space-info.h       |    4 +-
 fs/btrfs/super.c            |   18 +-
 fs/btrfs/sysfs.c            |    4 +
 fs/btrfs/transaction.c      |   10 +
 fs/btrfs/transaction.h      |    3 +
 fs/btrfs/tree-log.c         |   52 +-
 fs/btrfs/volumes.c          |  311 ++++++++-
 fs/btrfs/volumes.h          |    7 +
 fs/btrfs/zoned.c            | 1264 +++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h            |  273 ++++++++
 fs/iomap/direct-io.c        |   22 +-
 include/linux/bio.h         |    2 +
 include/linux/iomap.h       |    1 +
 include/uapi/linux/btrfs.h  |    1 +
 36 files changed, 3176 insertions(+), 112 deletions(-)
 create mode 100644 fs/btrfs/zoned.c
 create mode 100644 fs/btrfs/zoned.h
Signature · 1 line
2.27.0
#1 lore raw

PATCH v9 1/41 block: add bio_add_zone_append_page

From: Johannes Thumshirn <[email protected]>

Add bio_add_zone_append_page(), a wrapper around bio_add_hw_page() which is intended to be used by file systems that directly add pages to a bio instead of using bio_iov_iter_get_pages().

---
 block/bio.c         | 36 ++++++++++++++++++++++++++++++++++++
 include/linux/bio.h |  2 ++
 2 files changed, 38 insertions(+)
block/bio.c +36 -0
index 58d765400226..2dfe40be4d6b 100644
@@ -853,6 +853,42 @@ int bio_add_pc_page(struct request_queue *q, struct bio *bio,
 }
 EXPORT_SYMBOL(bio_add_pc_page);
 
+/**
+ * bio_add_zone_append_page - attempt to add page to zone-append bio
+ * @bio: destination bio
+ * @page: page to add
+ * @len: vec entry length
+ * @offset: vec entry offset
+ *
+ * Attempt to add a page to the bio_vec maplist of a bio that will be submitted
+ * for a zone-append request. This can fail for a number of reasons, such as the
+ * bio being full or the target block device is not a zoned block device or
+ * other limitations of the target block device. The target block device must
+ * allow bio's up to PAGE_SIZE, so it is always possible to add a single page
+ * to an empty bio.
+ */
+int bio_add_zone_append_page(struct bio *bio, struct page *page,
+			     unsigned int len, unsigned int offset)
+{
+	struct request_queue *q;
+	bool same_page = false;
+
+	if (WARN_ON_ONCE(bio_op(bio) != REQ_OP_ZONE_APPEND))
+		return 0;
+
+	if (WARN_ON_ONCE(!bio->bi_disk))
+		return 0;
+
+	q = bio->bi_disk->queue;
+
+	if (WARN_ON_ONCE(!blk_queue_is_zoned(q)))
+		return 0;
+
+	return bio_add_hw_page(q, bio, page, len, offset,
+			       queue_max_zone_append_sectors(q), &same_page);
+}
+EXPORT_SYMBOL_GPL(bio_add_zone_append_page);
+
 /**
  * __bio_try_merge_page - try appending data to an existing bvec.
  * @bio: destination bio
include/linux/bio.h +2 -0
index c6d765382926..7ef300cb4e9a 100644
@@ -442,6 +442,8 @@ void bio_chain(struct bio *, struct bio *);
 extern int bio_add_page(struct bio *, struct page *, unsigned int,unsigned int);
 extern int bio_add_pc_page(struct request_queue *, struct bio *, struct page *,
 			   unsigned int, unsigned int);
+int bio_add_zone_append_page(struct bio *bio, struct page *page,
+			     unsigned int len, unsigned int offset);
 bool __bio_try_merge_page(struct bio *bio, struct page *page,
 		unsigned int len, unsigned int off, bool *same_page);
 void __bio_add_page(struct bio *bio, struct page *page,
Signature · 1 line
2.27.0
#2 lore raw

PATCH v9 2/41 iomap: support REQ_OP_ZONE_APPEND

A ZONE_APPEND bio must follow hardware restrictions (e.g. not exceeding max_zone_append_sectors) not to be split. bio_iov_iter_get_pages builds such restricted bio using __bio_iov_append_get_pages if bio_op(bio) == REQ_OP_ZONE_APPEND.

To utilize it, we need to set the bio_op before calling bio_iov_iter_get_pages(). This commit introduces IOMAP_F_ZONE_APPEND, so that iomap user can set the flag to indicate they want REQ_OP_ZONE_APPEND and restricted bio.

Cc Christoph Hellwig <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
 fs/iomap/direct-io.c  | 22 ++++++++++++++++------
 include/linux/iomap.h |  1 +
 2 files changed, 17 insertions(+), 6 deletions(-)
fs/iomap/direct-io.c +16 -6
index c1aafb2ab990..e534703c5594 100644
@@ -210,6 +210,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 	struct bio *bio;
 	bool need_zeroout = false;
 	bool use_fua = false;
+	bool zone_append = false;
 	int nr_pages, ret = 0;
 	size_t copied = 0;
 	size_t orig_count;
@@ -241,6 +242,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 			use_fua = true;
 	}
 
+	zone_append = iomap->flags & IOMAP_F_ZONE_APPEND;
+
 	/*
 	 * Save the original count and trim the iter to just the extent we
 	 * are operating on right now.  The iter will be re-expanded once
@@ -278,6 +281,19 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 		bio->bi_private = dio;
 		bio->bi_end_io = iomap_dio_bio_end_io;
 
+		if (dio->flags & IOMAP_DIO_WRITE) {
+			bio->bi_opf = (zone_append ? REQ_OP_ZONE_APPEND : REQ_OP_WRITE) |
+				      REQ_SYNC | REQ_IDLE;
+			if (use_fua)
+				bio->bi_opf |= REQ_FUA;
+			else
+				dio->flags &= ~IOMAP_DIO_WRITE_FUA;
+		} else {
+			WARN_ON_ONCE(zone_append);
+
+			bio->bi_opf = REQ_OP_READ;
+		}
+
 		ret = bio_iov_iter_get_pages(bio, dio->submit.iter);
 		if (unlikely(ret)) {
 			/*
@@ -292,14 +308,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 
 		n = bio->bi_iter.bi_size;
 		if (dio->flags & IOMAP_DIO_WRITE) {
-			bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;
-			if (use_fua)
-				bio->bi_opf |= REQ_FUA;
-			else
-				dio->flags &= ~IOMAP_DIO_WRITE_FUA;
 			task_io_account_write(n);
 		} else {
-			bio->bi_opf = REQ_OP_READ;
 			if (dio->flags & IOMAP_DIO_DIRTY)
 				bio_set_pages_dirty(bio);
 		}
include/linux/iomap.h +1 -0
index 4d1d3c3469e9..1bccd1880d0d 100644
@@ -54,6 +54,7 @@ struct vm_fault;
 #define IOMAP_F_SHARED		0x04
 #define IOMAP_F_MERGED		0x08
 #define IOMAP_F_BUFFER_HEAD	0x10
+#define IOMAP_F_ZONE_APPEND	0x20
 
 /*
  * Flags set by the core iomap code during operations:
Signature · 1 line
2.27.0
#3 lore raw

Re: [PATCH v9 02/41] iomap: support REQ_OP_ZONE_APPEND

Note: this patch is for this series.

https://lore.kernel.org/linux-btrfs/[email protected]/T/

The patch below uses IOMAP_F_ZONE_APPEND flag to make the iomap bio to be REQ_OP_ZONE_APPEND.

https://lore.kernel.org/linux-btrfs/[email protected]/T/#m7b3f2d01ab554ea4cba244608655268211928799

Quoted 90 lines · Naohiro Aota

A ZONE_APPEND bio must follow hardware restrictions (e.g. not exceeding max_zone_append_sectors) not to be split. bio_iov_iter_get_pages builds such restricted bio using __bio_iov_append_get_pages if bio_op(bio) == REQ_OP_ZONE_APPEND.

To utilize it, we need to set the bio_op before calling bio_iov_iter_get_pages(). This commit introduces IOMAP_F_ZONE_APPEND, so that iomap user can set the flag to indicate they want REQ_OP_ZONE_APPEND and restricted bio.

Cc Christoph Hellwig <[email protected]>
Cc "Darrick J. Wong" <[email protected]>

---
fs/iomap/direct-io.c | 22 ++++++++++++++++------
include/linux/iomap.h | 1 +

2 files changed, 17 insertions(+), 6 deletions(-)
fs/iomap/direct-io.c +16 -6
index c1aafb2ab990..e534703c5594 100644
@@ -210,6 +210,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 	struct bio *bio;
 	bool need_zeroout = false;
 	bool use_fua = false;
+	bool zone_append = false;
 	int nr_pages, ret = 0;
 	size_t copied = 0;
 	size_t orig_count;
@@ -241,6 +242,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 			use_fua = true;
 	}
 
+	zone_append = iomap->flags & IOMAP_F_ZONE_APPEND;
+
 	/*
 	 * Save the original count and trim the iter to just the extent we
 	 * are operating on right now.  The iter will be re-expanded once
@@ -278,6 +281,19 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 		bio->bi_private = dio;
 		bio->bi_end_io = iomap_dio_bio_end_io;
 
+		if (dio->flags & IOMAP_DIO_WRITE) {
+			bio->bi_opf = (zone_append ? REQ_OP_ZONE_APPEND : REQ_OP_WRITE) |
+				      REQ_SYNC | REQ_IDLE;
+			if (use_fua)
+				bio->bi_opf |= REQ_FUA;
+			else
+				dio->flags &= ~IOMAP_DIO_WRITE_FUA;
+		} else {
+			WARN_ON_ONCE(zone_append);
+
+			bio->bi_opf = REQ_OP_READ;
+		}
+
 		ret = bio_iov_iter_get_pages(bio, dio->submit.iter);
 		if (unlikely(ret)) {
 			/*
@@ -292,14 +308,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 
 		n = bio->bi_iter.bi_size;
 		if (dio->flags & IOMAP_DIO_WRITE) {
-			bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;
-			if (use_fua)
-				bio->bi_opf |= REQ_FUA;
-			else
-				dio->flags &= ~IOMAP_DIO_WRITE_FUA;
 			task_io_account_write(n);
 		} else {
-			bio->bi_opf = REQ_OP_READ;
 			if (dio->flags & IOMAP_DIO_DIRTY)
 				bio_set_pages_dirty(bio);
 		}
include/linux/iomap.h +1 -0
index 4d1d3c3469e9..1bccd1880d0d 100644
@@ -54,6 +54,7 @@ struct vm_fault;
 #define IOMAP_F_SHARED		0x04
 #define IOMAP_F_MERGED		0x08
 #define IOMAP_F_BUFFER_HEAD	0x10
+#define IOMAP_F_ZONE_APPEND	0x20
 
 /*
 * Flags set by the core iomap code during operations:
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH v9 02/41] iomap: support REQ_OP_ZONE_APPEND

In reply to Naohiro Aota
Quoted 9 lines · Naohiro Aota

A ZONE_APPEND bio must follow hardware restrictions (e.g. not exceeding max_zone_append_sectors) not to be split. bio_iov_iter_get_pages builds such restricted bio using __bio_iov_append_get_pages if bio_op(bio) == REQ_OP_ZONE_APPEND.

To utilize it, we need to set the bio_op before calling bio_iov_iter_get_pages(). This commit introduces IOMAP_F_ZONE_APPEND, so that iomap user can set the flag to indicate they want REQ_OP_ZONE_APPEND and restricted bio.

Would you mind putting this paragraph in a comment before the bio_iov_iter_get_pages call? I could easily see someone (i.e. 2021 me) forgetting there's a data dependency and breaking this.

Quoted 36 lines
Cc Christoph Hellwig <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
 fs/iomap/direct-io.c  | 22 ++++++++++++++++------
 include/linux/iomap.h |  1 +
 2 files changed, 17 insertions(+), 6 deletions(-)
fs/iomap/direct-io.c +5 -0
index c1aafb2ab990..e534703c5594 100644
@@ -210,6 +210,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 	struct bio *bio;
 	bool need_zeroout = false;
 	bool use_fua = false;
+	bool zone_append = false;
 	int nr_pages, ret = 0;
 	size_t copied = 0;
 	size_t orig_count;
@@ -241,6 +242,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 			use_fua = true;
 	}
 
+	zone_append = ;
+
 	/*
 	 * Save the original count and trim the iter to just the extent we
 	 * are operating on right now.  The iter will be re-expanded once
@@ -278,6 +281,19 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 		bio->bi_private = dio;
 		bio->bi_end_io = iomap_dio_bio_end_io;
 
+		if (dio->flags & IOMAP_DIO_WRITE) {
+			bio->bi_opf = (zone_append ? REQ_OP_ZONE_APPEND : REQ_OP_WRITE) |

fs/iomap/direct-io.c iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, · line 295

Quoted 10 lines
fs/iomap/direct-io.c +10 -0
+				      REQ_SYNC | REQ_IDLE;
+			if (use_fua)
+				bio->bi_opf |= REQ_FUA;
+			else
+				dio->flags &= ~IOMAP_DIO_WRITE_FUA;
+		} else {
+			WARN_ON_ONCE(zone_append);
+
+			bio->bi_opf = REQ_OP_READ;
+		}

Also, I think this should be hoisted into a separate helper to return bi_opf rather than making iomap_dio_bio_actor even longer...

/*
* Figure out the bio's operation flags from the dio request, the
* mapping, and whether or not we want FUA. Note that we can end up
* clearing the WRITE_FUA flag in the dio request.
*/
static inline unsigned int
iomap_dio_bio_opflags(struct iomap_dio *dio, struct iomap *iomap, bool use_fua)

{
	unsigned int opflags = REQ_SYNC | REQ_IDLE;

	if (!(dio->flags & IOMAP_DIO_WRITE)) {
		WARN_ON_ONCE(iomap->flags & IOMAP_F_ZONE_APPEND);
		return REQ_OP_READ;
	}

	if (iomap->flags & IOMAP_F_ZONE_APPEND)
		opflags |= REQ_OP_ZONE_APPEND;
	else
		opflags |= REQ_OP_WRITE;

	if (use_fua)
		opflags |= REQ_FUA;
	else
		dio->flags &= ~IOMAP_DIO_WRITE_FUA;

	return opflags;
}
	/*
	 * Set the operation flags early so that bio_iov_iter_get_pages can
	 * set up the page vector appropriately for a ZONE_APPEND operation.
	 */
	bio->bi_opf = iomap_dio_bio_opflags(dio, iomap, use_fua);

	ret = bio_iov_iter_get_pages(bio, dio->submit.iter);
	if (unlikely(ret)) {

--D

Quoted 34 lines
fs/iomap/direct-io.c +1 -6
+
 		ret = bio_iov_iter_get_pages(bio, dio->submit.iter);
 		if (unlikely(ret)) {
 			/*
@@ -292,14 +308,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 
 		n = bio->bi_iter.bi_size;
 		if (dio->flags & IOMAP_DIO_WRITE) {
-			bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;
-			if (use_fua)
-				bio->bi_opf |= REQ_FUA;
-			else
-				dio->flags &= ~IOMAP_DIO_WRITE_FUA;
 			task_io_account_write(n);
 		} else {
-			bio->bi_opf = REQ_OP_READ;
 			if (dio->flags & IOMAP_DIO_DIRTY)
 				bio_set_pages_dirty(bio);
 		}
include/linux/iomap.h +1 -0
index 4d1d3c3469e9..1bccd1880d0d 100644
@@ -54,6 +54,7 @@ struct vm_fault;
 #define IOMAP_F_SHARED		0x04
 #define IOMAP_F_MERGED		0x08
 #define IOMAP_F_BUFFER_HEAD	0x10
+#define IOMAP_F_ZONE_APPEND	0x20
 
 /*
  * Flags set by the core iomap code during operations:
Signature · 1 line
2.27.0
#5 lore raw

Re: [PATCH v9 02/41] iomap: support REQ_OP_ZONE_APPEND

Quoted 2 lines · Darrick J. Wong

Also, I think this should be hoisted into a separate helper to return bi_opf rather than making iomap_dio_bio_actor even longer...

Fixed up, thanks.

(Speaking for Naohiro here)

#6 lore raw

PATCH v9 3/41 btrfs: introduce ZONED feature flag

In reply to Naohiro Aota

This patch introduces the ZONED incompat flag. The flag indicates that the volume management will satisfy the constraints imposed by host-managed zoned block devices.

---
 fs/btrfs/sysfs.c           | 2 ++
 include/uapi/linux/btrfs.h | 1 +
 2 files changed, 3 insertions(+)
fs/btrfs/sysfs.c +2 -0
index 279d9262b676..828006020bbd 100644
@@ -263,6 +263,7 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES);
 BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
 BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
 BTRFS_FEAT_ATTR_INCOMPAT(raid1c34, RAID1C34);
+BTRFS_FEAT_ATTR_INCOMPAT(zoned, ZONED);
 
 static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(mixed_backref),
@@ -278,6 +279,7 @@ static struct attribute *btrfs_supported_feature_attrs[] = {
 	BTRFS_FEAT_ATTR_PTR(metadata_uuid),
 	BTRFS_FEAT_ATTR_PTR(free_space_tree),
 	BTRFS_FEAT_ATTR_PTR(raid1c34),
+	BTRFS_FEAT_ATTR_PTR(zoned),
 	NULL
 };
 
include/uapi/linux/btrfs.h +1 -0
index 2c39d15a2beb..5df73001aad4 100644
@@ -307,6 +307,7 @@ struct btrfs_ioctl_fs_info_args {
 #define BTRFS_FEATURE_INCOMPAT_NO_HOLES		(1ULL << 9)
 #define BTRFS_FEATURE_INCOMPAT_METADATA_UUID	(1ULL << 10)
 #define BTRFS_FEATURE_INCOMPAT_RAID1C34		(1ULL << 11)
+#define BTRFS_FEATURE_INCOMPAT_ZONED		(1ULL << 12)
 
 struct btrfs_ioctl_feature_flags {
 	__u64 compat_flags;
Signature · 1 line
2.27.0
#7 lore raw

PATCH v9 4/41 btrfs: Get zone information of zoned block devices

In reply to Naohiro Aota

If a zoned block device is found, get its zone information (number of zones and zone size) using the new helper function btrfs_get_dev_zone_info(). To avoid costly run-time zone report commands to test the device zones type during block allocation, attach the seq_zones bitmap to the device structure to indicate if a zone is sequential or accept random writes. Also it attaches the empty_zones bitmap to indicate if a zone is empty or not.

This patch also introduces the helper function btrfs_dev_is_sequential() to test if the zone storing a block is a sequential write required zone and btrfs_dev_is_empty_zone() to test if the zone is a empty zone.

---
 fs/btrfs/Makefile      |   1 +
 fs/btrfs/dev-replace.c |   5 ++
 fs/btrfs/super.c       |   5 ++
 fs/btrfs/volumes.c     |  19 ++++-
 fs/btrfs/volumes.h     |   4 +
 fs/btrfs/zoned.c       | 176 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |  86 ++++++++++++++++++++
 7 files changed, 294 insertions(+), 2 deletions(-)
 create mode 100644 fs/btrfs/zoned.c
 create mode 100644 fs/btrfs/zoned.h
fs/btrfs/Makefile +1 -0
index e738f6206ea5..0497fdc37f90 100644
@@ -16,6 +16,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
 btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
 btrfs-$(CONFIG_BTRFS_FS_REF_VERIFY) += ref-verify.o
+btrfs-$(CONFIG_BLK_DEV_ZONED) += zoned.o
 
 btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
 	tests/extent-buffer-tests.o tests/btrfs-tests.o \
fs/btrfs/dev-replace.c +5 -0
index 20ce1970015f..6f6d77224c2b 100644
@@ -21,6 +21,7 @@
 #include "rcu-string.h"
 #include "dev-replace.h"
 #include "sysfs.h"
+#include "zoned.h"
 
 /*
  * Device replace overview
@@ -291,6 +292,10 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info,
 	set_blocksize(device->bdev, BTRFS_BDEV_BLOCKSIZE);
 	device->fs_devices = fs_info->fs_devices;
 
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret)
+		goto error;
+
 	mutex_lock(&fs_info->fs_devices->device_list_mutex);
 	list_add(&device->dev_list, &fs_info->fs_devices->devices);
 	fs_info->fs_devices->num_devices++;
fs/btrfs/super.c +5 -0
index 8840a4fa81eb..ed55014fd1bd 100644
@@ -2462,6 +2462,11 @@ static void __init btrfs_print_mod_info(void)
 #endif
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 			", ref-verify=on"
+#endif
+#ifdef CONFIG_BLK_DEV_ZONED
+			", zoned=yes"
+#else
+			", zoned=no"
 #endif
 			;
 	pr_info("Btrfs loaded, crc32c=%s%s\n", crc32c_impl(), options);
fs/btrfs/volumes.c +17 -2
index 58b9c419a2b6..e787bf89f761 100644
@@ -31,6 +31,7 @@
 #include "space-info.h"
 #include "block-group.h"
 #include "discard.h"
+#include "zoned.h"
 
 const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
 	[BTRFS_RAID_RAID10] = {
@@ -374,6 +375,7 @@ void btrfs_free_device(struct btrfs_device *device)
 	rcu_string_free(device->name);
 	extent_io_tree_release(&device->alloc_state);
 	bio_put(device->flush_bio);
+	btrfs_destroy_dev_zone_info(device);
 	kfree(device);
 }
 
@@ -667,6 +669,11 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
 	clear_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
 	device->mode = flags;
 
+	/* Get zone type information of zoned block devices */
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret != 0)
+		goto error_free_page;
+
 	fs_devices->open_devices++;
 	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
 	    device->devid != BTRFS_DEV_REPLACE_DEVID) {
@@ -1143,6 +1150,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 		device->bdev = NULL;
 	}
 	clear_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state);
+	btrfs_destroy_dev_zone_info(device);
 
 	device->fs_info = NULL;
 	atomic_set(&device->dev_stats_ccnt, 0);
@@ -2543,6 +2551,14 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	}
 	rcu_assign_pointer(device->name, name);
 
+	device->fs_info = fs_info;
+	device->bdev = bdev;
+
+	/* Get zone type information of zoned block devices */
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret)
+		goto error_free_device;
+
 	trans = btrfs_start_transaction(root, 0);
 	if (IS_ERR(trans)) {
 		ret = PTR_ERR(trans);
@@ -2559,8 +2575,6 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 					 fs_info->sectorsize);
 	device->disk_total_bytes = device->total_bytes;
 	device->commit_total_bytes = device->total_bytes;
-	device->fs_info = fs_info;
-	device->bdev = bdev;
 	set_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
 	clear_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state);
 	device->mode = FMODE_EXCL;
@@ -2707,6 +2721,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 		sb->s_flags |= SB_RDONLY;
 	if (trans)
 		btrfs_end_transaction(trans);
+	btrfs_destroy_dev_zone_info(device);
 error_free_device:
 	btrfs_free_device(device);
 error:
fs/btrfs/volumes.h +4 -0
index bf27ac07d315..9c07b97a2260 100644
@@ -51,6 +51,8 @@ struct btrfs_io_geometry {
 #define BTRFS_DEV_STATE_REPLACE_TGT	(3)
 #define BTRFS_DEV_STATE_FLUSH_SENT	(4)
 
+struct btrfs_zoned_device_info;
+
 struct btrfs_device {
 	struct list_head dev_list; /* device_list_mutex */
 	struct list_head dev_alloc_list; /* chunk mutex */
@@ -64,6 +66,8 @@ struct btrfs_device {
 
 	struct block_device *bdev;
 
+	struct btrfs_zoned_device_info *zone_info;
+
 	/* the mode sent to blkdev_get */
 	fmode_t mode;
 
fs/btrfs/zoned.c new file +176 -0

176 lines not on this page · raw message

fs/btrfs/zoned.h new file +86 -0

86 lines not on this page · raw message

Signature · 1 line
2.27.0
#8 lore raw

Re: [PATCH v9 04/41] btrfs: Get zone information of zoned block devices

Quoted 300 lines · Naohiro Aota

If a zoned block device is found, get its zone information (number of zones and zone size) using the new helper function btrfs_get_dev_zone_info(). To avoid costly run-time zone report commands to test the device zones type during block allocation, attach the seq_zones bitmap to the device structure to indicate if a zone is sequential or accept random writes. Also it attaches the empty_zones bitmap to indicate if a zone is empty or not.

This patch also introduces the helper function btrfs_dev_is_sequential() to test if the zone storing a block is a sequential write required zone and btrfs_dev_is_empty_zone() to test if the zone is a empty zone.

---
  fs/btrfs/Makefile      |   1 +
  fs/btrfs/dev-replace.c |   5 ++
  fs/btrfs/super.c       |   5 ++
  fs/btrfs/volumes.c     |  19 ++++-
  fs/btrfs/volumes.h     |   4 +
  fs/btrfs/zoned.c       | 176 +++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/zoned.h       |  86 ++++++++++++++++++++
  7 files changed, 294 insertions(+), 2 deletions(-)

create mode 100644 fs/btrfs/zoned.c
create mode 100644 fs/btrfs/zoned.h

fs/btrfs/Makefile +1 -0
index e738f6206ea5..0497fdc37f90 100644
@@ -16,6 +16,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
  btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
  btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
  btrfs-$(CONFIG_BTRFS_FS_REF_VERIFY) += ref-verify.o
+btrfs-$(CONFIG_BLK_DEV_ZONED) += zoned.o
 
  btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
  	tests/extent-buffer-tests.o tests/btrfs-tests.o \
fs/btrfs/dev-replace.c +5 -0
index 20ce1970015f..6f6d77224c2b 100644
@@ -21,6 +21,7 @@
  #include "rcu-string.h"
  #include "dev-replace.h"
  #include "sysfs.h"
+#include "zoned.h"
 
  /*
   * Device replace overview
@@ -291,6 +292,10 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info,
  	set_blocksize(device->bdev, BTRFS_BDEV_BLOCKSIZE);
  	device->fs_devices = fs_info->fs_devices;
 
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret)
+		goto error;
+
  	mutex_lock(&fs_info->fs_devices->device_list_mutex);
  	list_add(&device->dev_list, &fs_info->fs_devices->devices);
  	fs_info->fs_devices->num_devices++;
fs/btrfs/super.c +5 -0
index 8840a4fa81eb..ed55014fd1bd 100644
@@ -2462,6 +2462,11 @@ static void __init btrfs_print_mod_info(void)
  #endif
  #ifdef CONFIG_BTRFS_FS_REF_VERIFY
  			", ref-verify=on"
+#endif
+#ifdef CONFIG_BLK_DEV_ZONED
+			", zoned=yes"
+#else
+			", zoned=no"
  #endif
  			;
  	pr_info("Btrfs loaded, crc32c=%s%s\n", crc32c_impl(), options);
fs/btrfs/volumes.c +17 -2
index 58b9c419a2b6..e787bf89f761 100644
@@ -31,6 +31,7 @@
  #include "space-info.h"
  #include "block-group.h"
  #include "discard.h"
+#include "zoned.h"
 
  const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
  	[BTRFS_RAID_RAID10] = {
@@ -374,6 +375,7 @@ void btrfs_free_device(struct btrfs_device *device)
  	rcu_string_free(device->name);
  	extent_io_tree_release(&device->alloc_state);
  	bio_put(device->flush_bio);
+	btrfs_destroy_dev_zone_info(device);
  	kfree(device);
  }
 
@@ -667,6 +669,11 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
  	clear_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
  	device->mode = flags;
 
+	/* Get zone type information of zoned block devices */
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret != 0)
+		goto error_free_page;
+
  	fs_devices->open_devices++;
  	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
  	    device->devid != BTRFS_DEV_REPLACE_DEVID) {
@@ -1143,6 +1150,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
  		device->bdev = NULL;
  	}
  	clear_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state);
+	btrfs_destroy_dev_zone_info(device);
 
  	device->fs_info = NULL;
  	atomic_set(&device->dev_stats_ccnt, 0);
@@ -2543,6 +2551,14 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
  	}
  	rcu_assign_pointer(device->name, name);
 
+	device->fs_info = fs_info;
+	device->bdev = bdev;
+
+	/* Get zone type information of zoned block devices */
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret)
+		goto error_free_device;
+
  	trans = btrfs_start_transaction(root, 0);
  	if (IS_ERR(trans)) {
  		ret = PTR_ERR(trans);
@@ -2559,8 +2575,6 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
  					 fs_info->sectorsize);
  	device->disk_total_bytes = device->total_bytes;
  	device->commit_total_bytes = device->total_bytes;
-	device->fs_info = fs_info;
-	device->bdev = bdev;
  	set_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
  	clear_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state);
  	device->mode = FMODE_EXCL;
@@ -2707,6 +2721,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
  		sb->s_flags |= SB_RDONLY;
  	if (trans)
  		btrfs_end_transaction(trans);
+	btrfs_destroy_dev_zone_info(device);
  error_free_device:
  	btrfs_free_device(device);
  error:
fs/btrfs/volumes.h +4 -0
index bf27ac07d315..9c07b97a2260 100644
@@ -51,6 +51,8 @@ struct btrfs_io_geometry {
  #define BTRFS_DEV_STATE_REPLACE_TGT	(3)
  #define BTRFS_DEV_STATE_FLUSH_SENT	(4)
 
+struct btrfs_zoned_device_info;
+
  struct btrfs_device {
  	struct list_head dev_list; /* device_list_mutex */
  	struct list_head dev_alloc_list; /* chunk mutex */
@@ -64,6 +66,8 @@ struct btrfs_device {
 
  	struct block_device *bdev;
 
+	struct btrfs_zoned_device_info *zone_info;
+
  	/* the mode sent to blkdev_get */
  	fmode_t mode;
 
fs/btrfs/zoned.c new file +123 -0
new file mode 100644
index 000000000000..5657d654bc44
@@ -0,0 +1,176 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/slab.h>
+#include <linux/blkdev.h>
+#include "ctree.h"
+#include "volumes.h"
+#include "zoned.h"
+#include "rcu-string.h"
+
+/* Maximum number of zones to report per blkdev_report_zones() call */
+#define BTRFS_REPORT_NR_ZONES   4096
+
+static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
+			     void *data)
+{
+	struct blk_zone *zones = data;
+
+	memcpy(&zones[idx], zone, sizeof(*zone));
+
+	return 0;
+}
+
+static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
+			       struct blk_zone *zones, unsigned int *nr_zones)
+{
+	int ret;
+
+	if (!*nr_zones)
+		return 0;
+
+	ret = blkdev_report_zones(device->bdev, pos >> SECTOR_SHIFT, *nr_zones,
+				  copy_zone_info_cb, zones);
+	if (ret < 0) {
+		btrfs_err_in_rcu(device->fs_info,
+				 "get zone at %llu on %s failed %d", pos,
+				 rcu_str_deref(device->name), ret);
+		return ret;
+	}
+	*nr_zones = ret;
+	if (!ret)
+		return -EIO;
+
+	return 0;
+}
+
+int btrfs_get_dev_zone_info(struct btrfs_device *device)
+{
+	struct btrfs_zoned_device_info *zone_info = NULL;
+	struct block_device *bdev = device->bdev;
+	sector_t nr_sectors = bdev->bd_part->nr_sects;
+	sector_t sector = 0;
+	struct blk_zone *zones = NULL;
+	unsigned int i, nreported = 0, nr_zones;
+	unsigned int zone_sectors;
+	int ret;
+	char devstr[sizeof(device->fs_info->sb->s_id) +
+		    sizeof(" (device )") - 1];
+
+	if (!bdev_is_zoned(bdev))
+		return 0;
+
+	if (device->zone_info)
+		return 0;
+
+	zone_info = kzalloc(sizeof(*zone_info), GFP_KERNEL);
+	if (!zone_info)
+		return -ENOMEM;
+
+	zone_sectors = bdev_zone_sectors(bdev);
+	ASSERT(is_power_of_2(zone_sectors));
+	zone_info->zone_size = (u64)zone_sectors << SECTOR_SHIFT;
+	zone_info->zone_size_shift = ilog2(zone_info->zone_size);
+	zone_info->nr_zones = nr_sectors >> ilog2(bdev_zone_sectors(bdev));
+	if (!IS_ALIGNED(nr_sectors, zone_sectors))
+		zone_info->nr_zones++;
+
+	zone_info->seq_zones = bitmap_zalloc(zone_info->nr_zones, GFP_KERNEL);
+	if (!zone_info->seq_zones) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	zone_info->empty_zones = bitmap_zalloc(zone_info->nr_zones, GFP_KERNEL);
+	if (!zone_info->empty_zones) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	zones = kcalloc(BTRFS_REPORT_NR_ZONES,
+			sizeof(struct blk_zone), GFP_KERNEL);
+	if (!zones) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* Get zones type */
+	while (sector < nr_sectors) {
+		nr_zones = BTRFS_REPORT_NR_ZONES;
+		ret = btrfs_get_dev_zones(device, sector << SECTOR_SHIFT, zones,
+					  &nr_zones);
+		if (ret)
+			goto out;
+
+		for (i = 0; i < nr_zones; i++) {
+			if (zones[i].type == BLK_ZONE_TYPE_SEQWRITE_REQ)
+				set_bit(nreported, zone_info->seq_zones);
+			if (zones[i].cond == BLK_ZONE_COND_EMPTY)
+				set_bit(nreported, zone_info->empty_zones);
+			nreported++;
+		}
+		sector = zones[nr_zones - 1].start + zones[nr_zones - 1].len;
+	}
+
+	if (nreported != zone_info->nr_zones) {
+		btrfs_err_in_rcu(device->fs_info,
+				 "inconsistent number of zones on %s (%u / %u)",
+				 rcu_str_deref(device->name), nreported,
+				 zone_info->nr_zones);
+		ret = -EIO;
+		goto out;
+	}
+
+	kfree(zones);

fs/btrfs/zoned.c line 133

Quoted 10 lines
fs/btrfs/zoned.c new file +10 -0
+
+	device->zone_info = zone_info;
+
+	devstr[0] = 0;
+	if (device->fs_info)
+		snprintf(devstr, sizeof(devstr), " (device %s)",
+			 device->fs_info->sb->s_id);
+
+	rcu_read_lock();
+	pr_info(

Why isn't this btrfs_info() ? Thanks,

Josef

#9 lore raw

Re: [PATCH v9 04/41] btrfs: Get zone information of zoned block devices

Quoted 1 line · Josef Bacik

Why isn't this btrfs_info() ?

Plain oversight...

#10 lore raw

Re: [PATCH v9 04/41] btrfs: Get zone information of zoned block devices

In reply to Josef Bacik
Quoted 293 lines · Josef Bacik
Quoted 292 lines · Naohiro Aota

If a zoned block device is found, get its zone information (number of zones and zone size) using the new helper function btrfs_get_dev_zone_info(). To avoid costly run-time zone report commands to test the device zones type during block allocation, attach the seq_zones bitmap to the device structure to indicate if a zone is sequential or accept random writes. Also it attaches the empty_zones bitmap to indicate if a zone is empty or not.

This patch also introduces the helper function btrfs_dev_is_sequential() to test if the zone storing a block is a sequential write required zone and btrfs_dev_is_empty_zone() to test if the zone is a empty zone.

---
 fs/btrfs/Makefile      |   1 +
 fs/btrfs/dev-replace.c |   5 ++
 fs/btrfs/super.c       |   5 ++
 fs/btrfs/volumes.c     |  19 ++++-
 fs/btrfs/volumes.h     |   4 +
 fs/btrfs/zoned.c       | 176 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |  86 ++++++++++++++++++++
 7 files changed, 294 insertions(+), 2 deletions(-)
 create mode 100644 fs/btrfs/zoned.c
 create mode 100644 fs/btrfs/zoned.h
fs/btrfs/Makefile +1 -0
index e738f6206ea5..0497fdc37f90 100644
@@ -16,6 +16,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
 btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
 btrfs-$(CONFIG_BTRFS_FS_REF_VERIFY) += ref-verify.o
+btrfs-$(CONFIG_BLK_DEV_ZONED) += zoned.o
 btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
 	tests/extent-buffer-tests.o tests/btrfs-tests.o \
fs/btrfs/dev-replace.c +5 -0
index 20ce1970015f..6f6d77224c2b 100644
@@ -21,6 +21,7 @@
 #include "rcu-string.h"
 #include "dev-replace.h"
 #include "sysfs.h"
+#include "zoned.h"
 /*
  * Device replace overview
@@ -291,6 +292,10 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info,
 	set_blocksize(device->bdev, BTRFS_BDEV_BLOCKSIZE);
 	device->fs_devices = fs_info->fs_devices;
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret)
+		goto error;
+
 	mutex_lock(&fs_info->fs_devices->device_list_mutex);
 	list_add(&device->dev_list, &fs_info->fs_devices->devices);
 	fs_info->fs_devices->num_devices++;
fs/btrfs/super.c +5 -0
index 8840a4fa81eb..ed55014fd1bd 100644
@@ -2462,6 +2462,11 @@ static void __init btrfs_print_mod_info(void)
 #endif
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 			", ref-verify=on"
+#endif
+#ifdef CONFIG_BLK_DEV_ZONED
+			", zoned=yes"
+#else
+			", zoned=no"
 #endif
 			;
 	pr_info("Btrfs loaded, crc32c=%s%s\n", crc32c_impl(), options);
fs/btrfs/volumes.c +17 -2
index 58b9c419a2b6..e787bf89f761 100644
@@ -31,6 +31,7 @@
 #include "space-info.h"
 #include "block-group.h"
 #include "discard.h"
+#include "zoned.h"
 const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
 	[BTRFS_RAID_RAID10] = {
@@ -374,6 +375,7 @@ void btrfs_free_device(struct btrfs_device *device)
 	rcu_string_free(device->name);
 	extent_io_tree_release(&device->alloc_state);
 	bio_put(device->flush_bio);
+	btrfs_destroy_dev_zone_info(device);
 	kfree(device);
 }
@@ -667,6 +669,11 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
 	clear_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
 	device->mode = flags;
+	/* Get zone type information of zoned block devices */
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret != 0)
+		goto error_free_page;
+
 	fs_devices->open_devices++;
 	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
 	    device->devid != BTRFS_DEV_REPLACE_DEVID) {
@@ -1143,6 +1150,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 		device->bdev = NULL;
 	}
 	clear_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state);
+	btrfs_destroy_dev_zone_info(device);
 	device->fs_info = NULL;
 	atomic_set(&device->dev_stats_ccnt, 0);
@@ -2543,6 +2551,14 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	}
 	rcu_assign_pointer(device->name, name);
+	device->fs_info = fs_info;
+	device->bdev = bdev;
+
+	/* Get zone type information of zoned block devices */
+	ret = btrfs_get_dev_zone_info(device);
+	if (ret)
+		goto error_free_device;
+
 	trans = btrfs_start_transaction(root, 0);
 	if (IS_ERR(trans)) {
 		ret = PTR_ERR(trans);
@@ -2559,8 +2575,6 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 					 fs_info->sectorsize);
 	device->disk_total_bytes = device->total_bytes;
 	device->commit_total_bytes = device->total_bytes;
-	device->fs_info = fs_info;
-	device->bdev = bdev;
 	set_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
 	clear_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state);
 	device->mode = FMODE_EXCL;
@@ -2707,6 +2721,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 		sb->s_flags |= SB_RDONLY;
 	if (trans)
 		btrfs_end_transaction(trans);
+	btrfs_destroy_dev_zone_info(device);
 error_free_device:
 	btrfs_free_device(device);
 error:
fs/btrfs/volumes.h +4 -0
index bf27ac07d315..9c07b97a2260 100644
@@ -51,6 +51,8 @@ struct btrfs_io_geometry {
 #define BTRFS_DEV_STATE_REPLACE_TGT	(3)
 #define BTRFS_DEV_STATE_FLUSH_SENT	(4)
+struct btrfs_zoned_device_info;
+
 struct btrfs_device {
 	struct list_head dev_list; /* device_list_mutex */
 	struct list_head dev_alloc_list; /* chunk mutex */
@@ -64,6 +66,8 @@ struct btrfs_device {
 	struct block_device *bdev;
+	struct btrfs_zoned_device_info *zone_info;
+
 	/* the mode sent to blkdev_get */
 	fmode_t mode;
fs/btrfs/zoned.c new file +127 -0
new file mode 100644
index 000000000000..5657d654bc44
@@ -0,0 +1,176 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/slab.h>
+#include <linux/blkdev.h>
+#include "ctree.h"
+#include "volumes.h"
+#include "zoned.h"
+#include "rcu-string.h"
+
+/* Maximum number of zones to report per blkdev_report_zones() call */
+#define BTRFS_REPORT_NR_ZONES   4096
+
+static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
+			     void *data)
+{
+	struct blk_zone *zones = data;
+
+	memcpy(&zones[idx], zone, sizeof(*zone));
+
+	return 0;
+}
+
+static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
+			       struct blk_zone *zones, unsigned int *nr_zones)
+{
+	int ret;
+
+	if (!*nr_zones)
+		return 0;
+
+	ret = blkdev_report_zones(device->bdev, pos >> SECTOR_SHIFT, *nr_zones,
+				  copy_zone_info_cb, zones);
+	if (ret < 0) {
+		btrfs_err_in_rcu(device->fs_info,
+				 "get zone at %llu on %s failed %d", pos,
+				 rcu_str_deref(device->name), ret);
+		return ret;
+	}
+	*nr_zones = ret;
+	if (!ret)
+		return -EIO;
+
+	return 0;
+}
+
+int btrfs_get_dev_zone_info(struct btrfs_device *device)
+{
+	struct btrfs_zoned_device_info *zone_info = NULL;
+	struct block_device *bdev = device->bdev;
+	sector_t nr_sectors = bdev->bd_part->nr_sects;
+	sector_t sector = 0;
+	struct blk_zone *zones = NULL;
+	unsigned int i, nreported = 0, nr_zones;
+	unsigned int zone_sectors;
+	int ret;
+	char devstr[sizeof(device->fs_info->sb->s_id) +
+		    sizeof(" (device )") - 1];
+
+	if (!bdev_is_zoned(bdev))
+		return 0;
+
+	if (device->zone_info)
+		return 0;
+
+	zone_info = kzalloc(sizeof(*zone_info), GFP_KERNEL);
+	if (!zone_info)
+		return -ENOMEM;
+
+	zone_sectors = bdev_zone_sectors(bdev);
+	ASSERT(is_power_of_2(zone_sectors));
+	zone_info->zone_size = (u64)zone_sectors << SECTOR_SHIFT;
+	zone_info->zone_size_shift = ilog2(zone_info->zone_size);
+	zone_info->nr_zones = nr_sectors >> ilog2(bdev_zone_sectors(bdev));
+	if (!IS_ALIGNED(nr_sectors, zone_sectors))
+		zone_info->nr_zones++;
+
+	zone_info->seq_zones = bitmap_zalloc(zone_info->nr_zones, GFP_KERNEL);
+	if (!zone_info->seq_zones) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	zone_info->empty_zones = bitmap_zalloc(zone_info->nr_zones, GFP_KERNEL);
+	if (!zone_info->empty_zones) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	zones = kcalloc(BTRFS_REPORT_NR_ZONES,
+			sizeof(struct blk_zone), GFP_KERNEL);
+	if (!zones) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* Get zones type */
+	while (sector < nr_sectors) {
+		nr_zones = BTRFS_REPORT_NR_ZONES;
+		ret = btrfs_get_dev_zones(device, sector << SECTOR_SHIFT, zones,
+					  &nr_zones);
+		if (ret)
+			goto out;
+
+		for (i = 0; i < nr_zones; i++) {
+			if (zones[i].type == BLK_ZONE_TYPE_SEQWRITE_REQ)
+				set_bit(nreported, zone_info->seq_zones);
+			if (zones[i].cond == BLK_ZONE_COND_EMPTY)
+				set_bit(nreported, zone_info->empty_zones);
+			nreported++;
+		}
+		sector = zones[nr_zones - 1].start + zones[nr_zones - 1].len;
+	}
+
+	if (nreported != zone_info->nr_zones) {
+		btrfs_err_in_rcu(device->fs_info,
+				 "inconsistent number of zones on %s (%u / %u)",
+				 rcu_str_deref(device->name), nreported,
+				 zone_info->nr_zones);
+		ret = -EIO;
+		goto out;
+	}
+
+	kfree(zones);
+
+	device->zone_info = zone_info;
+
+	devstr[0] = 0;
Quoted 8 lines

fs/btrfs/zoned.c line 133

Quoted 6 lines
fs/btrfs/zoned.c new file +6 -0
+	if (device->fs_info)
+		snprintf(devstr, sizeof(devstr), " (device %s)",
+			 device->fs_info->sb->s_id);
+
+	rcu_read_lock();
+	pr_info(

Why isn't this btrfs_info() ? Thanks,

Because, we might not have device->fs_info set at this point. At the mount time, this function is called under btrfs_open_devices() in btrfs_mount_root(). OTOH, device->fs_info is set under open_ctree() which is called later than btrfs_open_devices().

Using btrfs_info() here results in printing "BTRFS info (device <unknown>):
..." at the mount time, which is an annoying look in my opinion. So, I followed the "scanned by ..." message style in device_list_add().

I'll add a description comment here.

#11 lore raw

Re: [PATCH v9 04/41] btrfs: Get zone information of zoned block devices

In reply to Naohiro Aota

fs/btrfs/zoned.c line 35

Quoted 13 lines · Naohiro Aota
fs/btrfs/zoned.c new file +13 -0
+static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
+			       struct blk_zone *zones, unsigned int *nr_zones)
+{
+	int ret;
+
+	if (!*nr_zones)
+		return 0;
+
+	ret = blkdev_report_zones(device->bdev, pos >> SECTOR_SHIFT, *nr_zones,
+				  copy_zone_info_cb, zones);
+	if (ret < 0) {
+		btrfs_err_in_rcu(device->fs_info,
+				 "get zone at %llu on %s failed %d", pos,

"zoned: failed to read zone %llu on %s (devid %llu)"

Quoted 12 lines
fs/btrfs/zoned.c new file +12 -0
+				 rcu_str_deref(device->name), ret);
+		return ret;
+	}
+	*nr_zones = ret;
+	if (!ret)
+		return -EIO;
+
+	return 0;
+}
+
+int btrfs_get_dev_zone_info(struct btrfs_device *device)
+{

fs/btrfs/zoned.c line 57

Quoted 10 lines
fs/btrfs/zoned.c new file +10 -0
+	struct btrfs_zoned_device_info *zone_info = NULL;
+	struct block_device *bdev = device->bdev;
+	sector_t nr_sectors = bdev->bd_part->nr_sects;
+	sector_t sector = 0;
+	struct blk_zone *zones = NULL;
+	unsigned int i, nreported = 0, nr_zones;
+	unsigned int zone_sectors;
+	int ret;
+	char devstr[sizeof(device->fs_info->sb->s_id) +
+		    sizeof(" (device )") - 1];

Can you avoid the local variable and print the contents conditionally?

fs/btrfs/zoned.c line 134

Quoted 12 lines
fs/btrfs/zoned.c new file +12 -0
+	kfree(zones);
+
+	device->zone_info = zone_info;
+
+	devstr[0] = 0;
+	if (device->fs_info)
+		snprintf(devstr, sizeof(devstr), " (device %s)",
+			 device->fs_info->sb->s_id);
+
+	rcu_read_lock();
+	pr_info(
+"BTRFS info%s: host-%s zoned block device %s, %u zones of %llu sectors",

I think zone size in bytes is more natural, is there a reason to print that in sectors?

Quoted 20 lines
fs/btrfs/zoned.c new file +16 -0
+		devstr,
+		bdev_zoned_model(bdev) == BLK_ZONED_HM ? "managed" : "aware",
+		rcu_str_deref(device->name), zone_info->nr_zones,
+		zone_info->zone_size >> SECTOR_SHIFT);
+	rcu_read_unlock();
+
+	return 0;
+
+out:
+	kfree(zones);
+	bitmap_free(zone_info->empty_zones);
+	bitmap_free(zone_info->seq_zones);
+	kfree(zone_info);
+
+	return ret;
+}

fs/btrfs/zoned.h line 4

Quoted 4 lines
fs/btrfs/zoned.h new file +4 -0
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef BTRFS_ZONED_H
+#define BTRFS_ZONED_H

This should at least include types.h as it's using u64

Quoted 14 lines
fs/btrfs/zoned.h new file +14 -0
+
+struct btrfs_zoned_device_info {
+	/*
+	 * Number of zones, zone size and types of zones if bdev is a
+	 * zoned block device.
+	 */
+	u64 zone_size;
+	u8  zone_size_shift;
+	u32 nr_zones;
+	unsigned long *seq_zones;
+	unsigned long *empty_zones;
+};
+
+#ifdef CONFIG_BLK_DEV_ZONED

fs/btrfs/zoned.h line 28

Quoted 10 lines
fs/btrfs/zoned.h new file +10 -0
+int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
+		       struct blk_zone *zone);
+int btrfs_get_dev_zone_info(struct btrfs_device *device);
+void btrfs_destroy_dev_zone_info(struct btrfs_device *device);
+#else /* CONFIG_BLK_DEV_ZONED */
+static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
+				     struct blk_zone *zone)
+{
+	return 0;
+}

newline

fs/btrfs/zoned.h line 32

Quoted 4 lines
fs/btrfs/zoned.h new file +4 -0
+static inline int btrfs_get_dev_zone_info(struct btrfs_device *device)
+{
+	return 0;
+}

newline

Quoted 2 lines
fs/btrfs/zoned.h new file +2 -0
+static inline void btrfs_destroy_dev_zone_info(struct btrfs_device *device) { }
+#endif
#12 lore raw

PATCH v9 5/41 btrfs: Check and enable ZONED mode

In reply to Naohiro Aota

This commit introduces the function btrfs_check_zoned_mode() to check if ZONED flag is enabled on the file system and if the file system consists of zoned devices with equal zone size.

---
 fs/btrfs/ctree.h       | 10 ++++++
 fs/btrfs/dev-replace.c |  7 ++++
 fs/btrfs/disk-io.c     | 11 ++++++
 fs/btrfs/super.c       |  1 +
 fs/btrfs/volumes.c     |  5 +++
 fs/btrfs/zoned.c       | 78 ++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       | 26 ++++++++++++++
 7 files changed, 138 insertions(+)
fs/btrfs/ctree.h +10 -0
index aac3d6f4e35b..25fd4e97dd2a 100644
@@ -948,6 +948,12 @@ struct btrfs_fs_info {
 	/* Type of exclusive operation running */
 	unsigned long exclusive_operation;
 
+	/* Zone size when in ZONED mode */
+	union {
+		u64 zone_size;
+		u64 zoned;
+	};
+
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 	spinlock_t ref_verify_lock;
 	struct rb_root block_tree;
@@ -3595,4 +3601,8 @@ static inline int btrfs_is_testing(struct btrfs_fs_info *fs_info)
 }
 #endif
 
+static inline bool btrfs_is_zoned(struct btrfs_fs_info *fs_info)
+{
+	return fs_info->zoned != 0;
+}
 #endif
fs/btrfs/dev-replace.c +7 -0
index 6f6d77224c2b..5e3554482af1 100644
@@ -238,6 +238,13 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info,
 		return PTR_ERR(bdev);
 	}
 
+	if (!btrfs_check_device_zone_type(fs_info, bdev)) {
+		btrfs_err(fs_info,
+			  "zone type of target device mismatch with the filesystem!");
+		ret = -EINVAL;
+		goto error;
+	}
+
 	sync_blockdev(bdev);
 
 	list_for_each_entry(device, &fs_info->fs_devices->devices, dev_list) {
fs/btrfs/disk-io.c +11 -0
index 764001609a15..9bc51cff48b8 100644
@@ -42,6 +42,7 @@
 #include "block-group.h"
 #include "discard.h"
 #include "space-info.h"
+#include "zoned.h"
 
 #define BTRFS_SUPER_FLAG_SUPP	(BTRFS_HEADER_FLAG_WRITTEN |\
 				 BTRFS_HEADER_FLAG_RELOC |\
@@ -2976,6 +2977,8 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 	if (features & BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA)
 		btrfs_info(fs_info, "has skinny extents");
 
+	fs_info->zoned = features & BTRFS_FEATURE_INCOMPAT_ZONED;
+
 	/*
 	 * flag our filesystem as having big metadata blocks if
 	 * they are bigger than the page size
@@ -3130,7 +3133,15 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 
 	btrfs_free_extra_devids(fs_devices, 1);
 
+	ret = btrfs_check_zoned_mode(fs_info);
+	if (ret) {
+		btrfs_err(fs_info, "failed to init ZONED mode: %d",
+				ret);
+		goto fail_block_groups;
+	}
+
 	ret = btrfs_sysfs_add_fsid(fs_devices);
+
 	if (ret) {
 		btrfs_err(fs_info, "failed to init sysfs fsid interface: %d",
 				ret);
fs/btrfs/super.c +1 -0
index ed55014fd1bd..3312fe08168f 100644
@@ -44,6 +44,7 @@
 #include "backref.h"
 #include "space-info.h"
 #include "sysfs.h"
+#include "zoned.h"
 #include "tests/btrfs-tests.h"
 #include "block-group.h"
 #include "discard.h"
fs/btrfs/volumes.c +5 -0
index e787bf89f761..10827892c086 100644
@@ -2518,6 +2518,11 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	if (IS_ERR(bdev))
 		return PTR_ERR(bdev);
 
+	if (!btrfs_check_device_zone_type(fs_info, bdev)) {
+		ret = -EINVAL;
+		goto error;
+	}
+
 	if (fs_devices->seeding) {
 		seeding_dev = 1;
 		down_write(&sb->s_umount);
fs/btrfs/zoned.c +78 -0

81 lines not on this page · raw message

fs/btrfs/zoned.h +26 -0
index 483229e27908..c4c63c4294f2 100644
@@ -3,6 +3,8 @@
 #ifndef BTRFS_ZONED_H
 #define BTRFS_ZONED_H
 
+#include <linux/blkdev.h>
+
 struct btrfs_zoned_device_info {
 	/*
 	 * Number of zones, zone size and types of zones if bdev is a
@@ -20,6 +22,7 @@ int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 		       struct blk_zone *zone);
 int btrfs_get_dev_zone_info(struct btrfs_device *device);
 void btrfs_destroy_dev_zone_info(struct btrfs_device *device);
+int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -31,6 +34,14 @@ static inline int btrfs_get_dev_zone_info(struct btrfs_device *device)
 	return 0;
 }
 static inline void btrfs_destroy_dev_zone_info(struct btrfs_device *device) { }
+static inline int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
+{
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	btrfs_err(fs_info, "Zoned block devices support is not enabled");
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
@@ -83,4 +94,19 @@ static inline void btrfs_dev_clear_zone_empty(struct btrfs_device *device,
 	btrfs_dev_set_empty_zone_bit(device, pos, false);
 }
 
+static inline bool btrfs_check_device_zone_type(struct btrfs_fs_info *fs_info,
+						struct block_device *bdev)
+{
+	u64 zone_size;
+
+	if (btrfs_is_zoned(fs_info)) {
+		zone_size = (u64)bdev_zone_sectors(bdev) << SECTOR_SHIFT;
+		/* Do not allow non-zoned device */
+		return bdev_is_zoned(bdev) && fs_info->zone_size == zone_size;
+	}
+
+	/* Do not allow Host Manged zoned device */
+	return bdev_zoned_model(bdev) != BLK_ZONED_HM;
+}
+
 #endif
Signature · 1 line
2.27.0
#13 lore raw

Re: [PATCH v9 05/41] btrfs: Check and enable ZONED mode

Below are suggested message updates, common prefix "zoned:" in case it happens inside the zone mode implementation. Some of them sound strange when repeating 'zoned', but for clarity I think it should stay, unless somebody has a better suggestion.

fs/btrfs/ctree.h static inline int btrfs_is_testing(struct btrfs_fs_info *fs_info) · line 3607

Quoted 11 lines · Naohiro Aota

index aac3d6f4e35b..25fd4e97dd2a 100644

fs/btrfs/ctree.h +4 -0
@@ -3595,4 +3601,8 @@ static inline int btrfs_is_testing(struct btrfs_fs_info *fs_info)
 }
 #endif
 
+static inline bool btrfs_is_zoned(struct btrfs_fs_info *fs_info)
+{
+	return fs_info->zoned != 0;
+}

newline

fs/btrfs/dev-replace.c static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info, · line 243

Quoted 13 lines
fs/btrfs/ctree.h +0 -0
 #endif
fs/btrfs/dev-replace.c +3 -0
index 6f6d77224c2b..5e3554482af1 100644
@@ -238,6 +238,13 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info,
 		return PTR_ERR(bdev);
 	}
 
+	if (!btrfs_check_device_zone_type(fs_info, bdev)) {
+		btrfs_err(fs_info,
+			  "zone type of target device mismatch with the filesystem!");

"dev-replace: zoned type of target device mismatch with filesystem"

Quoted 12 lines
fs/btrfs/dev-replace.c +4 -0
+		ret = -EINVAL;
+		goto error;
+	}
+
 	sync_blockdev(bdev);
 
 	list_for_each_entry(device, &fs_info->fs_devices->devices, dev_list) {
fs/btrfs/disk-io.c +0 -0
index 764001609a15..9bc51cff48b8 100644

fs/btrfs/disk-io.c int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device · line 3138

Quoted 6 lines
fs/btrfs/disk-io.c +3 -0
@@ -3130,7 +3133,15 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 
 	btrfs_free_extra_devids(fs_devices, 1);
 
+	ret = btrfs_check_zoned_mode(fs_info);
+	if (ret) {
+		btrfs_err(fs_info, "failed to init ZONED mode: %d",

"failed to inititialize zoned mode: %d"

Quoted 11 lines
fs/btrfs/disk-io.c +5 -0
+				ret);
+		goto fail_block_groups;
+	}
+
 	ret = btrfs_sysfs_add_fsid(fs_devices);
+
 	if (ret) {
 		btrfs_err(fs_info, "failed to init sysfs fsid interface: %d",
 				ret);

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 185

Quoted 3 lines
fs/btrfs/zoned.c +3 -0
+	u64 nr_devices = 0;
+	u64 zone_size = 0;
+	int incompat_zoned = btrfs_is_zoned(fs_info);

const bool

Quoted 8 lines
fs/btrfs/zoned.c +8 -0
+	int ret = 0;
+
+	/* Count zoned devices */
+	list_for_each_entry(device, &fs_devices->devices, dev_list) {
+		enum blk_zoned_model model;
+
+		if (!device->bdev)
+			continue;

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 203

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+
+		model = bdev_zoned_model(device->bdev);
+		if (model == BLK_ZONED_HM ||
+		    (model == BLK_ZONED_HA && incompat_zoned)) {
+			hmzoned_devices++;
+			if (!zone_size) {
+				zone_size = device->zone_info->zone_size;
+			} else if (device->zone_info->zone_size != zone_size) {
+				btrfs_err(fs_info,
+					  "Zoned block devices must have equal zone sizes");

"zoned: unequal block device zone sizes: have %u found %u"

Quoted 4 lines
fs/btrfs/zoned.c +4 -0
+				ret = -EINVAL;
+				goto out;
+			}
+		}

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 217

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		nr_devices++;
+	}
+
+	if (!hmzoned_devices && !incompat_zoned)
+		goto out;
+
+	if (!hmzoned_devices && incompat_zoned) {
+		/* No zoned block device found on ZONED FS */
+		btrfs_err(fs_info,
+			  "ZONED enabled file system should have zoned devices");

"zoned: no zoned devices found on a zoned filesystem"

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 224

Quoted 7 lines
fs/btrfs/zoned.c +7 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (hmzoned_devices && !incompat_zoned) {
+		btrfs_err(fs_info,
+			  "Enable ZONED mode to mount HMZONED device");

Is HMZONED reference leftover from previous iterations?

"zoned: mode not enabled but zoned device found"

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 231

Quoted 7 lines
fs/btrfs/zoned.c +7 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (hmzoned_devices != nr_devices) {
+		btrfs_err(fs_info,
+			  "zoned devices cannot be mixed with regular devices");

"zoned: cannot mix zoned and regular devices"

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 243

Quoted 12 lines
fs/btrfs/zoned.c +12 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/*
+	 * stripe_size is always aligned to BTRFS_STRIPE_LEN in
+	 * __btrfs_alloc_chunk(). Since we want stripe_len == zone_size,
+	 * check the alignment here.
+	 */
+	if (!IS_ALIGNED(zone_size, BTRFS_STRIPE_LEN)) {
+		btrfs_err(fs_info,
+			  "zone size is not aligned to BTRFS_STRIPE_LEN");

"zoned: zone size not aligned to stripe %u"

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 251

Quoted 8 lines
fs/btrfs/zoned.c +8 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	fs_info->zone_size = zone_size;
+
+	btrfs_info(fs_info, "ZONED mode enabled, zone size %llu B",
+		   fs_info->zone_size);

"zoned mode enabled with zone %llu"

fs/btrfs/zoned.h static inline int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 36

Quoted 10 lines
fs/btrfs/zoned.c +3 -0
+out:
+	return ret;
+}
fs/btrfs/zoned.h +0 -0
@@ -31,6 +34,14 @@ static inline int btrfs_get_dev_zone_info(struct btrfs_device *device)
 	return 0;
 }
 static inline void btrfs_destroy_dev_zone_info(struct btrfs_device *device) { }

newline

fs/btrfs/zoned.h static inline int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 44

Quoted 8 lines
fs/btrfs/zoned.h +8 -0
+static inline int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
+{
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	btrfs_err(fs_info, "Zoned block devices support is not enabled");
+	return -EOPNOTSUPP;
+}

newline

Quoted 3 lines
fs/btrfs/zoned.h +0 -0
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
#14 lore raw

Re: [PATCH v9 05/41] btrfs: Check and enable ZONED mode

Quoted 122 lines · David Sterba

Below are suggested message updates, common prefix "zoned:" in case it happens inside the zone mode implementation. Some of them sound strange when repeating 'zoned', but for clarity I think it should stay, unless somebody has a better suggestion.

fs/btrfs/ctree.h static inline int btrfs_is_testing(struct btrfs_fs_info *fs_info) · line 3607

Quoted 11 lines · Naohiro Aota

index aac3d6f4e35b..25fd4e97dd2a 100644

fs/btrfs/ctree.h +4 -0
@@ -3595,4 +3601,8 @@ static inline int btrfs_is_testing(struct btrfs_fs_info *fs_info)
 }
 #endif
 
+static inline bool btrfs_is_zoned(struct btrfs_fs_info *fs_info)
+{
+	return fs_info->zoned != 0;
+}

newline

fs/btrfs/dev-replace.c static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info, · line 243

Quoted 13 lines
fs/btrfs/ctree.h +0 -0
 #endif
fs/btrfs/dev-replace.c +3 -0
index 6f6d77224c2b..5e3554482af1 100644
@@ -238,6 +238,13 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info,
 		return PTR_ERR(bdev);
 	}
 
+	if (!btrfs_check_device_zone_type(fs_info, bdev)) {
+		btrfs_err(fs_info,
+			  "zone type of target device mismatch with the filesystem!");

"dev-replace: zoned type of target device mismatch with filesystem"

Quoted 12 lines
fs/btrfs/dev-replace.c +4 -0
+		ret = -EINVAL;
+		goto error;
+	}
+
 	sync_blockdev(bdev);
 
 	list_for_each_entry(device, &fs_info->fs_devices->devices, dev_list) {
fs/btrfs/disk-io.c +0 -0
index 764001609a15..9bc51cff48b8 100644

fs/btrfs/disk-io.c int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device · line 3138

Quoted 6 lines
fs/btrfs/disk-io.c +3 -0
@@ -3130,7 +3133,15 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 
 	btrfs_free_extra_devids(fs_devices, 1);
 
+	ret = btrfs_check_zoned_mode(fs_info);
+	if (ret) {
+		btrfs_err(fs_info, "failed to init ZONED mode: %d",

"failed to inititialize zoned mode: %d"

Quoted 11 lines
fs/btrfs/disk-io.c +5 -0
+				ret);
+		goto fail_block_groups;
+	}
+
 	ret = btrfs_sysfs_add_fsid(fs_devices);
+
 	if (ret) {
 		btrfs_err(fs_info, "failed to init sysfs fsid interface: %d",
 				ret);

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 185

Quoted 3 lines
fs/btrfs/zoned.c +3 -0
+	u64 nr_devices = 0;
+	u64 zone_size = 0;
+	int incompat_zoned = btrfs_is_zoned(fs_info);

const bool

Quoted 8 lines
fs/btrfs/zoned.c +8 -0
+	int ret = 0;
+
+	/* Count zoned devices */
+	list_for_each_entry(device, &fs_devices->devices, dev_list) {
+		enum blk_zoned_model model;
+
+		if (!device->bdev)
+			continue;

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 203

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+
+		model = bdev_zoned_model(device->bdev);
+		if (model == BLK_ZONED_HM ||
+		    (model == BLK_ZONED_HA && incompat_zoned)) {
+			hmzoned_devices++;
+			if (!zone_size) {
+				zone_size = device->zone_info->zone_size;
+			} else if (device->zone_info->zone_size != zone_size) {
+				btrfs_err(fs_info,
+					  "Zoned block devices must have equal zone sizes");

"zoned: unequal block device zone sizes: have %u found %u"

Quoted 4 lines
fs/btrfs/zoned.c +4 -0
+				ret = -EINVAL;
+				goto out;
+			}
+		}

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 217

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		nr_devices++;
+	}
+
+	if (!hmzoned_devices && !incompat_zoned)
+		goto out;
+
+	if (!hmzoned_devices && incompat_zoned) {
+		/* No zoned block device found on ZONED FS */
+		btrfs_err(fs_info,
+			  "ZONED enabled file system should have zoned devices");

"zoned: no zoned devices found on a zoned filesystem"

fs/btrfs/zoned.c int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos, · line 224

Quoted 7 lines
fs/btrfs/zoned.c +7 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (hmzoned_devices && !incompat_zoned) {
+		btrfs_err(fs_info,
+			  "Enable ZONED mode to mount HMZONED device");

Is HMZONED reference leftover from previous iterations?

Quoted 1 line

"zoned: mode not enabled but zoned device found"

It was intentional here to use "hmzoned", because we can technically use Host-Aware Zoned device as regular device, so I'd like to distinguish hmzoned vs hazoned. But, reading the code again, I changed my mind. We can just say "a HMZONED device" or "a HAZONED device in ZONED mode" as a zoned device. Actually, I was not counting "hazoned_devices" anyway. So, I'll rename hmzoned_devices to zoned_devices in the next version.

#15 lore raw

PATCH v9 6/41 btrfs: introduce max_zone_append_size

In reply to Naohiro Aota

The zone append write command has a maximum IO size restriction it accepts. This is because a zone append write command cannot be split, as we ask the device to place the data into a specific target zone and the device responds with the actual written location of the data.

Introduce max_zone_append_size to zone_info and fs_info to track the value, so we can limit all I/O to a zoned block device that we want to write using the zone append command to the device's limits.

---
 fs/btrfs/ctree.h |  2 ++
 fs/btrfs/zoned.c | 19 ++++++++++++++++---
 fs/btrfs/zoned.h |  1 +
 3 files changed, 19 insertions(+), 3 deletions(-)
fs/btrfs/ctree.h +2 -0
index 25fd4e97dd2a..383c83a1f5b5 100644
@@ -953,6 +953,8 @@ struct btrfs_fs_info {
 		u64 zone_size;
 		u64 zoned;
 	};
+	/* max size to emit ZONE_APPEND write command */
+	u64 max_zone_append_size;
 
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 	spinlock_t ref_verify_lock;
fs/btrfs/zoned.c +16 -3
index e1cdff5af3a3..1b42e13b8227 100644
@@ -47,6 +47,7 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
 {
 	struct btrfs_zoned_device_info *zone_info = NULL;
 	struct block_device *bdev = device->bdev;
+	struct request_queue *q = bdev_get_queue(bdev);
 	sector_t nr_sectors = bdev->bd_part->nr_sects;
 	sector_t sector = 0;
 	struct blk_zone *zones = NULL;
@@ -70,6 +71,8 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
 	ASSERT(is_power_of_2(zone_sectors));
 	zone_info->zone_size = (u64)zone_sectors << SECTOR_SHIFT;
 	zone_info->zone_size_shift = ilog2(zone_info->zone_size);
+	zone_info->max_zone_append_size =
+		(u64)queue_max_zone_append_sectors(q) << SECTOR_SHIFT;
 	zone_info->nr_zones = nr_sectors >> ilog2(bdev_zone_sectors(bdev));
 	if (!IS_ALIGNED(nr_sectors, zone_sectors))
 		zone_info->nr_zones++;
@@ -182,7 +185,8 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 	u64 hmzoned_devices = 0;
 	u64 nr_devices = 0;
 	u64 zone_size = 0;
-	int incompat_zoned = btrfs_is_zoned(fs_info);
+	u64 max_zone_append_size = 0;
+	bool incompat_zoned = btrfs_is_zoned(fs_info);
 	int ret = 0;
 
 	/* Count zoned devices */
@@ -195,15 +199,23 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 		model = bdev_zoned_model(device->bdev);
 		if (model == BLK_ZONED_HM ||
 		    (model == BLK_ZONED_HA && incompat_zoned)) {
+			struct btrfs_zoned_device_info *zone_info =
+				device->zone_info;
+
 			hmzoned_devices++;
 			if (!zone_size) {
-				zone_size = device->zone_info->zone_size;
-			} else if (device->zone_info->zone_size != zone_size) {
+				zone_size = zone_info->zone_size;
+			} else if (zone_info->zone_size != zone_size) {
 				btrfs_err(fs_info,
 					  "Zoned block devices must have equal zone sizes");
 				ret = -EINVAL;
 				goto out;
 			}
+			if (!max_zone_append_size ||
+			    (zone_info->max_zone_append_size &&
+			     zone_info->max_zone_append_size < max_zone_append_size))
+				max_zone_append_size =
+					zone_info->max_zone_append_size;
 		}
 		nr_devices++;
 	}
@@ -246,6 +258,7 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 	}
 
 	fs_info->zone_size = zone_size;
+	fs_info->max_zone_append_size = max_zone_append_size;
 
 	btrfs_info(fs_info, "ZONED mode enabled, zone size %llu B",
 		   fs_info->zone_size);
fs/btrfs/zoned.h +1 -0
index c4c63c4294f2..a63f6177f9ee 100644
@@ -12,6 +12,7 @@ struct btrfs_zoned_device_info {
 	 */
 	u64 zone_size;
 	u8  zone_size_shift;
+	u64 max_zone_append_size;
 	u32 nr_zones;
 	unsigned long *seq_zones;
 	unsigned long *empty_zones;
Signature · 1 line
2.27.0
Reviewed-by on v9 6/41
#16 lore raw

Re: [PATCH v9 06/41] btrfs: introduce max_zone_append_size

Quoted 10 lines · Naohiro Aota

The zone append write command has a maximum IO size restriction it accepts. This is because a zone append write command cannot be split, as we ask the device to place the data into a specific target zone and the device responds with the actual written location of the data.

Introduce max_zone_append_size to zone_info and fs_info to track the value, so we can limit all I/O to a zoned block device that we want to write using the zone append command to the device's limits.

Thanks,

Josef

#17 lore raw

Re: [PATCH v9 06/41] btrfs: introduce max_zone_append_size

In reply to Naohiro Aota

fs/btrfs/ctree.h struct btrfs_fs_info { · line 956

Quoted 7 lines · Naohiro Aota
fs/btrfs/ctree.h +1 -0
@@ -953,6 +953,8 @@ struct btrfs_fs_info {
 		u64 zone_size;
 		u64 zoned;
 	};
+	/* max size to emit ZONE_APPEND write command */
	/* Max size ... */

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 50

Quoted 13 lines
fs/btrfs/ctree.h +1 -0
+	u64 max_zone_append_size;
 
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 	spinlock_t ref_verify_lock;
fs/btrfs/zoned.c +1 -0
index e1cdff5af3a3..1b42e13b8227 100644
@@ -47,6 +47,7 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
 {
 	struct btrfs_zoned_device_info *zone_info = NULL;
 	struct block_device *bdev = device->bdev;
+	struct request_queue *q = bdev_get_queue(bdev);

No single letter variables (exception is 'i' for indexing)

Quoted 13 lines
fs/btrfs/zoned.c +2 -0
 	sector_t nr_sectors = bdev->bd_part->nr_sects;
 	sector_t sector = 0;
 	struct blk_zone *zones = NULL;
@@ -70,6 +71,8 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
 	ASSERT(is_power_of_2(zone_sectors));
 	zone_info->zone_size = (u64)zone_sectors << SECTOR_SHIFT;
 	zone_info->zone_size_shift = ilog2(zone_info->zone_size);
+	zone_info->max_zone_append_size =
+		(u64)queue_max_zone_append_sectors(q) << SECTOR_SHIFT;
 	zone_info->nr_zones = nr_sectors >> ilog2(bdev_zone_sectors(bdev));
 	if (!IS_ALIGNED(nr_sectors, zone_sectors))
 		zone_info->nr_zones++;

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 189

Quoted 6 lines
fs/btrfs/zoned.c +2 -1
@@ -182,7 +185,8 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 	u64 hmzoned_devices = 0;
 	u64 nr_devices = 0;
 	u64 zone_size = 0;
-	int incompat_zoned = btrfs_is_zoned(fs_info);
+	u64 max_zone_append_size = 0;
+	bool incompat_zoned = btrfs_is_zoned(fs_info);

const bool

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 205

Quoted 11 lines
fs/btrfs/zoned.c +3 -0
 	int ret = 0;
 
 	/* Count zoned devices */
@@ -195,15 +199,23 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 		model = bdev_zoned_model(device->bdev);
 		if (model == BLK_ZONED_HM ||
 		    (model == BLK_ZONED_HA && incompat_zoned)) {
+			struct btrfs_zoned_device_info *zone_info =
+				device->zone_info;
+
 			hmzoned_devices++;

I must have missed in some previous patch, 'hmzoned' should be zoned?

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 210

Quoted 7 lines
fs/btrfs/zoned.c +2 -2
 			if (!zone_size) {
-				zone_size = device->zone_info->zone_size;
-			} else if (device->zone_info->zone_size != zone_size) {
+				zone_size = zone_info->zone_size;
+			} else if (zone_info->zone_size != zone_size) {
 				btrfs_err(fs_info,
 					  "Zoned block devices must have equal zone sizes");

"zoned: devices must have equal zone sizes, have %u found %u"

Quoted 11 lines
fs/btrfs/zoned.c +5 -0
 				ret = -EINVAL;
 				goto out;
 			}
+			if (!max_zone_append_size ||
+			    (zone_info->max_zone_append_size &&
+			     zone_info->max_zone_append_size < max_zone_append_size))
+				max_zone_append_size =
+					zone_info->max_zone_append_size;
 		}
 		nr_devices++;
 	}
#18 lore raw

PATCH v9 7/41 btrfs: disallow space_cache in ZONED mode

In reply to Naohiro Aota

As updates to the space cache v1 are in-place, the space cache cannot be located over sequential zones and there is no guarantees that the device will have enough conventional zones to store this cache. Resolve this problem by disabling completely the space cache v1. This does not introduces any problems with sequential block groups: all the free space is located after the allocation pointer and no free space before the pointer. There is no need to have such cache.

Note: we can technically use free-space-tree (space cache v2) on ZONED mode. But, since ZONED mode now always allocate extents in a block group sequentially regardless of underlying device zone type, it's no use to enable and maintain the tree.

For the same reason, NODATACOW is also disabled.

Also INODE_MAP_CACHE is also disabled to avoid preallocation in the INODE_MAP_CACHE inode.

In summary, ZONED will disable:

| Disabled features | Reason                                              |
|-------------------+-----------------------------------------------------|
| RAID/Dup          | Cannot handle two zone append writes to different   |
|                   | zones                                               |
|-------------------+-----------------------------------------------------|
| space_cache (v1)  | In-place updating                                   |
| NODATACOW         | In-place updating                                   |
|-------------------+-----------------------------------------------------|
| fallocate         | Reserved extent will be a write hole                |
| INODE_MAP_CACHE   | Need pre-allocation. (and will be deprecated?)      |
|-------------------+-----------------------------------------------------|
| MIXED_BG          | Allocated metadata region will be write holes for   |
|                   | data writes                                         |
---
 fs/btrfs/super.c | 12 ++++++++++--
 fs/btrfs/zoned.c | 18 ++++++++++++++++++
 fs/btrfs/zoned.h |  5 +++++
 3 files changed, 33 insertions(+), 2 deletions(-)
fs/btrfs/super.c +10 -2
index 3312fe08168f..9064ca62b0a0 100644
@@ -525,8 +525,14 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
 	cache_gen = btrfs_super_cache_generation(info->super_copy);
 	if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
 		btrfs_set_opt(info->mount_opt, FREE_SPACE_TREE);
-	else if (cache_gen)
-		btrfs_set_opt(info->mount_opt, SPACE_CACHE);
+	else if (cache_gen) {
+		if (btrfs_is_zoned(info)) {
+			btrfs_info(info,
+			"clearring existing space cache in ZONED mode");
+			btrfs_set_super_cache_generation(info->super_copy, 0);
+		} else
+			btrfs_set_opt(info->mount_opt, SPACE_CACHE);
+	}
 
 	/*
 	 * Even the options are empty, we still need to do extra check
@@ -985,6 +991,8 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
 		ret = -EINVAL;
 
 	}
+	if (!ret)
+		ret = btrfs_check_mountopts_zoned(info);
 	if (!ret && btrfs_test_opt(info, SPACE_CACHE))
 		btrfs_info(info, "disk space caching is enabled");
 	if (!ret && btrfs_test_opt(info, FREE_SPACE_TREE))
fs/btrfs/zoned.c +18 -0
index 1b42e13b8227..3885fa327049 100644
@@ -265,3 +265,21 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 out:
 	return ret;
 }
+
+int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
+{
+	if (!btrfs_is_zoned(info))
+		return 0;
+
+	/*
+	 * SPACE CACHE writing is not CoWed. Disable that to avoid write
+	 * errors in sequential zones.
+	 */
+	if (btrfs_test_opt(info, SPACE_CACHE)) {
+		btrfs_err(info,
+			  "space cache v1 not supportted in ZONED mode");
+		return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
fs/btrfs/zoned.h +5 -0
index a63f6177f9ee..0b7756a7104d 100644
@@ -24,6 +24,7 @@ int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 int btrfs_get_dev_zone_info(struct btrfs_device *device);
 void btrfs_destroy_dev_zone_info(struct btrfs_device *device);
 int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info);
+int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -43,6 +44,10 @@ static inline int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 	btrfs_err(fs_info, "Zoned block devices support is not enabled");
 	return -EOPNOTSUPP;
 }
+static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
+{
+	return 0;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
Reviewed-by on v9 7/41
#19 lore raw

Re: [PATCH v9 07/41] btrfs: disallow space_cache in ZONED mode

Quoted 46 lines · Naohiro Aota

As updates to the space cache v1 are in-place, the space cache cannot be located over sequential zones and there is no guarantees that the device will have enough conventional zones to store this cache. Resolve this problem by disabling completely the space cache v1. This does not introduces any problems with sequential block groups: all the free space is located after the allocation pointer and no free space before the pointer. There is no need to have such cache.

Note: we can technically use free-space-tree (space cache v2) on ZONED mode. But, since ZONED mode now always allocate extents in a block group sequentially regardless of underlying device zone type, it's no use to enable and maintain the tree.

For the same reason, NODATACOW is also disabled.

Also INODE_MAP_CACHE is also disabled to avoid preallocation in the INODE_MAP_CACHE inode.

In summary, ZONED will disable:

| Disabled features | Reason                                              |
|-------------------+-----------------------------------------------------|
| RAID/Dup          | Cannot handle two zone append writes to different   |
|                   | zones                                               |
|-------------------+-----------------------------------------------------|
| space_cache (v1)  | In-place updating                                   |
| NODATACOW         | In-place updating                                   |
|-------------------+-----------------------------------------------------|
| fallocate         | Reserved extent will be a write hole                |
| INODE_MAP_CACHE   | Need pre-allocation. (and will be deprecated?)      |
|-------------------+-----------------------------------------------------|
| MIXED_BG          | Allocated metadata region will be write holes for   |
|                   | data writes                                         |
---
  fs/btrfs/super.c | 12 ++++++++++--
  fs/btrfs/zoned.c | 18 ++++++++++++++++++
  fs/btrfs/zoned.h |  5 +++++
  3 files changed, 33 insertions(+), 2 deletions(-)
fs/btrfs/super.c +0 -0
index 3312fe08168f..9064ca62b0a0 100644

fs/btrfs/super.c int btrfs_parse_options(struct btrfs_fs_info *info, char *options, · line 531

Quoted 9 lines
fs/btrfs/super.c +4 -2
@@ -525,8 +525,14 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
  	cache_gen = btrfs_super_cache_generation(info->super_copy);
  	if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
  		btrfs_set_opt(info->mount_opt, FREE_SPACE_TREE);
-	else if (cache_gen)
-		btrfs_set_opt(info->mount_opt, SPACE_CACHE);
+	else if (cache_gen) {
+		if (btrfs_is_zoned(info)) {
+			btrfs_info(info,
+			"clearring existing space cache in ZONED mode");

'clearing', and then you can add

Thanks,

Josef

#20 lore raw

Re: [PATCH v9 07/41] btrfs: disallow space_cache in ZONED mode

Quoted 1 line · Josef Bacik

'clearing', and then you can add

Fixed up, thanks

#21 lore raw

Re: [PATCH v9 07/41] btrfs: disallow space_cache in ZONED mode

In reply to Naohiro Aota
Quoted 4 lines · Naohiro Aota

INODE_MAP_CACHE inode.

In summary, ZONED will disable:

Quoted 10 lines
| Disabled features | Reason                                              |
|-------------------+-----------------------------------------------------|
| RAID/Dup          | Cannot handle two zone append writes to different   |
|                   | zones                                               |
|-------------------+-----------------------------------------------------|
| space_cache (v1)  | In-place updating                                   |
| NODATACOW         | In-place updating                                   |
|-------------------+-----------------------------------------------------|
| fallocate         | Reserved extent will be a write hole                |
| INODE_MAP_CACHE   | Need pre-allocation. (and will be deprecated?)      |

space_cache is deprecated and actually in current dev cycle (5.11)

Quoted 16 lines
|-------------------+-----------------------------------------------------|
| MIXED_BG          | Allocated metadata region will be write holes for   |
|                   | data writes                                         |
---
 fs/btrfs/super.c | 12 ++++++++++--
 fs/btrfs/zoned.c | 18 ++++++++++++++++++
 fs/btrfs/zoned.h |  5 +++++
 3 files changed, 33 insertions(+), 2 deletions(-)
fs/btrfs/super.c +0 -0
index 3312fe08168f..9064ca62b0a0 100644

fs/btrfs/super.c int btrfs_parse_options(struct btrfs_fs_info *info, char *options, · line 531

Quoted 9 lines
fs/btrfs/super.c +4 -2
@@ -525,8 +525,14 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
 	cache_gen = btrfs_super_cache_generation(info->super_copy);
 	if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
 		btrfs_set_opt(info->mount_opt, FREE_SPACE_TREE);
-	else if (cache_gen)
-		btrfs_set_opt(info->mount_opt, SPACE_CACHE);
+	else if (cache_gen) {
+		if (btrfs_is_zoned(info)) {
+			btrfs_info(info,
+			"clearring existing space cache in ZONED mode");

"zoned: clearing existing space cache"

Is it clearing or just invalidating it? We have the same problem with enabling v2 so this could share some code once Boris' patches are merged.

fs/btrfs/super.c int btrfs_parse_options(struct btrfs_fs_info *info, char *options, · line 533

Quoted 2 lines
fs/btrfs/super.c +2 -0
+			btrfs_set_super_cache_generation(info->super_copy, 0);
+		} else
		} else {

fs/btrfs/super.c int btrfs_parse_options(struct btrfs_fs_info *info, char *options, · line 534

Quoted 1 line
fs/btrfs/super.c +1 -0
+			btrfs_set_opt(info->mount_opt, SPACE_CACHE);
		}
Quoted 9 lines
fs/btrfs/super.c +1 -0
+	}
 
 	/*
 	 * Even the options are empty, we still need to do extra check
fs/btrfs/zoned.c +0 -0
@@ -265,3 +265,21 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 out:

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 275

Quoted 10 lines
fs/btrfs/zoned.c +8 -0
 	return ret;
 }
+
+int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
+{
+	if (!btrfs_is_zoned(info))
+		return 0;
+
+	/*
+	 * SPACE CACHE writing is not CoWed. Disable that to avoid write

* Space cache writing is nod COWed. ...

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 280

Quoted 5 lines
fs/btrfs/zoned.c +5 -0
+	 * errors in sequential zones.
+	 */
+	if (btrfs_test_opt(info, SPACE_CACHE)) {
+		btrfs_err(info,
+			  "space cache v1 not supportted in ZONED mode");

"zoned: space cache v1 is not supported"

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 281

Quoted 1 line
fs/btrfs/zoned.c +1 -0
+		return -EOPNOTSUPP;

This should be EINVAL, like invalid cobination. EOPNOTSUPP is for cases where it's not supported but could be if implemented.

Quoted 17 lines
fs/btrfs/zoned.c +4 -0
+	}
+
+	return 0;
+}
fs/btrfs/zoned.h +1 -0
index a63f6177f9ee..0b7756a7104d 100644
@@ -24,6 +24,7 @@ int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 int btrfs_get_dev_zone_info(struct btrfs_device *device);
 void btrfs_destroy_dev_zone_info(struct btrfs_device *device);
 int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info);
+int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)

fs/btrfs/zoned.h static inline int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 46

Quoted 3 lines
fs/btrfs/zoned.h +0 -0
@@ -43,6 +44,10 @@ static inline int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 	btrfs_err(fs_info, "Zoned block devices support is not enabled");
 	return -EOPNOTSUPP;
 }

newline

fs/btrfs/zoned.h static inline int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 50

Quoted 4 lines
fs/btrfs/zoned.h +4 -0
+static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
+{
+	return 0;
+}

newline

Quoted 3 lines
fs/btrfs/zoned.h +0 -0
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
#22 lore raw

Re: [PATCH v9 07/41] btrfs: disallow space_cache in ZONED mode

Quoted 46 lines · David Sterba
Quoted 4 lines · Naohiro Aota

INODE_MAP_CACHE inode.

In summary, ZONED will disable:

Quoted 10 lines
| Disabled features | Reason                                              |
|-------------------+-----------------------------------------------------|
| RAID/Dup          | Cannot handle two zone append writes to different   |
|                   | zones                                               |
|-------------------+-----------------------------------------------------|
| space_cache (v1)  | In-place updating                                   |
| NODATACOW         | In-place updating                                   |
|-------------------+-----------------------------------------------------|
| fallocate         | Reserved extent will be a write hole                |
| INODE_MAP_CACHE   | Need pre-allocation. (and will be deprecated?)      |

space_cache is deprecated and actually in current dev cycle (5.11)

Quoted 16 lines
|-------------------+-----------------------------------------------------|
| MIXED_BG          | Allocated metadata region will be write holes for   |
|                   | data writes                                         |
---
 fs/btrfs/super.c | 12 ++++++++++--
 fs/btrfs/zoned.c | 18 ++++++++++++++++++
 fs/btrfs/zoned.h |  5 +++++
 3 files changed, 33 insertions(+), 2 deletions(-)
fs/btrfs/super.c +0 -0
index 3312fe08168f..9064ca62b0a0 100644

fs/btrfs/super.c int btrfs_parse_options(struct btrfs_fs_info *info, char *options, · line 531

Quoted 9 lines
fs/btrfs/super.c +4 -2
@@ -525,8 +525,14 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
 	cache_gen = btrfs_super_cache_generation(info->super_copy);
 	if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
 		btrfs_set_opt(info->mount_opt, FREE_SPACE_TREE);
-	else if (cache_gen)
-		btrfs_set_opt(info->mount_opt, SPACE_CACHE);
+	else if (cache_gen) {
+		if (btrfs_is_zoned(info)) {
+			btrfs_info(info,
+			"clearring existing space cache in ZONED mode");

"zoned: clearing existing space cache"

Quoted 3 lines

Is it clearing or just invalidating it? We have the same problem with enabling v2 so this could share some code once Boris' patches are merged.

Yep, this is just invalidating it. I'll use btrfs_set_free_space_cache_v1_active here once the patches are merged.

#23 lore raw

PATCH v9 8/41 btrfs: disallow NODATACOW in ZONED mode

In reply to Naohiro Aota

NODATACOW implies overwriting the file data on a device, which is impossible in sequential required zones. Disable NODATACOW globally with mount option and per-file NODATACOW attribute by masking FS_NOCOW_FL.

---
 fs/btrfs/ioctl.c | 16 ++++++++++++++++
 fs/btrfs/zoned.c |  6 ++++++
 2 files changed, 22 insertions(+)
fs/btrfs/ioctl.c +16 -0
index ab408a23ba32..e1036a9ce881 100644
@@ -193,6 +193,18 @@ static int check_fsflags(unsigned int old_flags, unsigned int flags)
 	return 0;
 }
 
+static int check_fsflags_compatible(struct btrfs_fs_info *fs_info,
+				    unsigned int flags)
+{
+	bool zoned = btrfs_is_zoned(fs_info);
+
+	if (zoned && (flags & FS_NOCOW_FL))
+		return -EPERM;
+
+	return 0;
+}
+
+
 static int btrfs_ioctl_setflags(struct file *file, void __user *arg)
 {
 	struct inode *inode = file_inode(file);
@@ -230,6 +242,10 @@ static int btrfs_ioctl_setflags(struct file *file, void __user *arg)
 	if (ret)
 		goto out_unlock;
 
+	ret = check_fsflags_compatible(fs_info, fsflags);
+	if (ret)
+		goto out_unlock;
+
 	binode_flags = binode->flags;
 	if (fsflags & FS_SYNC_FL)
 		binode_flags |= BTRFS_INODE_SYNC;
fs/btrfs/zoned.c +6 -0
index 3885fa327049..1939b3ee6c10 100644
@@ -281,5 +281,11 @@ int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
 		return -EOPNOTSUPP;
 	}
 
+	if (btrfs_test_opt(info, NODATACOW)) {
+		btrfs_err(info,
+		  "cannot enable nodatacow with ZONED mode");
+		return -EOPNOTSUPP;
+	}
+
 	return 0;
 }
Signature · 1 line
2.27.0
Reviewed-by on v9 8/41
#24 lore raw

Re: [PATCH v9 08/41] btrfs: disallow NODATACOW in ZONED mode

Quoted 6 lines · Naohiro Aota

NODATACOW implies overwriting the file data on a device, which is impossible in sequential required zones. Disable NODATACOW globally with mount option and per-file NODATACOW attribute by masking FS_NOCOW_FL.

Thanks,

Josef

#25 lore raw

Re: [PATCH v9 08/41] btrfs: disallow NODATACOW in ZONED mode

In reply to Naohiro Aota

fs/btrfs/ioctl.c static int check_fsflags(unsigned int old_flags, unsigned int flags) · line 199

Quoted 5 lines · Naohiro Aota
fs/btrfs/ioctl.c +4 -0
+static int check_fsflags_compatible(struct btrfs_fs_info *fs_info,
+				    unsigned int flags)
+{
+	bool zoned = btrfs_is_zoned(fs_info);

No need for variable when it's one time use.

Quoted 12 lines
fs/btrfs/ioctl.c +8 -0
+
+	if (zoned && (flags & FS_NOCOW_FL))
+		return -EPERM;
+
+	return 0;
+}
+
+
 static int btrfs_ioctl_setflags(struct file *file, void __user *arg)
 {
 	struct inode *inode = file_inode(file);

fs/btrfs/ioctl.c static int btrfs_ioctl_setflags(struct file *file, void __user *arg) · line 247

Quoted 6 lines
fs/btrfs/ioctl.c +3 -0
@@ -230,6 +242,10 @@ static int btrfs_ioctl_setflags(struct file *file, void __user *arg)
 	if (ret)
 		goto out_unlock;
 
+	ret = check_fsflags_compatible(fs_info, fsflags);
+	if (ret)
+		goto out_unlock;

This is called under inode lock although it technically does not need to but I don't see a problem it logically fits to the sequence and rarely fails.

Quoted 9 lines
fs/btrfs/ioctl.c +1 -0
+
 	binode_flags = binode->flags;
 	if (fsflags & FS_SYNC_FL)
 		binode_flags |= BTRFS_INODE_SYNC;
fs/btrfs/zoned.c +0 -0
index 3885fa327049..1939b3ee6c10 100644

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 286

Quoted 6 lines
fs/btrfs/zoned.c +3 -0
@@ -281,5 +281,11 @@ int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
 		return -EOPNOTSUPP;
 	}
 
+	if (btrfs_test_opt(info, NODATACOW)) {
+		btrfs_err(info,
+		  "cannot enable nodatacow with ZONED mode");

"zoned: NODATACOW not supported"

Eventually all incompatible features will use same message style, there may be more in the patchset

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 287

Quoted 1 line
fs/btrfs/zoned.c +1 -0
+		return -EOPNOTSUPP;

Also EINVAL

Quoted 6 lines
fs/btrfs/zoned.c +2 -0
+	}
+
 	return 0;
 }
Signature · 1 line
2.27.0
#26 lore raw

PATCH v9 9/41 btrfs: disable fallocate in ZONED mode

In reply to Naohiro Aota

fallocate() is implemented by reserving actual extent instead of reservations. This can result in exposing the sequential write constraint of host-managed zoned block devices to the application, which would break the POSIX semantic for the fallocated file. To avoid this, report fallocate() as not supported when in ZONED mode for now.

In the future, we may be able to implement "in-memory" fallocate() in ZONED mode by utilizing space_info->bytes_may_use or so.

---
 fs/btrfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/btrfs/file.c +4 -0
index 0ff659455b1e..68938a43081e 100644
@@ -3341,6 +3341,10 @@ static long btrfs_fallocate(struct file *file, int mode,
 	alloc_end = round_up(offset + len, blocksize);
 	cur_offset = alloc_start;
 
+	/* Do not allow fallocate in ZONED mode */
+	if (btrfs_is_zoned(btrfs_sb(inode->i_sb)))
+		return -EOPNOTSUPP;
+
 	/* Make sure we aren't being give some crap mode */
 	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE |
 		     FALLOC_FL_ZERO_RANGE))
Signature · 1 line
2.27.0
#27 lore raw

Re: [PATCH v9 09/41] btrfs: disable fallocate in ZONED mode

fs/btrfs/file.c static long btrfs_fallocate(struct file *file, int mode, · line 3346

Quoted 6 lines · Naohiro Aota

In the future, we may be able to implement "in-memory" fallocate() in ZONED mode by utilizing space_info->bytes_may_use or so.

fs/btrfs/file.c +3 -0
+	/* Do not allow fallocate in ZONED mode */
+	if (btrfs_is_zoned(btrfs_sb(inode->i_sb)))
+		return -EOPNOTSUPP;

So here EOPNOTSUPP is ok.

Quoted 6 lines
fs/btrfs/file.c +1 -0
+
 	/* Make sure we aren't being give some crap mode */
 	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE |
 		     FALLOC_FL_ZERO_RANGE))
Signature · 1 line
2.27.0
#28 lore raw

PATCH v9 10/41 btrfs: disallow mixed-bg in ZONED mode

In reply to Naohiro Aota

Placing both data and metadata in a block group is impossible in ZONED mode. For data, we can allocate a space for it and write it immediately after the allocation. For metadata, however, we cannot do so, because the logical addresses are recorded in other metadata buffers to build up the trees. As a result, a data buffer can be placed after a metadata buffer, which is not written yet. Writing out the data buffer will break the sequential write rule.

This commit check and disallow MIXED_BG with ZONED mode.

---
 fs/btrfs/zoned.c | 7 +++++++
 1 file changed, 7 insertions(+)
fs/btrfs/zoned.c +7 -0
index 1939b3ee6c10..ae509699da14 100644
@@ -257,6 +257,13 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 		goto out;
 	}
 
+	if (btrfs_fs_incompat(fs_info, MIXED_GROUPS)) {
+		btrfs_err(fs_info,
+			  "ZONED mode is not allowed for mixed block groups");
+		ret = -EINVAL;
+		goto out;
+	}
+
 	fs_info->zone_size = zone_size;
 	fs_info->max_zone_append_size = max_zone_append_size;
 
Signature · 1 line
2.27.0
#29 lore raw

Re: [PATCH v9 10/41] btrfs: disallow mixed-bg in ZONED mode

fs/btrfs/zoned.c int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · line 262

Quoted 9 lines · Naohiro Aota
fs/btrfs/zoned.c +3 -0
@@ -257,6 +257,13 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 		goto out;
 	}
 
+	if (btrfs_fs_incompat(fs_info, MIXED_GROUPS)) {
+		btrfs_err(fs_info,
+			  "ZONED mode is not allowed for mixed block groups");

"zoned: mixed block groups not supported"

Quoted 9 lines
fs/btrfs/zoned.c +4 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
 	fs_info->zone_size = zone_size;
 	fs_info->max_zone_append_size = max_zone_append_size;
 
Signature · 1 line
2.27.0
#30 lore raw

PATCH v9 11/41 btrfs: implement log-structured superblock for ZONED mode

In reply to Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

---
 fs/btrfs/block-group.c |   9 ++
 fs/btrfs/disk-io.c     |  41 +++++-
 fs/btrfs/scrub.c       |   3 +
 fs/btrfs/volumes.c     |  21 ++-
 fs/btrfs/zoned.c       | 311 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |  40 ++++++
 6 files changed, 413 insertions(+), 12 deletions(-)
fs/btrfs/block-group.c +9 -0
index c0f1d6818df7..e989c66aa764 100644
@@ -1723,6 +1723,7 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 static int exclude_super_stripes(struct btrfs_block_group *cache)
 {
 	struct btrfs_fs_info *fs_info = cache->fs_info;
+	bool zoned = btrfs_is_zoned(fs_info);
 	u64 bytenr;
 	u64 *logical;
 	int stripe_len;
@@ -1744,6 +1745,14 @@ static int exclude_super_stripes(struct btrfs_block_group *cache)
 		if (ret)
 			return ret;
 
+		/* shouldn't have super stripes in sequential zones */
+		if (zoned && nr) {
+			btrfs_err(fs_info,
+				  "Zoned btrfs's block group %llu should not have super blocks",
+				  cache->start);
+			return -EUCLEAN;
+		}
+
 		while (nr--) {
 			u64 len = min_t(u64, stripe_len,
 				cache->start + cache->length - logical[nr]);
fs/btrfs/disk-io.c +34 -7

89 lines not on this page · raw message

fs/btrfs/scrub.c +3 -0
index cf63f1e27a27..aa1b36cf5c88 100644
@@ -20,6 +20,7 @@
 #include "rcu-string.h"
 #include "raid56.h"
 #include "block-group.h"
+#include "zoned.h"
 
 /*
  * This is only the first step towards a full-features scrub. It reads all
@@ -3704,6 +3705,8 @@ static noinline_for_stack int scrub_supers(struct scrub_ctx *sctx,
 		if (bytenr + BTRFS_SUPER_INFO_SIZE >
 		    scrub_dev->commit_total_bytes)
 			break;
+		if (!btrfs_check_super_location(scrub_dev, bytenr))
+			continue;
 
 		ret = scrub_pages(sctx, bytenr, BTRFS_SUPER_INFO_SIZE, bytenr,
 				  scrub_dev, BTRFS_EXTENT_FLAG_SUPER, gen, i,
fs/btrfs/volumes.c +16 -5
index 10827892c086..db884b96a5ea 100644
@@ -1282,7 +1282,8 @@ void btrfs_release_disk_super(struct btrfs_super_block *super)
 }
 
 static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev,
-						       u64 bytenr)
+						       u64 bytenr,
+						       u64 bytenr_orig)
 {
 	struct btrfs_super_block *disk_super;
 	struct page *page;
@@ -1313,7 +1314,7 @@ static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev
 	/* align our pointer to the offset of the super block */
 	disk_super = p + offset_in_page(bytenr);
 
-	if (btrfs_super_bytenr(disk_super) != bytenr ||
+	if (btrfs_super_bytenr(disk_super) != bytenr_orig ||
 	    btrfs_super_magic(disk_super) != BTRFS_MAGIC) {
 		btrfs_release_disk_super(p);
 		return ERR_PTR(-EINVAL);
@@ -1348,7 +1349,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
 	bool new_device_added = false;
 	struct btrfs_device *device = NULL;
 	struct block_device *bdev;
-	u64 bytenr;
+	u64 bytenr, bytenr_orig;
+	int ret;
 
 	lockdep_assert_held(&uuid_mutex);
 
@@ -1358,14 +1360,18 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
 	 * So, we need to add a special mount option to scan for
 	 * later supers, using BTRFS_SUPER_MIRROR_MAX instead
 	 */
-	bytenr = btrfs_sb_offset(0);
 	flags |= FMODE_EXCL;
 
 	bdev = blkdev_get_by_path(path, flags, holder);
 	if (IS_ERR(bdev))
 		return ERR_CAST(bdev);
 
-	disk_super = btrfs_read_disk_super(bdev, bytenr);
+	bytenr_orig = btrfs_sb_offset(0);
+	ret = btrfs_sb_log_location_bdev(bdev, 0, READ, &bytenr);
+	if (ret)
+		return ERR_PTR(ret);
+
+	disk_super = btrfs_read_disk_super(bdev, bytenr, bytenr_orig);
 	if (IS_ERR(disk_super)) {
 		device = ERR_CAST(disk_super);
 		goto error_bdev_put;
@@ -2029,6 +2035,11 @@ void btrfs_scratch_superblocks(struct btrfs_fs_info *fs_info,
 		if (IS_ERR(disk_super))
 			continue;
 
+		if (bdev_is_zoned(bdev)) {
+			btrfs_reset_sb_log_zones(bdev, copy_num);
+			continue;
+		}
+
 		memset(&disk_super->magic, 0, sizeof(disk_super->magic));
 
 		page = virt_to_page(disk_super);
fs/btrfs/zoned.c +311 -0

326 lines not on this page · raw message

fs/btrfs/zoned.h +40 -0
index 0b7756a7104d..447c4e5ffcbb 100644
@@ -4,6 +4,8 @@
 #define BTRFS_ZONED_H
 
 #include <linux/blkdev.h>
+#include "volumes.h"
+#include "disk-io.h"
 
 struct btrfs_zoned_device_info {
 	/*
@@ -16,6 +18,7 @@ struct btrfs_zoned_device_info {
 	u32 nr_zones;
 	unsigned long *seq_zones;
 	unsigned long *empty_zones;
+	struct blk_zone sb_zones[2 * BTRFS_SUPER_MIRROR_MAX];
 };
 
 #ifdef CONFIG_BLK_DEV_ZONED
@@ -25,6 +28,12 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device);
 void btrfs_destroy_dev_zone_info(struct btrfs_device *device);
 int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info);
 int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info);
+int btrfs_sb_log_location_bdev(struct block_device *bdev, int mirror, int rw,
+			       u64 *bytenr_ret);
+int btrfs_sb_log_location(struct btrfs_device *device, int mirror, int rw,
+			  u64 *bytenr_ret);
+void btrfs_advance_sb_log(struct btrfs_device *device, int mirror);
+int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -48,6 +57,26 @@ static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
 {
 	return 0;
 }
+static inline int btrfs_sb_log_location_bdev(struct block_device *bdev,
+					     int mirror, int rw,
+					     u64 *bytenr_ret)
+{
+	*bytenr_ret = btrfs_sb_offset(mirror);
+	return 0;
+}
+static inline int btrfs_sb_log_location(struct btrfs_device *device, int mirror,
+					int rw, u64 *bytenr_ret)
+{
+	*bytenr_ret = btrfs_sb_offset(mirror);
+	return 0;
+}
+static inline void btrfs_advance_sb_log(struct btrfs_device *device,
+					int mirror) { }
+static inline int btrfs_reset_sb_log_zones(struct block_device *bdev,
+					   int mirror)
+{
+	return 0;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
@@ -115,4 +144,15 @@ static inline bool btrfs_check_device_zone_type(struct btrfs_fs_info *fs_info,
 	return bdev_zoned_model(bdev) != BLK_ZONED_HM;
 }
 
+static inline bool btrfs_check_super_location(struct btrfs_device *device,
+					      u64 pos)
+{
+	/*
+	 * On a non-zoned device, any address is OK. On a zoned device,
+	 * non-SEQUENTIAL WRITE REQUIRED zones are capable.
+	 */
+	return device->zone_info == NULL ||
+	       !btrfs_dev_is_sequential(device, pos);
+}
+
 #endif
Signature · 1 line
2.27.0
#31 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

Quoted 56 lines · Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

---
  fs/btrfs/block-group.c |   9 ++
  fs/btrfs/disk-io.c     |  41 +++++-
  fs/btrfs/scrub.c       |   3 +
  fs/btrfs/volumes.c     |  21 ++-
  fs/btrfs/zoned.c       | 311 +++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/zoned.h       |  40 ++++++
  6 files changed, 413 insertions(+), 12 deletions(-)
fs/btrfs/block-group.c +1 -0
index c0f1d6818df7..e989c66aa764 100644
@@ -1723,6 +1723,7 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
  static int exclude_super_stripes(struct btrfs_block_group *cache)
  {
  	struct btrfs_fs_info *fs_info = cache->fs_info;
+	bool zoned = btrfs_is_zoned(fs_info);
  	u64 bytenr;
  	u64 *logical;
  	int stripe_len;
@@ -1744,6 +1745,14 @@ static int exclude_super_stripes(struct btrfs_block_group *cache)
  		if (ret)

fs/btrfs/block-group.c static int exclude_super_stripes(struct btrfs_block_group *cache) · line 1755

Quoted 10 lines
fs/btrfs/block-group.c +8 -0
  			return ret;
 
+		/* shouldn't have super stripes in sequential zones */
+		if (zoned && nr) {
+			btrfs_err(fs_info,
+				  "Zoned btrfs's block group %llu should not have super blocks",
+				  cache->start);
+			return -EUCLEAN;
+		}
+

I'm very confused about this check, namely how you've been able to test without it blowing up, which makes me feel like I'm missing something.

We _always_ call exclude_super_stripes(), and we're simply looking up the bytenr for that block, which appears to not do anything special for zoned. This should be looking up and failing whenever it looks for super stripes far enough out. How are you not failing here everytime you mount the fs? Thanks,

Josef

#32 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

Quoted 11 lines · Josef Bacik

fs/btrfs/block-group.c static int exclude_super_stripes(struct btrfs_block_group *cache) · line 1755

Quoted 8 lines
fs/btrfs/block-group.c +8 -0
+		/* shouldn't have super stripes in sequential zones */
+		if (zoned && nr) {
+			btrfs_err(fs_info,
+				  "Zoned btrfs's block group %llu should not have super blocks",
+				  cache->start);
+			return -EUCLEAN;
+		}
+

I'm very confused about this check, namely how you've been able to test without it blowing up, which makes me feel like I'm missing something.

Quoted 4 lines

We _always_ call exclude_super_stripes(), and we're simply looking up the bytenr for that block, which appears to not do anything special for zoned. This should be looking up and failing whenever it looks for super stripes far enough out. How are you not failing here everytime you mount the fs? Thanks,

Naohiro (or Josef and everyone else as well of cause), please correct me if I'm wrong, but on zoned btrfs we're not supporting any RAID type. So the call to btrfs_rmap_block() above will return 'nr = 0' (as we're always having map->num_stripes = 1) so this won't evaluate to true.

Byte,
Johannes

#33 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

Quoted 17 lines · Johannes Thumshirn
Quoted 11 lines · Josef Bacik

fs/btrfs/block-group.c static int exclude_super_stripes(struct btrfs_block_group *cache) · line 1755

Quoted 8 lines
fs/btrfs/block-group.c +8 -0
+		/* shouldn't have super stripes in sequential zones */
+		if (zoned && nr) {
+			btrfs_err(fs_info,
+				  "Zoned btrfs's block group %llu should not have super blocks",
+				  cache->start);
+			return -EUCLEAN;
+		}
+

I'm very confused about this check, namely how you've been able to test without it blowing up, which makes me feel like I'm missing something.

Quoted 4 lines

We _always_ call exclude_super_stripes(), and we're simply looking up the bytenr for that block, which appears to not do anything special for zoned. This should be looking up and failing whenever it looks for super stripes far enough out. How are you not failing here everytime you mount the fs? Thanks,

Quoted 5 lines

Naohiro (or Josef and everyone else as well of cause), please correct me if I'm wrong, but on zoned btrfs we're not supporting any RAID type. So the call to btrfs_rmap_block() above will return 'nr = 0' (as we're always having map->num_stripes = 1) so this won't evaluate to true.

No it should return nr == 1 in the single case. This maps physical address to a logical address in the block group, so it could be multiple, but if that bytenr falls inside the block group it'll return with something set. Hence my confusion. Thanks,

Josef

#34 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

Quoted 4 lines · Josef Bacik

No it should return nr == 1 in the single case. This maps physical address to a logical address in the block group, so it could be multiple, but if that bytenr falls inside the block group it'll return with something set. Hence my confusion.

OK so from my debugging [1] it looks like we're hitting the !in_range() continue case in __btrfs_rmap_block()'s loop.

But I'll need to defer to Naohiro to answer this question.

[1]:
mount -t btrfs /dev/nullb0 /mnt/test

[    2.189080] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 65536                                                                       
[    2.191168] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(65536, 536870912, 268435456)
[    2.193068] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0                                                                                                     
[    2.194603] BTRFS error (device nullb0): exclude_super_stripes: nr: 0                                                                                                              
[    2.195973] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 67108864                                                                    
[    2.197378] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(67108864, 536870912, 268435456)
[    2.198382] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.199160] BTRFS error (device nullb0): exclude_super_stripes: nr: 0                                                                                                              
[    2.199871] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 274877906944                                                                
[    2.201030] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(274877906944, 536870912, 268435456)
[    2.202088] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.202864] BTRFS error (device nullb0): exclude_super_stripes: nr: 0                                                                                                              
[    2.203549] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 65536                                                                       
[    2.204621] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(65536, 805306368, 268435456)
[    2.205590] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.206394] BTRFS error (device nullb0): exclude_super_stripes: nr: 0
[    2.207078] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 67108864
[    2.208131] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(67108864, 805306368, 268435456)
[    2.209111] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.209885] BTRFS error (device nullb0): exclude_super_stripes: nr: 0
[    2.210540] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 274877906944
[    2.211595] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(274877906944, 805306368, 268435456)
[    2.212620] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.213388] BTRFS error (device nullb0): exclude_super_stripes: nr: 0
[    2.214076] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 65536
[    2.215079] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(65536, 1073741824, 268435456)
[    2.216039] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.216801] BTRFS error (device nullb0): exclude_super_stripes: nr: 0
[    2.217491] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 67108864
[    2.218548] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(67108864, 1073741824, 268435456)
[    2.219537] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.220322] BTRFS error (device nullb0): exclude_super_stripes: nr: 0
[    2.221020] BTRFS error (device nullb0): exclude_super_stripes: calling btrfs_rmap_block() for bytenr: 274877906944
[    2.222150] BTRFS error (device nullb0): __btrfs_rmap_block: !in_range(274877906944, 1073741824, 268435456)
[    2.223165] BTRFS error (device nullb0): __btrfs_rmap_block: nr: 0, *naddrs: 0
[    2.223945] BTRFS error (device nullb0): exclude_super_stripes: nr: 0
#35 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

In reply to Josef Bacik
Quoted 69 lines · Josef Bacik
Quoted 55 lines · Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

---
 fs/btrfs/block-group.c |   9 ++
 fs/btrfs/disk-io.c     |  41 +++++-
 fs/btrfs/scrub.c       |   3 +
 fs/btrfs/volumes.c     |  21 ++-
 fs/btrfs/zoned.c       | 311 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |  40 ++++++
 6 files changed, 413 insertions(+), 12 deletions(-)
fs/btrfs/block-group.c +1 -0
index c0f1d6818df7..e989c66aa764 100644
@@ -1723,6 +1723,7 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 static int exclude_super_stripes(struct btrfs_block_group *cache)
 {
 	struct btrfs_fs_info *fs_info = cache->fs_info;
+	bool zoned = btrfs_is_zoned(fs_info);
 	u64 bytenr;
 	u64 *logical;
 	int stripe_len;

fs/btrfs/block-group.c static int exclude_super_stripes(struct btrfs_block_group *cache) · line 1754

Quoted 10 lines
fs/btrfs/block-group.c +8 -0
@@ -1744,6 +1745,14 @@ static int exclude_super_stripes(struct btrfs_block_group *cache)
 		if (ret)
 			return ret;
+		/* shouldn't have super stripes in sequential zones */
+		if (zoned && nr) {
+			btrfs_err(fs_info,
+				  "Zoned btrfs's block group %llu should not have super blocks",
+				  cache->start);
+			return -EUCLEAN;
+		}
+

I'm very confused about this check, namely how you've been able to test without it blowing up, which makes me feel like I'm missing something.

Quoted 3 lines

We _always_ call exclude_super_stripes(), and we're simply looking up the bytenr for that block, which appears to not do anything special for zoned. This should be looking up and failing whenever it looks for super stripes far enough out. How are you not failing here everytime you mount the fs? Thanks,

Josef

As previous discussion with David, we decided to exclude superblock position of regular btrfs being allocated for zoned block groups, because superblock is one of on-disk specification. (Sorry, I could not find a pointer of the discussion) We also need to ensure some user data which looks like a superblock won't corrupt the FS.

btrfs_find_allocatable_zones() is doing that exclusion ... this was my understanding. But, to be precise, the function is just excluding superblock log zones, not regular superblock positions.

However, it accidentally excludes superblock positions of regualr device. We have superblocks on 16KB, 64MB and 256GB. And, we exclude zones 0, 1, 16, 17 and min(zones at 256GB, 1024). With typical 256MB zone size setup, 16KB and 64MB drop on zone 0, and 256GB on zone 1024. So, I was not hitting this "if" on my test runs.

I'll fix btrfs_find_allocatable_zones() so that it also excludes regular superblock positions.

Thanks,

#36 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

In reply to Naohiro Aota
Quoted 32 lines · Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

<snip>

fs/btrfs/scrub.c static noinline_for_stack int scrub_supers(struct scrub_ctx *sctx, · line 3709

Quoted 9 lines

/*
* This is only the first step towards a full-features scrub. It reads all

fs/btrfs/scrub.c +2 -0
@@ -3704,6 +3705,8 @@ static noinline_for_stack int scrub_supers(struct scrub_ctx *sctx,
  		if (bytenr + BTRFS_SUPER_INFO_SIZE >
  		    scrub_dev->commit_total_bytes)
  			break;
+		if (!btrfs_check_super_location(scrub_dev, bytenr))
+			continue;

Any reason in particular we're skipping scrubbing supers here? Can't we just lookup the bytenr and do the right thing here?

Quoted 83 lines
fs/btrfs/scrub.c +0 -0
  		ret = scrub_pages(sctx, bytenr, BTRFS_SUPER_INFO_SIZE, bytenr,
  				  scrub_dev, BTRFS_EXTENT_FLAG_SUPER, gen, i,
fs/btrfs/volumes.c +16 -5
index 10827892c086..db884b96a5ea 100644
@@ -1282,7 +1282,8 @@ void btrfs_release_disk_super(struct btrfs_super_block *super)
  }
 
  static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev,
-						       u64 bytenr)
+						       u64 bytenr,
+						       u64 bytenr_orig)
  {
  	struct btrfs_super_block *disk_super;
  	struct page *page;
@@ -1313,7 +1314,7 @@ static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev
  	/* align our pointer to the offset of the super block */
  	disk_super = p + offset_in_page(bytenr);
 
-	if (btrfs_super_bytenr(disk_super) != bytenr ||
+	if (btrfs_super_bytenr(disk_super) != bytenr_orig ||
  	    btrfs_super_magic(disk_super) != BTRFS_MAGIC) {
  		btrfs_release_disk_super(p);
  		return ERR_PTR(-EINVAL);
@@ -1348,7 +1349,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
  	bool new_device_added = false;
  	struct btrfs_device *device = NULL;
  	struct block_device *bdev;
-	u64 bytenr;
+	u64 bytenr, bytenr_orig;
+	int ret;
 
  	lockdep_assert_held(&uuid_mutex);
 
@@ -1358,14 +1360,18 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
  	 * So, we need to add a special mount option to scan for
  	 * later supers, using BTRFS_SUPER_MIRROR_MAX instead
  	 */
-	bytenr = btrfs_sb_offset(0);
  	flags |= FMODE_EXCL;
 
  	bdev = blkdev_get_by_path(path, flags, holder);
  	if (IS_ERR(bdev))
  		return ERR_CAST(bdev);
 
-	disk_super = btrfs_read_disk_super(bdev, bytenr);
+	bytenr_orig = btrfs_sb_offset(0);
+	ret = btrfs_sb_log_location_bdev(bdev, 0, READ, &bytenr);
+	if (ret)
+		return ERR_PTR(ret);
+
+	disk_super = btrfs_read_disk_super(bdev, bytenr, bytenr_orig);
  	if (IS_ERR(disk_super)) {
  		device = ERR_CAST(disk_super);
  		goto error_bdev_put;
@@ -2029,6 +2035,11 @@ void btrfs_scratch_superblocks(struct btrfs_fs_info *fs_info,
  		if (IS_ERR(disk_super))
  			continue;
 
+		if (bdev_is_zoned(bdev)) {
+			btrfs_reset_sb_log_zones(bdev, copy_num);
+			continue;
+		}
+
  		memset(&disk_super->magic, 0, sizeof(disk_super->magic));
 
  		page = virt_to_page(disk_super);
fs/btrfs/zoned.c +6 -0
index ae509699da14..d5487cba203b 100644
@@ -20,6 +20,25 @@ static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
  	return 0;
  }
 
+static int sb_write_pointer(struct block_device *bdev, struct blk_zone *zone,
+			    u64 *wp_ret);
+
+static inline u32 sb_zone_number(u8 shift, int mirror)
+{
+	ASSERT(mirror < BTRFS_SUPER_MIRROR_MAX);

fs/btrfs/zoned.c static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx, · line 38

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+
+	switch (mirror) {
+	case 0:
+		return 0;
+	case 1:
+		return 16;
+	case 2:
+		return min(btrfs_sb_offset(mirror) >> shift, 1024ULL);
+	}
+

Can we get a comment here explaining the zone numbers?

Quoted 7 lines
fs/btrfs/zoned.c +3 -0
+	return 0;
+}
+
  static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
  			       struct blk_zone *zones, unsigned int *nr_zones)
  {

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 150

Quoted 9 lines
fs/btrfs/zoned.c +6 -0
@@ -123,6 +142,49 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
  		goto out;
  	}
 
+	/* validate superblock log */
+	nr_zones = 2;
+	for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
+		u32 sb_zone = sb_zone_number(zone_info->zone_size_shift, i);
+		u64 sb_wp;
+

I'd rather see

#define BTRFS_NR_ZONED_SB_ZONES 2

or something equally poorly named and use that instead of our magic 2 everywhere.

Then you can just do

int index = i * BTRFS_NR_ZONED_SB_ZONES;
&zone_info->sb_zones[index];

<snip>

Quoted 19 lines
fs/btrfs/zoned.c +19 -0
+static int sb_log_location(struct block_device *bdev, struct blk_zone *zones,
+			   int rw, u64 *bytenr_ret)
+{
+	u64 wp;
+	int ret;
+
+	if (zones[0].type == BLK_ZONE_TYPE_CONVENTIONAL) {
+		*bytenr_ret = zones[0].start << SECTOR_SHIFT;
+		return 0;
+	}
+
+	ret = sb_write_pointer(bdev, zones, &wp);
+	if (ret != -ENOENT && ret < 0)
+		return ret;
+
+	if (rw == WRITE) {
+		struct blk_zone *reset = NULL;
+
+		if (wp == zones[0].start << SECTOR_SHIFT)

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 467

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+			reset = &zones[0];
+		else if (wp == zones[1].start << SECTOR_SHIFT)
+			reset = &zones[1];
+
+		if (reset && reset->cond != BLK_ZONE_COND_EMPTY) {
+			ASSERT(reset->cond == BLK_ZONE_COND_FULL);
+
+			ret = blkdev_zone_mgmt(bdev, REQ_OP_ZONE_RESET,
+					       reset->start, reset->len,
+					       GFP_NOFS);

What happens if we crash right after this? Is the WP set to the start of the zone here? Does this mean we'll simply miss the super block? I understand we're resetting one zone here, but we're doing this in order, so we'll reset one and write one, then reset the other and write the next. We don't wait until we've issued the writes for everything, so it appears to me that there's a gap where we could have the WP pointed at the start of the zone, which we view as an invalid state and thus won't be able to mount the file system. Or am I missing something? Thanks,

Josef

#37 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

Quoted 36 lines · Josef Bacik
Quoted 32 lines · Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

<snip>

Quoted 10 lines

fs/btrfs/scrub.c static noinline_for_stack int scrub_supers(struct scrub_ctx *sctx, · line 3709

Quoted 8 lines

/*
* This is only the first step towards a full-features scrub. It reads all

fs/btrfs/scrub.c +2 -0
@@ -3704,6 +3705,8 @@ static noinline_for_stack int scrub_supers(struct scrub_ctx *sctx,
 		if (bytenr + BTRFS_SUPER_INFO_SIZE >
 		    scrub_dev->commit_total_bytes)
 			break;
+		if (!btrfs_check_super_location(scrub_dev, bytenr))
+			continue;

Any reason in particular we're skipping scrubbing supers here? Can't we just lookup the bytenr and do the right thing here?

Hmm, technically, we can do something here, but I'm not sure it's useful to scrub superblocks for zoned devices where superblocks are log-structured. We can read and check if the latest superblock in the log is valid. But, when we find it's not correct, we cannot overwrite it anyway. Instead, we can append a new superblock to the log. But this is no different than normal sync... Furthermore, the scrub-checked superblock might already be out-dated at the time of reading.

We might want to read and check each entry of the log. And warn the user when a superblock is corrupted. It's totally different from current scrub_supers(), so we will need another helper function for it.

Quoted 78 lines
Quoted 78 lines
fs/btrfs/scrub.c +0 -0
 		ret = scrub_pages(sctx, bytenr, BTRFS_SUPER_INFO_SIZE, bytenr,
 				  scrub_dev, BTRFS_EXTENT_FLAG_SUPER, gen, i,
fs/btrfs/volumes.c +16 -5
index 10827892c086..db884b96a5ea 100644
@@ -1282,7 +1282,8 @@ void btrfs_release_disk_super(struct btrfs_super_block *super)
 }
 static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev,
-						       u64 bytenr)
+						       u64 bytenr,
+						       u64 bytenr_orig)
 {
 	struct btrfs_super_block *disk_super;
 	struct page *page;
@@ -1313,7 +1314,7 @@ static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev
 	/* align our pointer to the offset of the super block */
 	disk_super = p + offset_in_page(bytenr);
-	if (btrfs_super_bytenr(disk_super) != bytenr ||
+	if (btrfs_super_bytenr(disk_super) != bytenr_orig ||
 	    btrfs_super_magic(disk_super) != BTRFS_MAGIC) {
 		btrfs_release_disk_super(p);
 		return ERR_PTR(-EINVAL);
@@ -1348,7 +1349,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
 	bool new_device_added = false;
 	struct btrfs_device *device = NULL;
 	struct block_device *bdev;
-	u64 bytenr;
+	u64 bytenr, bytenr_orig;
+	int ret;
 	lockdep_assert_held(&uuid_mutex);
@@ -1358,14 +1360,18 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
 	 * So, we need to add a special mount option to scan for
 	 * later supers, using BTRFS_SUPER_MIRROR_MAX instead
 	 */
-	bytenr = btrfs_sb_offset(0);
 	flags |= FMODE_EXCL;
 	bdev = blkdev_get_by_path(path, flags, holder);
 	if (IS_ERR(bdev))
 		return ERR_CAST(bdev);
-	disk_super = btrfs_read_disk_super(bdev, bytenr);
+	bytenr_orig = btrfs_sb_offset(0);
+	ret = btrfs_sb_log_location_bdev(bdev, 0, READ, &bytenr);
+	if (ret)
+		return ERR_PTR(ret);
+
+	disk_super = btrfs_read_disk_super(bdev, bytenr, bytenr_orig);
 	if (IS_ERR(disk_super)) {
 		device = ERR_CAST(disk_super);
 		goto error_bdev_put;
@@ -2029,6 +2035,11 @@ void btrfs_scratch_superblocks(struct btrfs_fs_info *fs_info,
 		if (IS_ERR(disk_super))
 			continue;
+		if (bdev_is_zoned(bdev)) {
+			btrfs_reset_sb_log_zones(bdev, copy_num);
+			continue;
+		}
+
 		memset(&disk_super->magic, 0, sizeof(disk_super->magic));
 		page = virt_to_page(disk_super);
fs/btrfs/zoned.c +10 -0
index ae509699da14..d5487cba203b 100644
@@ -20,6 +20,25 @@ static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
 	return 0;
 }
+static int sb_write_pointer(struct block_device *bdev, struct blk_zone *zone,
+			    u64 *wp_ret);
+
+static inline u32 sb_zone_number(u8 shift, int mirror)
+{
+	ASSERT(mirror < BTRFS_SUPER_MIRROR_MAX);
+
+	switch (mirror) {
+	case 0:
+		return 0;
Quoted 8 lines

fs/btrfs/zoned.c static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx, · line 38

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+	case 1:
+		return 16;
+	case 2:
+		return min(btrfs_sb_offset(mirror) >> shift, 1024ULL);
+	}
+

Can we get a comment here explaining the zone numbers?

Sure. I'll add one.

Quoted 19 lines
Quoted 8 lines
fs/btrfs/zoned.c +3 -0
+	return 0;
+}
+
 static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
 			       struct blk_zone *zones, unsigned int *nr_zones)
 {

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 149

Quoted 8 lines
fs/btrfs/zoned.c +6 -0
@@ -123,6 +142,49 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
 		goto out;
 	}
+	/* validate superblock log */
+	nr_zones = 2;
+	for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
+		u32 sb_zone = sb_zone_number(zone_info->zone_size_shift, i);
+		u64 sb_wp;
+

I'd rather see

Quoted 8 lines

#define BTRFS_NR_ZONED_SB_ZONES 2

or something equally poorly named and use that instead of our magic 2 everywhere.

Then you can just do

int index = i * BTRFS_NR_ZONED_SB_ZONES;
&zone_info->sb_zones[index];

I'll do so. BTRFS_NR_ZONED_SB_ZONES is duplicating "ZONE", so how about BTRFS_NR_SB_LOG_ZONES ?

Quoted 26 lines

<snip>

Quoted 23 lines
fs/btrfs/zoned.c +23 -0
+static int sb_log_location(struct block_device *bdev, struct blk_zone *zones,
+			   int rw, u64 *bytenr_ret)
+{
+	u64 wp;
+	int ret;
+
+	if (zones[0].type == BLK_ZONE_TYPE_CONVENTIONAL) {
+		*bytenr_ret = zones[0].start << SECTOR_SHIFT;
+		return 0;
+	}
+
+	ret = sb_write_pointer(bdev, zones, &wp);
+	if (ret != -ENOENT && ret < 0)
+		return ret;
+
+	if (rw == WRITE) {
+		struct blk_zone *reset = NULL;
+
+		if (wp == zones[0].start << SECTOR_SHIFT)
+			reset = &zones[0];
+		else if (wp == zones[1].start << SECTOR_SHIFT)
+			reset = &zones[1];
+
Quoted 8 lines

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 467

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+		if (reset && reset->cond != BLK_ZONE_COND_EMPTY) {
+			ASSERT(reset->cond == BLK_ZONE_COND_FULL);
+
+			ret = blkdev_zone_mgmt(bdev, REQ_OP_ZONE_RESET,
+					       reset->start, reset->len,
+					       GFP_NOFS);

What happens if we crash right after this? Is the WP set to the start of the zone here? Does this mean we'll simply miss the super block? I understand we're resetting one zone here, but we're doing this in order, so we'll reset one and write one, then reset the other and write the next. We don't wait until we've issued the writes for everything, so it appears to me that there's a gap where we could have the WP pointed at the start of the zone, which we view as an invalid state and thus won't be able to mount the file system. Or am I missing something? Thanks,

Here, we reset a zone we're going to write which contains the older superblocks. And in this case, we should have the other zone fully written. So, even after a reset and a crash, we still have the latest superblock in the other zone.

Quoted 2 lines

Josef

#38 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

In reply to Naohiro Aota

fs/btrfs/block-group.c int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start, · line 1726

Quoted 7 lines · Naohiro Aota
fs/btrfs/block-group.c +1 -0
@@ -1723,6 +1723,7 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 static int exclude_super_stripes(struct btrfs_block_group *cache)
 {
 	struct btrfs_fs_info *fs_info = cache->fs_info;
+	bool zoned = btrfs_is_zoned(fs_info);

const bool

fs/btrfs/block-group.c static int exclude_super_stripes(struct btrfs_block_group *cache) · line 1748

Quoted 8 lines
fs/btrfs/block-group.c +1 -0
 	u64 bytenr;
 	u64 *logical;
 	int stripe_len;
@@ -1744,6 +1745,14 @@ static int exclude_super_stripes(struct btrfs_block_group *cache)
 		if (ret)
 			return ret;
 
+		/* shouldn't have super stripes in sequential zones */
		/* Shouldn't ... */

fs/btrfs/block-group.c static int exclude_super_stripes(struct btrfs_block_group *cache) · line 1751

Quoted 3 lines
fs/btrfs/block-group.c +3 -0
+		if (zoned && nr) {
+			btrfs_err(fs_info,
+				  "Zoned btrfs's block group %llu should not have super blocks",

"zoned: block group %llu must not contain super block"

Quoted 91 lines
fs/btrfs/block-group.c +4 -0
+				  cache->start);
+			return -EUCLEAN;
+		}
+
 		while (nr--) {
 			u64 len = min_t(u64, stripe_len,
 				cache->start + cache->length - logical[nr]);
fs/btrfs/disk-io.c +25 -6
index 9bc51cff48b8..fd8b970ee92c 100644
@@ -3423,10 +3423,17 @@ struct btrfs_super_block *btrfs_read_dev_one_super(struct block_device *bdev,
 {
 	struct btrfs_super_block *super;
 	struct page *page;
-	u64 bytenr;
+	u64 bytenr, bytenr_orig;
 	struct address_space *mapping = bdev->bd_inode->i_mapping;
+	int ret;
+
+	bytenr_orig = btrfs_sb_offset(copy_num);
+	ret = btrfs_sb_log_location_bdev(bdev, copy_num, READ, &bytenr);
+	if (ret == -ENOENT)
+		return ERR_PTR(-EINVAL);
+	else if (ret)
+		return ERR_PTR(ret);
 
-	bytenr = btrfs_sb_offset(copy_num);
 	if (bytenr + BTRFS_SUPER_INFO_SIZE >= i_size_read(bdev->bd_inode))
 		return ERR_PTR(-EINVAL);
 
@@ -3440,7 +3447,7 @@ struct btrfs_super_block *btrfs_read_dev_one_super(struct block_device *bdev,
 		return ERR_PTR(-ENODATA);
 	}
 
-	if (btrfs_super_bytenr(super) != bytenr) {
+	if (btrfs_super_bytenr(super) != bytenr_orig) {
 		btrfs_release_disk_super(super);
 		return ERR_PTR(-EINVAL);
 	}
@@ -3495,7 +3502,8 @@ static int write_dev_supers(struct btrfs_device *device,
 	SHASH_DESC_ON_STACK(shash, fs_info->csum_shash);
 	int i;
 	int errors = 0;
-	u64 bytenr;
+	int ret;
+	u64 bytenr, bytenr_orig;
 
 	if (max_mirrors == 0)
 		max_mirrors = BTRFS_SUPER_MIRROR_MAX;
@@ -3507,12 +3515,21 @@ static int write_dev_supers(struct btrfs_device *device,
 		struct bio *bio;
 		struct btrfs_super_block *disk_super;
 
-		bytenr = btrfs_sb_offset(i);
+		bytenr_orig = btrfs_sb_offset(i);
+		ret = btrfs_sb_log_location(device, i, WRITE, &bytenr);
+		if (ret == -ENOENT)
+			continue;
+		else if (ret < 0) {
+			btrfs_err(device->fs_info, "couldn't get super block location for mirror %d",
+				  i);
+			errors++;
+			continue;
+		}
 		if (bytenr + BTRFS_SUPER_INFO_SIZE >=
 		    device->commit_total_bytes)
 			break;
 
-		btrfs_set_super_bytenr(sb, bytenr);
+		btrfs_set_super_bytenr(sb, bytenr_orig);
 
 		crypto_shash_digest(shash, (const char *)sb + BTRFS_CSUM_SIZE,
 				    BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE,
@@ -3557,6 +3574,7 @@ static int write_dev_supers(struct btrfs_device *device,
 			bio->bi_opf |= REQ_FUA;
 
 		btrfsic_submit_bio(bio);
+		btrfs_advance_sb_log(device, i);
 	}
 	return errors < i ? 0 : -1;
 }
@@ -3573,6 +3591,7 @@ static int wait_dev_supers(struct btrfs_device *device, int max_mirrors)
 	int i;
 	int errors = 0;
 	bool primary_failed = false;
+	int ret;
 	u64 bytenr;
 
 	if (max_mirrors == 0)

fs/btrfs/disk-io.c static int wait_dev_supers(struct btrfs_device *device, int max_mirrors) · line 3604

Quoted 6 lines
fs/btrfs/disk-io.c +2 -1
@@ -3581,7 +3600,15 @@ static int wait_dev_supers(struct btrfs_device *device, int max_mirrors)
 	for (i = 0; i < max_mirrors; i++) {
 		struct page *page;
 
-		bytenr = btrfs_sb_offset(i);
+		ret = btrfs_sb_log_location(device, i, READ, &bytenr);
+		if (ret == -ENOENT)
		if (...) {

fs/btrfs/disk-io.c static int wait_dev_supers(struct btrfs_device *device, int max_mirrors) · line 3605

Quoted 1 line
fs/btrfs/disk-io.c +1 -0
+			break;
		} else if (...) {
Quoted 78 lines
fs/btrfs/disk-io.c +6 -0
+		else if (ret < 0) {
+			errors++;
+			if (i == 0)
+				primary_failed = true;
+			continue;
+		}
 		if (bytenr + BTRFS_SUPER_INFO_SIZE >=
 		    device->commit_total_bytes)
 			break;
fs/btrfs/volumes.c +16 -5
@@ -1282,7 +1282,8 @@ void btrfs_release_disk_super(struct btrfs_super_block *super)
 }
 
 static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev,
-						       u64 bytenr)
+						       u64 bytenr,
+						       u64 bytenr_orig)
 {
 	struct btrfs_super_block *disk_super;
 	struct page *page;
@@ -1313,7 +1314,7 @@ static struct btrfs_super_block *btrfs_read_disk_super(struct block_device *bdev
 	/* align our pointer to the offset of the super block */
 	disk_super = p + offset_in_page(bytenr);
 
-	if (btrfs_super_bytenr(disk_super) != bytenr ||
+	if (btrfs_super_bytenr(disk_super) != bytenr_orig ||
 	    btrfs_super_magic(disk_super) != BTRFS_MAGIC) {
 		btrfs_release_disk_super(p);
 		return ERR_PTR(-EINVAL);
@@ -1348,7 +1349,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
 	bool new_device_added = false;
 	struct btrfs_device *device = NULL;
 	struct block_device *bdev;
-	u64 bytenr;
+	u64 bytenr, bytenr_orig;
+	int ret;
 
 	lockdep_assert_held(&uuid_mutex);
 
@@ -1358,14 +1360,18 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
 	 * So, we need to add a special mount option to scan for
 	 * later supers, using BTRFS_SUPER_MIRROR_MAX instead
 	 */
-	bytenr = btrfs_sb_offset(0);
 	flags |= FMODE_EXCL;
 
 	bdev = blkdev_get_by_path(path, flags, holder);
 	if (IS_ERR(bdev))
 		return ERR_CAST(bdev);
 
-	disk_super = btrfs_read_disk_super(bdev, bytenr);
+	bytenr_orig = btrfs_sb_offset(0);
+	ret = btrfs_sb_log_location_bdev(bdev, 0, READ, &bytenr);
+	if (ret)
+		return ERR_PTR(ret);
+
+	disk_super = btrfs_read_disk_super(bdev, bytenr, bytenr_orig);
 	if (IS_ERR(disk_super)) {
 		device = ERR_CAST(disk_super);
 		goto error_bdev_put;
@@ -2029,6 +2035,11 @@ void btrfs_scratch_superblocks(struct btrfs_fs_info *fs_info,
 		if (IS_ERR(disk_super))
 			continue;
 
+		if (bdev_is_zoned(bdev)) {
+			btrfs_reset_sb_log_zones(bdev, copy_num);
+			continue;
+		}
+
 		memset(&disk_super->magic, 0, sizeof(disk_super->magic));
 
 		page = virt_to_page(disk_super);
fs/btrfs/zoned.c +0 -0
index ae509699da14..d5487cba203b 100644

fs/btrfs/zoned.c static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx, · line 24

Quoted 5 lines
fs/btrfs/zoned.c +2 -0
@@ -20,6 +20,25 @@ static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
 	return 0;
 }
 
+static int sb_write_pointer(struct block_device *bdev, struct blk_zone *zone,
+			    u64 *wp_ret);

Please define sb_write_pointer here instead of the prototype for a static function.

Quoted 21 lines
fs/btrfs/zoned.c +17 -0
+
+static inline u32 sb_zone_number(u8 shift, int mirror)
+{
+	ASSERT(mirror < BTRFS_SUPER_MIRROR_MAX);
+
+	switch (mirror) {
+	case 0:
+		return 0;
+	case 1:
+		return 16;
+	case 2:
+		return min(btrfs_sb_offset(mirror) >> shift, 1024ULL);
+	}
+
+	return 0;
+}
+
 static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
 			       struct blk_zone *zones, unsigned int *nr_zones)
 {

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 145

Quoted 4 lines
fs/btrfs/zoned.c +1 -0
@@ -123,6 +142,49 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device)
 		goto out;
 	}
 
+	/* validate superblock log */
	/* Validate ... */
Quoted 8 lines
fs/btrfs/zoned.c +8 -0
+	nr_zones = 2;
+	for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
+		u32 sb_zone = sb_zone_number(zone_info->zone_size_shift, i);
+		u64 sb_wp;
+
+		if (sb_zone + 1 >= zone_info->nr_zones)
+			continue;
+

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 163

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		sector = sb_zone << (zone_info->zone_size_shift - SECTOR_SHIFT);
+		ret = btrfs_get_dev_zones(device, sector << SECTOR_SHIFT,
+					  &zone_info->sb_zones[2 * i],
+					  &nr_zones);
+		if (ret)
+			goto out;
+		if (nr_zones != 2) {
+			btrfs_err_in_rcu(device->fs_info,
+			"failed to read SB log zone info at device %s zone %u",
+					 rcu_str_deref(device->name), sb_zone);

"zoned: failed to read super block log zone info at devid %llu zone %u"

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 164

Quoted 1 line
fs/btrfs/zoned.c +1 -0
+			ret = -EIO;

What are the possible reasons here? EIO would fit reading error but if the zone is missing it's more like EUCLEAN.

Quoted 7 lines
fs/btrfs/zoned.c +7 -0
+			goto out;
+		}
+
+		/*
+		 * If zones[0] is conventional, always use the beggining of
+		 * the zone to record superblock. No need to validate in
+		 * that case.

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device) · line 181

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		 */
+		if (zone_info->sb_zones[2 * i].type == BLK_ZONE_TYPE_CONVENTIONAL)
+			continue;
+
+		ret = sb_write_pointer(device->bdev,
+				       &zone_info->sb_zones[2 * i], &sb_wp);
+		if (ret != -ENOENT && ret) {
+			btrfs_err_in_rcu(device->fs_info,
+				"SB log zone corrupted: device %s zone %u",
+					 rcu_str_deref(device->name), sb_zone);

"zoned: super block log zone corrupted devid %llu zone %u"

The device path would be also good in all the messages, this could be tweaked later.

Quoted 15 lines
fs/btrfs/zoned.c +8 -0
+			ret = -EUCLEAN;
+			goto out;
+		}
+	}
+
+
 	kfree(zones);
 
 	device->zone_info = zone_info;
@@ -296,3 +358,252 @@ int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
 
 	return 0;
 }
+
+static int sb_write_pointer(struct block_device *bdev, struct blk_zone *zones,

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 372

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+			    u64 *wp_ret)
+{
+	bool empty[2];
+	bool full[2];
+	sector_t sector;
+
+	ASSERT(zones[0].type != BLK_ZONE_TYPE_CONVENTIONAL &&
+	       zones[1].type != BLK_ZONE_TYPE_CONVENTIONAL);
+
+	empty[0] = zones[0].cond == BLK_ZONE_COND_EMPTY;
	empty[0] = (zones[0].cond == BLK_ZONE_COND_EMPTY);

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 375

Quoted 3 lines
fs/btrfs/zoned.c +3 -0
+	empty[1] = zones[1].cond == BLK_ZONE_COND_EMPTY;
+	full[0] = zones[0].cond == BLK_ZONE_COND_FULL;
+	full[1] = zones[1].cond == BLK_ZONE_COND_FULL;

Same

Quoted 13 lines
fs/btrfs/zoned.c +13 -0
+
+	/*
+	 * Possible state of log buffer zones
+	 *
+	 *   E I F
+	 * E * x 0
+	 * I 0 x 0
+	 * F 1 1 C
+	 *
+	 * Row: zones[0]
+	 * Col: zones[1]
+	 * State:
+	 *   E: Empty, I: In-Use, F: Full

fs/btrfs/zoned.c int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 398

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	 * Log position:
+	 *   *: Special case, no superblock is written
+	 *   0: Use write pointer of zones[0]
+	 *   1: Use write pointer of zones[1]
+	 *   C: Compare SBs from zones[0] and zones[1], use the newer one
+	 *   x: Invalid state
+	 */
+
+	if (empty[0] && empty[1]) {
+		/* special case to distinguish no superblock to read */
		/* Special ... */
Quoted 72 lines
fs/btrfs/zoned.c +39 -0
+		*wp_ret = zones[0].start << SECTOR_SHIFT;
+		return -ENOENT;
+	} else if (full[0] && full[1]) {
+		/* Compare two super blocks */
+		struct address_space *mapping = bdev->bd_inode->i_mapping;
+		struct page *page[2];
+		struct btrfs_super_block *super[2];
+		int i;
+
+		for (i = 0; i < 2; i++) {
+			u64 bytenr = ((zones[i].start + zones[i].len) << SECTOR_SHIFT) -
+				BTRFS_SUPER_INFO_SIZE;
+
+			page[i] = read_cache_page_gfp(mapping, bytenr >> PAGE_SHIFT, GFP_NOFS);
+			if (IS_ERR(page[i])) {
+				if (i == 1)
+					btrfs_release_disk_super(super[0]);
+				return PTR_ERR(page[i]);
+			}
+			super[i] = page_address(page[i]);
+		}
+
+		if (super[0]->generation > super[1]->generation)
+			sector = zones[1].start;
+		else
+			sector = zones[0].start;
+
+		for (i = 0; i < 2; i++)
+			btrfs_release_disk_super(super[i]);
+	} else if (!full[0] && (empty[1] || full[1])) {
+		sector = zones[0].wp;
+	} else if (full[0]) {
+		sector = zones[1].wp;
+	} else {
+		return -EUCLEAN;
+	}
+	*wp_ret = sector << SECTOR_SHIFT;
+	return 0;
+}
fs/btrfs/zoned.h +9 -0
@@ -4,6 +4,8 @@
 #define BTRFS_ZONED_H
 
 #include <linux/blkdev.h>
+#include "volumes.h"
+#include "disk-io.h"
 
 struct btrfs_zoned_device_info {
 	/*
@@ -16,6 +18,7 @@ struct btrfs_zoned_device_info {
 	u32 nr_zones;
 	unsigned long *seq_zones;
 	unsigned long *empty_zones;
+	struct blk_zone sb_zones[2 * BTRFS_SUPER_MIRROR_MAX];
 };
 
 #ifdef CONFIG_BLK_DEV_ZONED
@@ -25,6 +28,12 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device);
 void btrfs_destroy_dev_zone_info(struct btrfs_device *device);
 int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info);
 int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info);
+int btrfs_sb_log_location_bdev(struct block_device *bdev, int mirror, int rw,
+			       u64 *bytenr_ret);
+int btrfs_sb_log_location(struct btrfs_device *device, int mirror, int rw,
+			  u64 *bytenr_ret);
+void btrfs_advance_sb_log(struct btrfs_device *device, int mirror);
+int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)

fs/btrfs/zoned.h static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 59

Quoted 3 lines
fs/btrfs/zoned.h +0 -0
@@ -48,6 +57,26 @@ static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info)
 {
 	return 0;
 }

newline

fs/btrfs/zoned.h static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 66

Quoted 7 lines
fs/btrfs/zoned.h +7 -0
+static inline int btrfs_sb_log_location_bdev(struct block_device *bdev,
+					     int mirror, int rw,
+					     u64 *bytenr_ret)
+{
+	*bytenr_ret = btrfs_sb_offset(mirror);
+	return 0;
+}

newline

fs/btrfs/zoned.h static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 72

Quoted 6 lines
fs/btrfs/zoned.h +6 -0
+static inline int btrfs_sb_log_location(struct btrfs_device *device, int mirror,
+					int rw, u64 *bytenr_ret)
+{
+	*bytenr_ret = btrfs_sb_offset(mirror);
+	return 0;
+}

newline

fs/btrfs/zoned.h static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 74

Quoted 2 lines
fs/btrfs/zoned.h +2 -0
+static inline void btrfs_advance_sb_log(struct btrfs_device *device,
+					int mirror) { }

newline

fs/btrfs/zoned.h static inline int btrfs_check_mountopts_zoned(struct btrfs_fs_info *info) · line 79

Quoted 5 lines
fs/btrfs/zoned.h +5 -0
+static inline int btrfs_reset_sb_log_zones(struct block_device *bdev,
+					   int mirror)
+{
+	return 0;
+}

newline

Quoted 21 lines
fs/btrfs/zoned.h +11 -0
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
@@ -115,4 +144,15 @@ static inline bool btrfs_check_device_zone_type(struct btrfs_fs_info *fs_info,
 	return bdev_zoned_model(bdev) != BLK_ZONED_HM;
 }
 
+static inline bool btrfs_check_super_location(struct btrfs_device *device,
+					      u64 pos)
+{
+	/*
+	 * On a non-zoned device, any address is OK. On a zoned device,
+	 * non-SEQUENTIAL WRITE REQUIRED zones are capable.
+	 */
+	return device->zone_info == NULL ||
+	       !btrfs_dev_is_sequential(device, pos);
+}
+
 #endif
Signature · 1 line
2.27.0
#39 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

In reply to Naohiro Aota
Quoted 24 lines · Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

Quoted 7 lines

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

I don't have a clear picture here.

In case there's a conventional zone covering 0 and 1st copy (64K and 64M) it'll be overwritten. What happens for 2nd copy that's at 256G? sb-log?

For all-sequential drive, the 0 and 1st copy are in the first zone. You say 0 and 1, but how come if the minimum zone size we ever expect is 256M?

The circular buffer comprises zones covering all superblock copies? I mean one buffer for 2 or more sb copies? The problem is that we'll have just one copy of the current superblock. Or I misunderstood.

My idea is that we have primary zone, unfortunatelly covering 2 superblocks but let it be. Second zone contains 2nd superblock copy (256G), we can assume that devices will be bigger than that.

Then the circular buffers happen in each zone, so first one will go from offset 64K up to the zone size (256M or 1G). Second zone rotates from offset 0 to end of the zone.

The positive outcome of that is that both zones contain the latest superblock after succesful write and their write pointer is slightly out of sync, so they never have to be reset at the same time.

In numbers:
- first zone 64K .. 256M, 65520 superblocks
- second zone 256G .. 245G+256M, 65536 superblocks

The difference is 16 superblock updates, which should be enough to let the zone resets happen far apart.

#40 lore raw

Re: [PATCH v9 11/41] btrfs: implement log-structured superblock for ZONED mode

Quoted 35 lines · David Sterba
Quoted 24 lines · Naohiro Aota

Superblock (and its copies) is the only data structure in btrfs which has a fixed location on a device. Since we cannot overwrite in a sequential write required zone, we cannot place superblock in the zone. One easy solution is limiting superblock and copies to be placed only in conventional zones. However, this method has two downsides: one is reduced number of superblock copies. The location of the second copy of superblock is 256GB, which is in a sequential write required zone on typical devices in the market today. So, the number of superblock and copies is limited to be two. Second downside is that we cannot support devices which have no conventional zones at all.

To solve these two problems, we employ superblock log writing. It uses two zones as a circular buffer to write updated superblocks. Once the first zone is filled up, start writing into the second buffer. Then, when the both zones are filled up and before start writing to the first zone again, it reset the first zone.

We can determine the position of the latest superblock by reading write pointer information from a device. One corner case is when the both zones are full. For this situation, we read out the last superblock of each zone, and compare them to determine which zone is older.

The following zones are reserved as the circular buffer on ZONED btrfs.

Quoted 7 lines

- The primary superblock: zones 0 and 1
- The first copy: zones 16 and 17
- The second copy: zones 1024 or zone at 256GB which is minimum, and next to it

If these reserved zones are conventional, superblock is written fixed at the start of the zone without logging.

I don't have a clear picture here.

Quoted 7 lines

In case there's a conventional zone covering 0 and 1st copy (64K and 64M) it'll be overwritten. What happens for 2nd copy that's at 256G? sb-log?

For all-sequential drive, the 0 and 1st copy are in the first zone. You say 0 and 1, but how come if the minimum zone size we ever expect is 256M?

On zoned device, we always reserve the above zones (0, 1, 16, 17, 1024, 1025 (or zones at 256G)) regardless of it is sequential or conventional. And, if the reserved zones is conventional, we write a superblock always at the beginning of the reserved zone. So, if a drive have 32 conventional zones, superblocks are placed at the beginning of zone 0 and zone 16. And, zone 1024 and 1025 are written with sb-log.

Quoted 4 lines

The circular buffer comprises zones covering all superblock copies? I mean one buffer for 2 or more sb copies? The problem is that we'll have just one copy of the current superblock. Or I misunderstood.

A circular buffer consists with a pair of the zones, so we'll have three sb-logs for each on zone pairs 0 & 1, zones 16 & 17, and 1024 & 1025.

Quoted 13 lines

My idea is that we have primary zone, unfortunatelly covering 2 superblocks but let it be. Second zone contains 2nd superblock copy (256G), we can assume that devices will be bigger than that.

Then the circular buffers happen in each zone, so first one will go from offset 64K up to the zone size (256M or 1G). Second zone rotates from offset 0 to end of the zone.

The positive outcome of that is that both zones contain the latest superblock after succesful write and their write pointer is slightly out of sync, so they never have to be reset at the same time.

Quoted 6 lines

In numbers:
- first zone 64K .. 256M, 65520 superblocks
- second zone 256G .. 245G+256M, 65536 superblocks

The difference is 16 superblock updates, which should be enough to let the zone resets happen far apart.

Hmm, this makes the minimal FS size requirement to 256 GB to survive a crash after resetting the first zone... So, that's why we have two zones as a circular buffer.

#41 lore raw

PATCH v9 12/41 btrfs: implement zoned chunk allocator

In reply to Naohiro Aota

This commit implements a zoned chunk/dev_extent allocator. The zoned allocator aligns the device extents to zone boundaries, so that a zone reset affects only the device extent and does not change the state of blocks in the neighbor device extents.

Also, it checks that a region allocation is not overlapping any of the super block zones, and ensures the region is empty.

---
 fs/btrfs/volumes.c | 131 +++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |   1 +
 fs/btrfs/zoned.c   | 126 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h   |  30 +++++++++++
 4 files changed, 288 insertions(+)
fs/btrfs/volumes.c +131 -0
index db884b96a5ea..78c62ef02e6f 100644
@@ -1416,6 +1416,14 @@ static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
 	return false;
 }
 
+static inline u64 dev_extent_search_start_zoned(struct btrfs_device *device,
+						u64 start)
+{
+	start = max_t(u64, start,
+		      max_t(u64, device->zone_info->zone_size, SZ_1M));
+	return btrfs_zone_align(device, start);
+}
+
 static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
 {
 	switch (device->fs_devices->chunk_alloc_policy) {
@@ -1426,11 +1434,57 @@ static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
 		 * make sure to start at an offset of at least 1MB.
 		 */
 		return max_t(u64, start, SZ_1M);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return dev_extent_search_start_zoned(device, start);
 	default:
 		BUG();
 	}
 }
 
+static bool dev_extent_hole_check_zoned(struct btrfs_device *device,
+					u64 *hole_start, u64 *hole_size,
+					u64 num_bytes)
+{
+	u64 zone_size = device->zone_info->zone_size;
+	u64 pos;
+	int ret;
+	int changed = 0;
+
+	ASSERT(IS_ALIGNED(*hole_start, zone_size));
+
+	while (*hole_size > 0) {
+		pos = btrfs_find_allocatable_zones(device, *hole_start,
+						   *hole_start + *hole_size,
+						   num_bytes);
+		if (pos != *hole_start) {
+			*hole_size = *hole_start + *hole_size - pos;
+			*hole_start = pos;
+			changed = 1;
+			if (*hole_size < num_bytes)
+				break;
+		}
+
+		ret = btrfs_ensure_empty_zones(device, pos, num_bytes);
+
+		/* range is ensured to be empty */
+		if (!ret)
+			return changed;
+
+		/* given hole range was invalid (outside of device) */
+		if (ret == -ERANGE) {
+			*hole_start += *hole_size;
+			*hole_size = 0;
+			return 1;
+		}
+
+		*hole_start += zone_size;
+		*hole_size -= zone_size;
+		changed = 1;
+	}
+
+	return changed;
+}
+
 /**
  * dev_extent_hole_check - check if specified hole is suitable for allocation
  * @device:	the device which we have the hole
@@ -1463,6 +1517,10 @@ static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start,
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		/* No extra check */
 		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		changed |= dev_extent_hole_check_zoned(device, hole_start,
+						       hole_size, num_bytes);
+		break;
 	default:
 		BUG();
 	}
@@ -1517,6 +1575,9 @@ static int find_free_dev_extent_start(struct btrfs_device *device,
 
 	search_start = dev_extent_search_start(device, search_start);
 
+	WARN_ON(device->zone_info &&
+		!IS_ALIGNED(num_bytes, device->zone_info->zone_size));
+
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
@@ -4907,6 +4968,37 @@ static void init_alloc_chunk_ctl_policy_regular(
 	ctl->dev_extent_min = BTRFS_STRIPE_LEN * ctl->dev_stripes;
 }
 
+static void
+init_alloc_chunk_ctl_policy_zoned(struct btrfs_fs_devices *fs_devices,
+				  struct alloc_chunk_ctl *ctl)
+{
+	u64 zone_size = fs_devices->fs_info->zone_size;
+	u64 limit;
+	int min_num_stripes = ctl->devs_min * ctl->dev_stripes;
+	int min_data_stripes = (min_num_stripes - ctl->nparity) / ctl->ncopies;
+	u64 min_chunk_size = min_data_stripes * zone_size;
+	u64 type = ctl->type;
+
+	ctl->max_stripe_size = zone_size;
+	if (type & BTRFS_BLOCK_GROUP_DATA) {
+		ctl->max_chunk_size = round_down(BTRFS_MAX_DATA_CHUNK_SIZE,
+						 zone_size);
+	} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
+		ctl->max_chunk_size = ctl->max_stripe_size;
+	} else if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
+		ctl->max_chunk_size = 2 * ctl->max_stripe_size;
+		ctl->devs_max = min_t(int, ctl->devs_max,
+				      BTRFS_MAX_DEVS_SYS_CHUNK);
+	}
+
+	/* We don't want a chunk larger than 10% of writable space */
+	limit = max(round_down(div_factor(fs_devices->total_rw_bytes, 1),
+			       zone_size),
+		    min_chunk_size);
+	ctl->max_chunk_size = min(limit, ctl->max_chunk_size);
+	ctl->dev_extent_min = zone_size * ctl->dev_stripes;
+}
+
 static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
 				 struct alloc_chunk_ctl *ctl)
 {
@@ -4927,6 +5019,9 @@ static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		init_alloc_chunk_ctl_policy_regular(fs_devices, ctl);
 		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		init_alloc_chunk_ctl_policy_zoned(fs_devices, ctl);
+		break;
 	default:
 		BUG();
 	}
@@ -5053,6 +5148,40 @@ static int decide_stripe_size_regular(struct alloc_chunk_ctl *ctl,
 	return 0;
 }
 
+static int decide_stripe_size_zoned(struct alloc_chunk_ctl *ctl,
+				    struct btrfs_device_info *devices_info)
+{
+	u64 zone_size = devices_info[0].dev->zone_info->zone_size;
+	/* number of stripes that count for block group size */
+	int data_stripes;
+
+	/*
+	 * It should hold because:
+	 *    dev_extent_min == dev_extent_want == zone_size * dev_stripes
+	 */
+	ASSERT(devices_info[ctl->ndevs - 1].max_avail == ctl->dev_extent_min);
+
+	ctl->stripe_size = zone_size;
+	ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+	data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+
+	/*
+	 * stripe_size is fixed in ZONED. Reduce ndevs instead.
+	 */
+	if (ctl->stripe_size * data_stripes > ctl->max_chunk_size) {
+		ctl->ndevs = div_u64(div_u64(ctl->max_chunk_size * ctl->ncopies,
+					     ctl->stripe_size) + ctl->nparity,
+				     ctl->dev_stripes);
+		ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+		data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+		ASSERT(ctl->stripe_size * data_stripes <= ctl->max_chunk_size);
+	}
+
+	ctl->chunk_size = ctl->stripe_size * data_stripes;
+
+	return 0;
+}
+
 static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
 			      struct alloc_chunk_ctl *ctl,
 			      struct btrfs_device_info *devices_info)
@@ -5080,6 +5209,8 @@ static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
 	switch (fs_devices->chunk_alloc_policy) {
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		return decide_stripe_size_regular(ctl, devices_info);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return decide_stripe_size_zoned(ctl, devices_info);
 	default:
 		BUG();
 	}
fs/btrfs/volumes.h +1 -0
index 9c07b97a2260..0249aca668fb 100644
@@ -213,6 +213,7 @@ BTRFS_DEVICE_GETSET_FUNCS(bytes_used);
 
 enum btrfs_chunk_allocation_policy {
 	BTRFS_CHUNK_ALLOC_REGULAR,
+	BTRFS_CHUNK_ALLOC_ZONED,
 };
 
 struct btrfs_fs_devices {
fs/btrfs/zoned.c +126 -0

146 lines not on this page · raw message

fs/btrfs/zoned.h +30 -0
index 447c4e5ffcbb..24dd0c9561f9 100644
@@ -34,6 +34,11 @@ int btrfs_sb_log_location(struct btrfs_device *device, int mirror, int rw,
 			  u64 *bytenr_ret);
 void btrfs_advance_sb_log(struct btrfs_device *device, int mirror);
 int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror);
+u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
+				 u64 hole_end, u64 num_bytes);
+int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
+			    u64 length, u64 *bytes);
+int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -77,6 +82,23 @@ static inline int btrfs_reset_sb_log_zones(struct block_device *bdev,
 {
 	return 0;
 }
+static inline u64 btrfs_find_allocatable_zones(struct btrfs_device *device,
+					       u64 hole_start, u64 hole_end,
+					       u64 num_bytes)
+{
+	return hole_start;
+}
+static inline int btrfs_reset_device_zone(struct btrfs_device *device,
+					  u64 physical, u64 length, u64 *bytes)
+{
+	*bytes = 0;
+	return 0;
+}
+static inline int btrfs_ensure_empty_zones(struct btrfs_device *device,
+					   u64 start, u64 size)
+{
+	return 0;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
@@ -155,4 +177,12 @@ static inline bool btrfs_check_super_location(struct btrfs_device *device,
 	       !btrfs_dev_is_sequential(device, pos);
 }
 
+static inline u64 btrfs_zone_align(struct btrfs_device *device, u64 pos)
+{
+	if (!device->zone_info)
+		return pos;
+
+	return ALIGN(pos, device->zone_info->zone_size);
+}
+
 #endif
Signature · 1 line
2.27.0
#42 lore raw

Re: [PATCH v9 12/41] btrfs: implement zoned chunk allocator

Quoted 94 lines · Naohiro Aota

This commit implements a zoned chunk/dev_extent allocator. The zoned allocator aligns the device extents to zone boundaries, so that a zone reset affects only the device extent and does not change the state of blocks in the neighbor device extents.

Also, it checks that a region allocation is not overlapping any of the super block zones, and ensures the region is empty.

---
  fs/btrfs/volumes.c | 131 +++++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |   1 +
  fs/btrfs/zoned.c   | 126 +++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/zoned.h   |  30 +++++++++++
  4 files changed, 288 insertions(+)
fs/btrfs/volumes.c +54 -0
index db884b96a5ea..78c62ef02e6f 100644
@@ -1416,6 +1416,14 @@ static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
  	return false;
  }
 
+static inline u64 dev_extent_search_start_zoned(struct btrfs_device *device,
+						u64 start)
+{
+	start = max_t(u64, start,
+		      max_t(u64, device->zone_info->zone_size, SZ_1M));
+	return btrfs_zone_align(device, start);
+}
+
  static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
  {
  	switch (device->fs_devices->chunk_alloc_policy) {
@@ -1426,11 +1434,57 @@ static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
  		 * make sure to start at an offset of at least 1MB.
  		 */
  		return max_t(u64, start, SZ_1M);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return dev_extent_search_start_zoned(device, start);
  	default:
  		BUG();
  	}
  }
 
+static bool dev_extent_hole_check_zoned(struct btrfs_device *device,
+					u64 *hole_start, u64 *hole_size,
+					u64 num_bytes)
+{
+	u64 zone_size = device->zone_info->zone_size;
+	u64 pos;
+	int ret;
+	int changed = 0;
+
+	ASSERT(IS_ALIGNED(*hole_start, zone_size));
+
+	while (*hole_size > 0) {
+		pos = btrfs_find_allocatable_zones(device, *hole_start,
+						   *hole_start + *hole_size,
+						   num_bytes);
+		if (pos != *hole_start) {
+			*hole_size = *hole_start + *hole_size - pos;
+			*hole_start = pos;
+			changed = 1;
+			if (*hole_size < num_bytes)
+				break;
+		}
+
+		ret = btrfs_ensure_empty_zones(device, pos, num_bytes);
+
+		/* range is ensured to be empty */
+		if (!ret)
+			return changed;
+
+		/* given hole range was invalid (outside of device) */
+		if (ret == -ERANGE) {
+			*hole_start += *hole_size;
+			*hole_size = 0;
+			return 1;
+		}
+
+		*hole_start += zone_size;
+		*hole_size -= zone_size;
+		changed = 1;
+	}
+
+	return changed;
+}
+
  /**
   * dev_extent_hole_check - check if specified hole is suitable for allocation
   * @device:	the device which we have the hole

fs/btrfs/volumes.c static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start, · line 1522

Quoted 6 lines
fs/btrfs/volumes.c +3 -0
@@ -1463,6 +1517,10 @@ static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start,
  	case BTRFS_CHUNK_ALLOC_REGULAR:
  		/* No extra check */
  		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		changed |= dev_extent_hole_check_zoned(device, hole_start,
+						       hole_size, num_bytes);

I'm confused here, we check to make sure the pending stuff doesn't overlap with non-empty zones. However we don't ever actually mark zones as non-empty except on mount. I realize that if we allocate this zone then it appears pending and thus we won't allocate with this zone again while the fs is mounted, but it took me a while to realize this. Is there a reason to not mark a zone as non-empty when we allocate from it?

Quoted 240 lines
fs/btrfs/volumes.c +74 -0
+		break;
  	default:
  		BUG();
  	}
@@ -1517,6 +1575,9 @@ static int find_free_dev_extent_start(struct btrfs_device *device,
 
  	search_start = dev_extent_search_start(device, search_start);
 
+	WARN_ON(device->zone_info &&
+		!IS_ALIGNED(num_bytes, device->zone_info->zone_size));
+
  	path = btrfs_alloc_path();
  	if (!path)
  		return -ENOMEM;
@@ -4907,6 +4968,37 @@ static void init_alloc_chunk_ctl_policy_regular(
  	ctl->dev_extent_min = BTRFS_STRIPE_LEN * ctl->dev_stripes;
  }
 
+static void
+init_alloc_chunk_ctl_policy_zoned(struct btrfs_fs_devices *fs_devices,
+				  struct alloc_chunk_ctl *ctl)
+{
+	u64 zone_size = fs_devices->fs_info->zone_size;
+	u64 limit;
+	int min_num_stripes = ctl->devs_min * ctl->dev_stripes;
+	int min_data_stripes = (min_num_stripes - ctl->nparity) / ctl->ncopies;
+	u64 min_chunk_size = min_data_stripes * zone_size;
+	u64 type = ctl->type;
+
+	ctl->max_stripe_size = zone_size;
+	if (type & BTRFS_BLOCK_GROUP_DATA) {
+		ctl->max_chunk_size = round_down(BTRFS_MAX_DATA_CHUNK_SIZE,
+						 zone_size);
+	} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
+		ctl->max_chunk_size = ctl->max_stripe_size;
+	} else if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
+		ctl->max_chunk_size = 2 * ctl->max_stripe_size;
+		ctl->devs_max = min_t(int, ctl->devs_max,
+				      BTRFS_MAX_DEVS_SYS_CHUNK);
+	}
+
+	/* We don't want a chunk larger than 10% of writable space */
+	limit = max(round_down(div_factor(fs_devices->total_rw_bytes, 1),
+			       zone_size),
+		    min_chunk_size);
+	ctl->max_chunk_size = min(limit, ctl->max_chunk_size);
+	ctl->dev_extent_min = zone_size * ctl->dev_stripes;
+}
+
  static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
  				 struct alloc_chunk_ctl *ctl)
  {
@@ -4927,6 +5019,9 @@ static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
  	case BTRFS_CHUNK_ALLOC_REGULAR:
  		init_alloc_chunk_ctl_policy_regular(fs_devices, ctl);
  		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		init_alloc_chunk_ctl_policy_zoned(fs_devices, ctl);
+		break;
  	default:
  		BUG();
  	}
@@ -5053,6 +5148,40 @@ static int decide_stripe_size_regular(struct alloc_chunk_ctl *ctl,
  	return 0;
  }
 
+static int decide_stripe_size_zoned(struct alloc_chunk_ctl *ctl,
+				    struct btrfs_device_info *devices_info)
+{
+	u64 zone_size = devices_info[0].dev->zone_info->zone_size;
+	/* number of stripes that count for block group size */
+	int data_stripes;
+
+	/*
+	 * It should hold because:
+	 *    dev_extent_min == dev_extent_want == zone_size * dev_stripes
+	 */
+	ASSERT(devices_info[ctl->ndevs - 1].max_avail == ctl->dev_extent_min);
+
+	ctl->stripe_size = zone_size;
+	ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+	data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+
+	/*
+	 * stripe_size is fixed in ZONED. Reduce ndevs instead.
+	 */
+	if (ctl->stripe_size * data_stripes > ctl->max_chunk_size) {
+		ctl->ndevs = div_u64(div_u64(ctl->max_chunk_size * ctl->ncopies,
+					     ctl->stripe_size) + ctl->nparity,
+				     ctl->dev_stripes);
+		ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+		data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+		ASSERT(ctl->stripe_size * data_stripes <= ctl->max_chunk_size);
+	}
+
+	ctl->chunk_size = ctl->stripe_size * data_stripes;
+
+	return 0;
+}
+
  static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
  			      struct alloc_chunk_ctl *ctl,
  			      struct btrfs_device_info *devices_info)
@@ -5080,6 +5209,8 @@ static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
  	switch (fs_devices->chunk_alloc_policy) {
  	case BTRFS_CHUNK_ALLOC_REGULAR:
  		return decide_stripe_size_regular(ctl, devices_info);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return decide_stripe_size_zoned(ctl, devices_info);
  	default:
  		BUG();
  	}
fs/btrfs/volumes.h +1 -0
index 9c07b97a2260..0249aca668fb 100644
@@ -213,6 +213,7 @@ BTRFS_DEVICE_GETSET_FUNCS(bytes_used);
 
  enum btrfs_chunk_allocation_policy {
  	BTRFS_CHUNK_ALLOC_REGULAR,
+	BTRFS_CHUNK_ALLOC_ZONED,
  };
 
  struct btrfs_fs_devices {
fs/btrfs/zoned.c +89 -0
index d5487cba203b..4411d786597a 100644
@@ -1,11 +1,13 @@
  // SPDX-License-Identifier: GPL-2.0
 
+#include <linux/bitops.h>
  #include <linux/slab.h>
  #include <linux/blkdev.h>
  #include "ctree.h"
  #include "volumes.h"
  #include "zoned.h"
  #include "rcu-string.h"
+#include "disk-io.h"
 
  /* Maximum number of zones to report per blkdev_report_zones() call */
  #define BTRFS_REPORT_NR_ZONES   4096
@@ -328,6 +330,7 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 
  	fs_info->zone_size = zone_size;
  	fs_info->max_zone_append_size = max_zone_append_size;
+	fs_info->fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_ZONED;
 
  	btrfs_info(fs_info, "ZONED mode enabled, zone size %llu B",
  		   fs_info->zone_size);
@@ -607,3 +610,126 @@ int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror)
  				sb_zone << zone_sectors_shift, zone_sectors * 2,
  				GFP_NOFS);
  }
+
+/*
+ * btrfs_check_allocatable_zones - find allocatable zones within give region
+ * @device:	the device to allocate a region
+ * @hole_start: the position of the hole to allocate the region
+ * @num_bytes:	the size of wanted region
+ * @hole_size:	the size of hole
+ *
+ * Allocatable region should not contain any superblock locations.
+ */
+u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
+				 u64 hole_end, u64 num_bytes)
+{
+	struct btrfs_zoned_device_info *zinfo = device->zone_info;
+	u8 shift = zinfo->zone_size_shift;
+	u64 nzones = num_bytes >> shift;
+	u64 pos = hole_start;
+	u64 begin, end;
+	u64 sb_pos;
+	bool have_sb;
+	int i;
+
+	ASSERT(IS_ALIGNED(hole_start, zinfo->zone_size));
+	ASSERT(IS_ALIGNED(num_bytes, zinfo->zone_size));
+
+	while (pos < hole_end) {
+		begin = pos >> shift;
+		end = begin + nzones;
+
+		if (end > zinfo->nr_zones)
+			return hole_end;
+
+		/* check if zones in the region are all empty */
+		if (btrfs_dev_is_sequential(device, pos) &&
+		    find_next_zero_bit(zinfo->empty_zones, end, begin) != end) {
+			pos += zinfo->zone_size;
+			continue;
+		}
+
+		have_sb = false;
+		for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
+			sb_pos = sb_zone_number(zinfo->zone_size, i);
+			if (!(end < sb_pos || sb_pos + 1 < begin)) {
+				have_sb = true;
+				pos = (sb_pos + 2) << shift;
+				break;
+			}
+		}
+		if (!have_sb)
+			break;
+	}
+
+	return pos;
+}
+
+int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
+			    u64 length, u64 *bytes)
+{
+	int ret;
+
+	*bytes = 0;
+	ret = blkdev_zone_mgmt(device->bdev, REQ_OP_ZONE_RESET,
+			       physical >> SECTOR_SHIFT, length >> SECTOR_SHIFT,
+			       GFP_NOFS);
+	if (ret)
+		return ret;
+
+	*bytes = length;
+	while (length) {
+		btrfs_dev_set_zone_empty(device, physical);
+		physical += device->zone_info->zone_size;
+		length -= device->zone_info->zone_size;
+	}
+
+	return 0;
+}
+
+int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
+{
+	struct btrfs_zoned_device_info *zinfo = device->zone_info;
+	u8 shift = zinfo->zone_size_shift;
+	unsigned long begin = start >> shift;
+	unsigned long end = (start + size) >> shift;
+	u64 pos;
+	int ret;
+

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 708

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	ASSERT(IS_ALIGNED(start, zinfo->zone_size));
+	ASSERT(IS_ALIGNED(size, zinfo->zone_size));
+
+	if (end > zinfo->nr_zones)
+		return -ERANGE;
+
+	/* all the zones are conventional */
+	if (find_next_bit(zinfo->seq_zones, begin, end) == end)
+		return 0;
+

This check is duplicated below.

Quoted 13 lines
fs/btrfs/zoned.c +13 -0
+	/* all the zones are sequential and empty */
+	if (find_next_zero_bit(zinfo->seq_zones, begin, end) == end &&
+	    find_next_zero_bit(zinfo->empty_zones, begin, end) == end)
+		return 0;
+
+	for (pos = start; pos < start + size; pos += zinfo->zone_size) {
+		u64 reset_bytes;
+
+		if (!btrfs_dev_is_sequential(device, pos) ||
+		    btrfs_dev_is_empty_zone(device, pos))
+			continue;
+
+		/* free regions should be empty */

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 731

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		btrfs_warn_in_rcu(
+			device->fs_info,
+			"resetting device %s zone %llu for allocation",
+			rcu_str_deref(device->name), pos >> shift);
+		WARN_ON_ONCE(1);
+
+		ret = btrfs_reset_device_zone(device, pos, zinfo->zone_size,
+					      &reset_bytes);
+		if (ret)
+			return ret;

This seems bad, as we could just have corruption right? So we're resetting the zone which could lose us data right? Shouldn't we just bail here? Thanks,

Josef

#43 lore raw

Re: [PATCH v9 12/41] btrfs: implement zoned chunk allocator

Quoted 92 lines · Josef Bacik
Quoted 91 lines · Naohiro Aota

This commit implements a zoned chunk/dev_extent allocator. The zoned allocator aligns the device extents to zone boundaries, so that a zone reset affects only the device extent and does not change the state of blocks in the neighbor device extents.

Also, it checks that a region allocation is not overlapping any of the super block zones, and ensures the region is empty.

---
 fs/btrfs/volumes.c | 131 +++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |   1 +
 fs/btrfs/zoned.c   | 126 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h   |  30 +++++++++++
 4 files changed, 288 insertions(+)
fs/btrfs/volumes.c +54 -0
index db884b96a5ea..78c62ef02e6f 100644
@@ -1416,6 +1416,14 @@ static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
 	return false;
 }
+static inline u64 dev_extent_search_start_zoned(struct btrfs_device *device,
+						u64 start)
+{
+	start = max_t(u64, start,
+		      max_t(u64, device->zone_info->zone_size, SZ_1M));
+	return btrfs_zone_align(device, start);
+}
+
 static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
 {
 	switch (device->fs_devices->chunk_alloc_policy) {
@@ -1426,11 +1434,57 @@ static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
 		 * make sure to start at an offset of at least 1MB.
 		 */
 		return max_t(u64, start, SZ_1M);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return dev_extent_search_start_zoned(device, start);
 	default:
 		BUG();
 	}
 }
+static bool dev_extent_hole_check_zoned(struct btrfs_device *device,
+					u64 *hole_start, u64 *hole_size,
+					u64 num_bytes)
+{
+	u64 zone_size = device->zone_info->zone_size;
+	u64 pos;
+	int ret;
+	int changed = 0;
+
+	ASSERT(IS_ALIGNED(*hole_start, zone_size));
+
+	while (*hole_size > 0) {
+		pos = btrfs_find_allocatable_zones(device, *hole_start,
+						   *hole_start + *hole_size,
+						   num_bytes);
+		if (pos != *hole_start) {
+			*hole_size = *hole_start + *hole_size - pos;
+			*hole_start = pos;
+			changed = 1;
+			if (*hole_size < num_bytes)
+				break;
+		}
+
+		ret = btrfs_ensure_empty_zones(device, pos, num_bytes);
+
+		/* range is ensured to be empty */
+		if (!ret)
+			return changed;
+
+		/* given hole range was invalid (outside of device) */
+		if (ret == -ERANGE) {
+			*hole_start += *hole_size;
+			*hole_size = 0;
+			return 1;
+		}
+
+		*hole_start += zone_size;
+		*hole_size -= zone_size;
+		changed = 1;
+	}
+
+	return changed;
+}
+
 /**
  * dev_extent_hole_check - check if specified hole is suitable for allocation
  * @device:	the device which we have the hole
Quoted 8 lines

fs/btrfs/volumes.c static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start, · line 1522

Quoted 7 lines
fs/btrfs/volumes.c +3 -0
@@ -1463,6 +1517,10 @@ static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start,
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		/* No extra check */
 		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		changed |= dev_extent_hole_check_zoned(device, hole_start,
+						       hole_size, num_bytes);

I'm confused here, we check to make sure the pending stuff doesn't overlap with non-empty zones. However we don't ever actually mark zones as non-empty except on mount. I realize that if we allocate this zone then it appears pending and thus we won't allocate with this zone again while the fs is mounted, but it took me a while to realize this. Is there a reason to not mark a zone as non-empty when we allocate from it?

It's marking the zones as non-empty. Allocated zones eventually construct a block group in btrfs_make_block_group(). The function calls btrfs_load_block_group_zone_info() and that will clear the empty flag.

Quoted 236 lines
Quoted 236 lines
fs/btrfs/volumes.c +74 -0
+		break;
 	default:
 		BUG();
 	}
@@ -1517,6 +1575,9 @@ static int find_free_dev_extent_start(struct btrfs_device *device,
 	search_start = dev_extent_search_start(device, search_start);
+	WARN_ON(device->zone_info &&
+		!IS_ALIGNED(num_bytes, device->zone_info->zone_size));
+
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
@@ -4907,6 +4968,37 @@ static void init_alloc_chunk_ctl_policy_regular(
 	ctl->dev_extent_min = BTRFS_STRIPE_LEN * ctl->dev_stripes;
 }
+static void
+init_alloc_chunk_ctl_policy_zoned(struct btrfs_fs_devices *fs_devices,
+				  struct alloc_chunk_ctl *ctl)
+{
+	u64 zone_size = fs_devices->fs_info->zone_size;
+	u64 limit;
+	int min_num_stripes = ctl->devs_min * ctl->dev_stripes;
+	int min_data_stripes = (min_num_stripes - ctl->nparity) / ctl->ncopies;
+	u64 min_chunk_size = min_data_stripes * zone_size;
+	u64 type = ctl->type;
+
+	ctl->max_stripe_size = zone_size;
+	if (type & BTRFS_BLOCK_GROUP_DATA) {
+		ctl->max_chunk_size = round_down(BTRFS_MAX_DATA_CHUNK_SIZE,
+						 zone_size);
+	} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
+		ctl->max_chunk_size = ctl->max_stripe_size;
+	} else if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
+		ctl->max_chunk_size = 2 * ctl->max_stripe_size;
+		ctl->devs_max = min_t(int, ctl->devs_max,
+				      BTRFS_MAX_DEVS_SYS_CHUNK);
+	}
+
+	/* We don't want a chunk larger than 10% of writable space */
+	limit = max(round_down(div_factor(fs_devices->total_rw_bytes, 1),
+			       zone_size),
+		    min_chunk_size);
+	ctl->max_chunk_size = min(limit, ctl->max_chunk_size);
+	ctl->dev_extent_min = zone_size * ctl->dev_stripes;
+}
+
 static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
 				 struct alloc_chunk_ctl *ctl)
 {
@@ -4927,6 +5019,9 @@ static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		init_alloc_chunk_ctl_policy_regular(fs_devices, ctl);
 		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		init_alloc_chunk_ctl_policy_zoned(fs_devices, ctl);
+		break;
 	default:
 		BUG();
 	}
@@ -5053,6 +5148,40 @@ static int decide_stripe_size_regular(struct alloc_chunk_ctl *ctl,
 	return 0;
 }
+static int decide_stripe_size_zoned(struct alloc_chunk_ctl *ctl,
+				    struct btrfs_device_info *devices_info)
+{
+	u64 zone_size = devices_info[0].dev->zone_info->zone_size;
+	/* number of stripes that count for block group size */
+	int data_stripes;
+
+	/*
+	 * It should hold because:
+	 *    dev_extent_min == dev_extent_want == zone_size * dev_stripes
+	 */
+	ASSERT(devices_info[ctl->ndevs - 1].max_avail == ctl->dev_extent_min);
+
+	ctl->stripe_size = zone_size;
+	ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+	data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+
+	/*
+	 * stripe_size is fixed in ZONED. Reduce ndevs instead.
+	 */
+	if (ctl->stripe_size * data_stripes > ctl->max_chunk_size) {
+		ctl->ndevs = div_u64(div_u64(ctl->max_chunk_size * ctl->ncopies,
+					     ctl->stripe_size) + ctl->nparity,
+				     ctl->dev_stripes);
+		ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+		data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+		ASSERT(ctl->stripe_size * data_stripes <= ctl->max_chunk_size);
+	}
+
+	ctl->chunk_size = ctl->stripe_size * data_stripes;
+
+	return 0;
+}
+
 static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
 			      struct alloc_chunk_ctl *ctl,
 			      struct btrfs_device_info *devices_info)
@@ -5080,6 +5209,8 @@ static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
 	switch (fs_devices->chunk_alloc_policy) {
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		return decide_stripe_size_regular(ctl, devices_info);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return decide_stripe_size_zoned(ctl, devices_info);
 	default:
 		BUG();
 	}
fs/btrfs/volumes.h +1 -0
index 9c07b97a2260..0249aca668fb 100644
@@ -213,6 +213,7 @@ BTRFS_DEVICE_GETSET_FUNCS(bytes_used);
 enum btrfs_chunk_allocation_policy {
 	BTRFS_CHUNK_ALLOC_REGULAR,
+	BTRFS_CHUNK_ALLOC_ZONED,
 };
 struct btrfs_fs_devices {
fs/btrfs/zoned.c +93 -0
index d5487cba203b..4411d786597a 100644
@@ -1,11 +1,13 @@
 // SPDX-License-Identifier: GPL-2.0
+#include <linux/bitops.h>
 #include <linux/slab.h>
 #include <linux/blkdev.h>
 #include "ctree.h"
 #include "volumes.h"
 #include "zoned.h"
 #include "rcu-string.h"
+#include "disk-io.h"
 /* Maximum number of zones to report per blkdev_report_zones() call */
 #define BTRFS_REPORT_NR_ZONES   4096
@@ -328,6 +330,7 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info)
 	fs_info->zone_size = zone_size;
 	fs_info->max_zone_append_size = max_zone_append_size;
+	fs_info->fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_ZONED;
 	btrfs_info(fs_info, "ZONED mode enabled, zone size %llu B",
 		   fs_info->zone_size);
@@ -607,3 +610,126 @@ int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror)
 				sb_zone << zone_sectors_shift, zone_sectors * 2,
 				GFP_NOFS);
 }
+
+/*
+ * btrfs_check_allocatable_zones - find allocatable zones within give region
+ * @device:	the device to allocate a region
+ * @hole_start: the position of the hole to allocate the region
+ * @num_bytes:	the size of wanted region
+ * @hole_size:	the size of hole
+ *
+ * Allocatable region should not contain any superblock locations.
+ */
+u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
+				 u64 hole_end, u64 num_bytes)
+{
+	struct btrfs_zoned_device_info *zinfo = device->zone_info;
+	u8 shift = zinfo->zone_size_shift;
+	u64 nzones = num_bytes >> shift;
+	u64 pos = hole_start;
+	u64 begin, end;
+	u64 sb_pos;
+	bool have_sb;
+	int i;
+
+	ASSERT(IS_ALIGNED(hole_start, zinfo->zone_size));
+	ASSERT(IS_ALIGNED(num_bytes, zinfo->zone_size));
+
+	while (pos < hole_end) {
+		begin = pos >> shift;
+		end = begin + nzones;
+
+		if (end > zinfo->nr_zones)
+			return hole_end;
+
+		/* check if zones in the region are all empty */
+		if (btrfs_dev_is_sequential(device, pos) &&
+		    find_next_zero_bit(zinfo->empty_zones, end, begin) != end) {
+			pos += zinfo->zone_size;
+			continue;
+		}
+
+		have_sb = false;
+		for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
+			sb_pos = sb_zone_number(zinfo->zone_size, i);
+			if (!(end < sb_pos || sb_pos + 1 < begin)) {
+				have_sb = true;
+				pos = (sb_pos + 2) << shift;
+				break;
+			}
+		}
+		if (!have_sb)
+			break;
+	}
+
+	return pos;
+}
+
+int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
+			    u64 length, u64 *bytes)
+{
+	int ret;
+
+	*bytes = 0;
+	ret = blkdev_zone_mgmt(device->bdev, REQ_OP_ZONE_RESET,
+			       physical >> SECTOR_SHIFT, length >> SECTOR_SHIFT,
+			       GFP_NOFS);
+	if (ret)
+		return ret;
+
+	*bytes = length;
+	while (length) {
+		btrfs_dev_set_zone_empty(device, physical);
+		physical += device->zone_info->zone_size;
+		length -= device->zone_info->zone_size;
+	}
+
+	return 0;
+}
+
+int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
+{
+	struct btrfs_zoned_device_info *zinfo = device->zone_info;
+	u8 shift = zinfo->zone_size_shift;
+	unsigned long begin = start >> shift;
+	unsigned long end = (start + size) >> shift;
+	u64 pos;
+	int ret;
+
+	ASSERT(IS_ALIGNED(start, zinfo->zone_size));
+	ASSERT(IS_ALIGNED(size, zinfo->zone_size));
+
+	if (end > zinfo->nr_zones)
Quoted 9 lines

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 708

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+		return -ERANGE;
+
+	/* all the zones are conventional */
+	if (find_next_bit(zinfo->seq_zones, begin, end) == end)
+		return 0;
+

This check is duplicated below.

This one checks if bits from begin to end is all cleared (= all zones are conventional). OTOH, the below checks if all the bits are set (= all zones are sequential).

Quoted 26 lines
Quoted 13 lines
fs/btrfs/zoned.c +13 -0
+	/* all the zones are sequential and empty */
+	if (find_next_zero_bit(zinfo->seq_zones, begin, end) == end &&
+	    find_next_zero_bit(zinfo->empty_zones, begin, end) == end)
+		return 0;
+
+	for (pos = start; pos < start + size; pos += zinfo->zone_size) {
+		u64 reset_bytes;
+
+		if (!btrfs_dev_is_sequential(device, pos) ||
+		    btrfs_dev_is_empty_zone(device, pos))
+			continue;
+
+		/* free regions should be empty */

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 731

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		btrfs_warn_in_rcu(
+			device->fs_info,
+			"resetting device %s zone %llu for allocation",
+			rcu_str_deref(device->name), pos >> shift);
+		WARN_ON_ONCE(1);
+
+		ret = btrfs_reset_device_zone(device, pos, zinfo->zone_size,
+					      &reset_bytes);
+		if (ret)
+			return ret;

This seems bad, as we could just have corruption right? So we're resetting the zone which could lose us data right? Shouldn't we just bail here? Thanks,

Quoted 1 line

Josef

Yes.. This happens 1) when we freed up the region but forget to reset the zones, or 2) when we lost the allocation information. For the first case, it's OK to reset the zones here. For the second case, it's still as much dangerous as regular device since the regular btrfs will overwrite the data in that case anyway. I admit it's much safer to bail out here, so I can change this to error exiting.

#44 lore raw

Re: [PATCH v9 12/41] btrfs: implement zoned chunk allocator

In reply to Naohiro Aota

fs/btrfs/volumes.c static bool contains_pending_extent(struct btrfs_device *device, u64 *start, · line 1423

Quoted 11 lines · Naohiro Aota
fs/btrfs/volumes.c +5 -0
@@ -1416,6 +1416,14 @@ static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
 	return false;
 }
 
+static inline u64 dev_extent_search_start_zoned(struct btrfs_device *device,
+						u64 start)
+{
+	start = max_t(u64, start,
+		      max_t(u64, device->zone_info->zone_size, SZ_1M));

Can you rewrite that as ifs?

Quoted 33 lines
fs/btrfs/volumes.c +21 -0
+	return btrfs_zone_align(device, start);
+}
+
 static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
 {
 	switch (device->fs_devices->chunk_alloc_policy) {
@@ -1426,11 +1434,57 @@ static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
 		 * make sure to start at an offset of at least 1MB.
 		 */
 		return max_t(u64, start, SZ_1M);
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		return dev_extent_search_start_zoned(device, start);
 	default:
 		BUG();
 	}
 }
 
+static bool dev_extent_hole_check_zoned(struct btrfs_device *device,
+					u64 *hole_start, u64 *hole_size,
+					u64 num_bytes)
+{
+	u64 zone_size = device->zone_info->zone_size;
+	u64 pos;
+	int ret;
+	int changed = 0;
+
+	ASSERT(IS_ALIGNED(*hole_start, zone_size));
+
+	while (*hole_size > 0) {
+		pos = btrfs_find_allocatable_zones(device, *hole_start,
+						   *hole_start + *hole_size,
+						   num_bytes);
+		if (pos != *hole_start) {

fs/btrfs/volumes.c static u64 dev_extent_search_start(struct btrfs_device *device, u64 start) · line 1469

Quoted 10 lines
fs/btrfs/volumes.c +10 -0
+			*hole_size = *hole_start + *hole_size - pos;
+			*hole_start = pos;
+			changed = 1;
+			if (*hole_size < num_bytes)
+				break;
+		}
+
+		ret = btrfs_ensure_empty_zones(device, pos, num_bytes);
+
+		/* range is ensured to be empty */
		/* Range ... */

fs/btrfs/volumes.c static u64 dev_extent_search_start(struct btrfs_device *device, u64 start) · line 1473

Quoted 4 lines
fs/btrfs/volumes.c +4 -0
+		if (!ret)
+			return changed;
+
+		/* given hole range was invalid (outside of device) */
		/* Given ... */
Quoted 39 lines
fs/btrfs/volumes.c +21 -0
+		if (ret == -ERANGE) {
+			*hole_start += *hole_size;
+			*hole_size = 0;
+			return 1;
+		}
+
+		*hole_start += zone_size;
+		*hole_size -= zone_size;
+		changed = 1;
+	}
+
+	return changed;
+}
+
 /**
  * dev_extent_hole_check - check if specified hole is suitable for allocation
  * @device:	the device which we have the hole
@@ -1463,6 +1517,10 @@ static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start,
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		/* No extra check */
 		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		changed |= dev_extent_hole_check_zoned(device, hole_start,
+						       hole_size, num_bytes);
+		break;
 	default:
 		BUG();
 	}
@@ -1517,6 +1575,9 @@ static int find_free_dev_extent_start(struct btrfs_device *device,
 
 	search_start = dev_extent_search_start(device, search_start);
 
+	WARN_ON(device->zone_info &&
+		!IS_ALIGNED(num_bytes, device->zone_info->zone_size));
+
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;

fs/btrfs/volumes.c static void init_alloc_chunk_ctl_policy_regular( · line 4973

Quoted 6 lines
fs/btrfs/volumes.c +3 -0
@@ -4907,6 +4968,37 @@ static void init_alloc_chunk_ctl_policy_regular(
 	ctl->dev_extent_min = BTRFS_STRIPE_LEN * ctl->dev_stripes;
 }
 
+static void
+init_alloc_chunk_ctl_policy_zoned(struct btrfs_fs_devices *fs_devices,
+				  struct alloc_chunk_ctl *ctl)

static void init_alloc_chunk_ctl_policy_zoned(

Ie. type and name on one line

Quoted 42 lines
fs/btrfs/volumes.c +31 -0
+{
+	u64 zone_size = fs_devices->fs_info->zone_size;
+	u64 limit;
+	int min_num_stripes = ctl->devs_min * ctl->dev_stripes;
+	int min_data_stripes = (min_num_stripes - ctl->nparity) / ctl->ncopies;
+	u64 min_chunk_size = min_data_stripes * zone_size;
+	u64 type = ctl->type;
+
+	ctl->max_stripe_size = zone_size;
+	if (type & BTRFS_BLOCK_GROUP_DATA) {
+		ctl->max_chunk_size = round_down(BTRFS_MAX_DATA_CHUNK_SIZE,
+						 zone_size);
+	} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
+		ctl->max_chunk_size = ctl->max_stripe_size;
+	} else if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
+		ctl->max_chunk_size = 2 * ctl->max_stripe_size;
+		ctl->devs_max = min_t(int, ctl->devs_max,
+				      BTRFS_MAX_DEVS_SYS_CHUNK);
+	}
+
+	/* We don't want a chunk larger than 10% of writable space */
+	limit = max(round_down(div_factor(fs_devices->total_rw_bytes, 1),
+			       zone_size),
+		    min_chunk_size);
+	ctl->max_chunk_size = min(limit, ctl->max_chunk_size);
+	ctl->dev_extent_min = zone_size * ctl->dev_stripes;
+}
+
 static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
 				 struct alloc_chunk_ctl *ctl)
 {
@@ -4927,6 +5019,9 @@ static void init_alloc_chunk_ctl(struct btrfs_fs_devices *fs_devices,
 	case BTRFS_CHUNK_ALLOC_REGULAR:
 		init_alloc_chunk_ctl_policy_regular(fs_devices, ctl);
 		break;
+	case BTRFS_CHUNK_ALLOC_ZONED:
+		init_alloc_chunk_ctl_policy_zoned(fs_devices, ctl);
+		break;
 	default:
 		BUG();
 	}

fs/btrfs/volumes.c static int decide_stripe_size_regular(struct alloc_chunk_ctl *ctl, · line 5155

Quoted 8 lines
fs/btrfs/volumes.c +5 -0
@@ -5053,6 +5148,40 @@ static int decide_stripe_size_regular(struct alloc_chunk_ctl *ctl,
 	return 0;
 }
 
+static int decide_stripe_size_zoned(struct alloc_chunk_ctl *ctl,
+				    struct btrfs_device_info *devices_info)
+{
+	u64 zone_size = devices_info[0].dev->zone_info->zone_size;
+	/* number of stripes that count for block group size */
	/* Number ... */
Quoted 4 lines
fs/btrfs/volumes.c +4 -0
+	int data_stripes;
+
+	/*
+	 * It should hold because:

fs/btrfs/volumes.c static int decide_stripe_size_regular(struct alloc_chunk_ctl *ctl, · line 5169

Quoted 10 lines
fs/btrfs/volumes.c +10 -0
+	 *    dev_extent_min == dev_extent_want == zone_size * dev_stripes
+	 */
+	ASSERT(devices_info[ctl->ndevs - 1].max_avail == ctl->dev_extent_min);
+
+	ctl->stripe_size = zone_size;
+	ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+	data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+
+	/*
+	 * stripe_size is fixed in ZONED. Reduce ndevs instead.

/* One line comment if it fits to 80 cols */

Quoted 47 lines
fs/btrfs/volumes.c +15 -0
+	 */
+	if (ctl->stripe_size * data_stripes > ctl->max_chunk_size) {
+		ctl->ndevs = div_u64(div_u64(ctl->max_chunk_size * ctl->ncopies,
+					     ctl->stripe_size) + ctl->nparity,
+				     ctl->dev_stripes);
+		ctl->num_stripes = ctl->ndevs * ctl->dev_stripes;
+		data_stripes = (ctl->num_stripes - ctl->nparity) / ctl->ncopies;
+		ASSERT(ctl->stripe_size * data_stripes <= ctl->max_chunk_size);
+	}
+
+	ctl->chunk_size = ctl->stripe_size * data_stripes;
+
+	return 0;
+}
+
 static int decide_stripe_size(struct btrfs_fs_devices *fs_devices,
 			      struct alloc_chunk_ctl *ctl,
 			      struct btrfs_device_info *devices_info)
fs/btrfs/zoned.c +23 -0
@@ -607,3 +610,126 @@ int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror)
 				sb_zone << zone_sectors_shift, zone_sectors * 2,
 				GFP_NOFS);
 }
+
+/*
+ * btrfs_check_allocatable_zones - find allocatable zones within give region
+ * @device:	the device to allocate a region
+ * @hole_start: the position of the hole to allocate the region
+ * @num_bytes:	the size of wanted region
+ * @hole_size:	the size of hole
+ *
+ * Allocatable region should not contain any superblock locations.
+ */
+u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
+				 u64 hole_end, u64 num_bytes)
+{
+	struct btrfs_zoned_device_info *zinfo = device->zone_info;
+	u8 shift = zinfo->zone_size_shift;
+	u64 nzones = num_bytes >> shift;
+	u64 pos = hole_start;
+	u64 begin, end;
+	u64 sb_pos;
+	bool have_sb;
+	int i;
+
+	ASSERT(IS_ALIGNED(hole_start, zinfo->zone_size));

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 645

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	ASSERT(IS_ALIGNED(num_bytes, zinfo->zone_size));
+
+	while (pos < hole_end) {
+		begin = pos >> shift;
+		end = begin + nzones;
+
+		if (end > zinfo->nr_zones)
+			return hole_end;
+
+		/* check if zones in the region are all empty */
		/* Check ... */
Quoted 50 lines
fs/btrfs/zoned.c +50 -0
+		if (btrfs_dev_is_sequential(device, pos) &&
+		    find_next_zero_bit(zinfo->empty_zones, end, begin) != end) {
+			pos += zinfo->zone_size;
+			continue;
+		}
+
+		have_sb = false;
+		for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
+			sb_pos = sb_zone_number(zinfo->zone_size, i);
+			if (!(end < sb_pos || sb_pos + 1 < begin)) {
+				have_sb = true;
+				pos = (sb_pos + 2) << shift;
+				break;
+			}
+		}
+		if (!have_sb)
+			break;
+	}
+
+	return pos;
+}
+
+int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
+			    u64 length, u64 *bytes)
+{
+	int ret;
+
+	*bytes = 0;
+	ret = blkdev_zone_mgmt(device->bdev, REQ_OP_ZONE_RESET,
+			       physical >> SECTOR_SHIFT, length >> SECTOR_SHIFT,
+			       GFP_NOFS);
+	if (ret)
+		return ret;
+
+	*bytes = length;
+	while (length) {
+		btrfs_dev_set_zone_empty(device, physical);
+		physical += device->zone_info->zone_size;
+		length -= device->zone_info->zone_size;
+	}
+
+	return 0;
+}
+
+int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
+{
+	struct btrfs_zoned_device_info *zinfo = device->zone_info;
+	u8 shift = zinfo->zone_size_shift;
+	unsigned long begin = start >> shift;
+	unsigned long end = (start + size) >> shift;

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 705

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	u64 pos;
+	int ret;
+
+	ASSERT(IS_ALIGNED(start, zinfo->zone_size));
+	ASSERT(IS_ALIGNED(size, zinfo->zone_size));
+
+	if (end > zinfo->nr_zones)
+		return -ERANGE;
+
+	/* all the zones are conventional */
	/* All ... */

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 709

Quoted 4 lines
fs/btrfs/zoned.c +4 -0
+	if (find_next_bit(zinfo->seq_zones, begin, end) == end)
+		return 0;
+
+	/* all the zones are sequential and empty */
	/* All ... */

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 721

Quoted 12 lines
fs/btrfs/zoned.c +12 -0
+	if (find_next_zero_bit(zinfo->seq_zones, begin, end) == end &&
+	    find_next_zero_bit(zinfo->empty_zones, begin, end) == end)
+		return 0;
+
+	for (pos = start; pos < start + size; pos += zinfo->zone_size) {
+		u64 reset_bytes;
+
+		if (!btrfs_dev_is_sequential(device, pos) ||
+		    btrfs_dev_is_empty_zone(device, pos))
+			continue;
+
+		/* free regions should be empty */
		/* Free ... */

fs/btrfs/zoned.c int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror) · line 724

Quoted 3 lines
fs/btrfs/zoned.c +3 -0
+		btrfs_warn_in_rcu(
+			device->fs_info,
+			"resetting device %s zone %llu for allocation",

"zoned: resetting device %s (devid %llu) zone %llu for allocation

Quoted 28 lines
fs/btrfs/zoned.c +11 -0
+			rcu_str_deref(device->name), pos >> shift);
+		WARN_ON_ONCE(1);
+
+		ret = btrfs_reset_device_zone(device, pos, zinfo->zone_size,
+					      &reset_bytes);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
fs/btrfs/zoned.h +5 -0
index 447c4e5ffcbb..24dd0c9561f9 100644
@@ -34,6 +34,11 @@ int btrfs_sb_log_location(struct btrfs_device *device, int mirror, int rw,
 			  u64 *bytenr_ret);
 void btrfs_advance_sb_log(struct btrfs_device *device, int mirror);
 int btrfs_reset_sb_log_zones(struct block_device *bdev, int mirror);
+u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
+				 u64 hole_end, u64 num_bytes);
+int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
+			    u64 length, u64 *bytes);
+int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)

fs/btrfs/zoned.h static inline int btrfs_reset_sb_log_zones(struct block_device *bdev, · line 84

Quoted 3 lines
fs/btrfs/zoned.h +0 -0
@@ -77,6 +82,23 @@ static inline int btrfs_reset_sb_log_zones(struct block_device *bdev,
 {
 	return 0;
 }

newline

fs/btrfs/zoned.h static inline int btrfs_reset_sb_log_zones(struct block_device *bdev, · line 90

Quoted 6 lines
fs/btrfs/zoned.h +6 -0
+static inline u64 btrfs_find_allocatable_zones(struct btrfs_device *device,
+					       u64 hole_start, u64 hole_end,
+					       u64 num_bytes)
+{
+	return hole_start;
+}

newline

fs/btrfs/zoned.h static inline int btrfs_reset_sb_log_zones(struct block_device *bdev, · line 96

Quoted 6 lines
fs/btrfs/zoned.h +6 -0
+static inline int btrfs_reset_device_zone(struct btrfs_device *device,
+					  u64 physical, u64 length, u64 *bytes)
+{
+	*bytes = 0;
+	return 0;
+}

newline

fs/btrfs/zoned.h static inline int btrfs_reset_sb_log_zones(struct block_device *bdev, · line 101

Quoted 5 lines
fs/btrfs/zoned.h +5 -0
+static inline int btrfs_ensure_empty_zones(struct btrfs_device *device,
+					   u64 start, u64 size)
+{
+	return 0;
+}

newline

fs/btrfs/zoned.h static inline bool btrfs_check_super_location(struct btrfs_device *device, · line 180

Quoted 8 lines
fs/btrfs/zoned.h +1 -0
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
@@ -155,4 +177,12 @@ static inline bool btrfs_check_super_location(struct btrfs_device *device,
 	       !btrfs_dev_is_sequential(device, pos);
 }
 
+static inline u64 btrfs_zone_align(struct btrfs_device *device, u64 pos)

I can't tell from the name what it does, something like btrfs_align_offset_to_zone would be more clear.

Quoted 10 lines
fs/btrfs/zoned.h +7 -0
+{
+	if (!device->zone_info)
+		return pos;
+
+	return ALIGN(pos, device->zone_info->zone_size);
+}
+
 #endif
Signature · 1 line
2.27.0
#45 lore raw

PATCH v9 13/41 btrfs: verify device extent is aligned to zone

In reply to Naohiro Aota

Add a check in verify_one_dev_extent() to check if a device extent on a zoned block device is aligned to the respective zone boundary.

---
 fs/btrfs/volumes.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
fs/btrfs/volumes.c +14 -0
index 78c62ef02e6f..9e69222086ae 100644
@@ -7778,6 +7778,20 @@ static int verify_one_dev_extent(struct btrfs_fs_info *fs_info,
 		ret = -EUCLEAN;
 		goto out;
 	}
+
+	if (dev->zone_info) {
+		u64 zone_size = dev->zone_info->zone_size;
+
+		if (!IS_ALIGNED(physical_offset, zone_size) ||
+		    !IS_ALIGNED(physical_len, zone_size)) {
+			btrfs_err(fs_info,
+"dev extent devid %llu physical offset %llu len %llu is not aligned to device zone",
+				  devid, physical_offset, physical_len);
+			ret = -EUCLEAN;
+			goto out;
+		}
+	}
+
 out:
 	free_extent_map(em);
 	return ret;
Signature · 1 line
2.27.0
#47 lore raw

PATCH v9 14/41 btrfs: load zone's alloction offset

In reply to Naohiro Aota

Zoned btrfs must allocate blocks at the zones' write pointer. The device's write pointer position can be mapped to a logical address within a block group. This commit adds "alloc_offset" to track the logical address.

This logical address is populated in btrfs_load_block-group_zone_info() from write pointers of corresponding zones.

For now, zoned btrfs only support the SINGLE profile. Supporting non-SINGLE profile with zone append writing is not trivial. For example, in the DUP profile, we send a zone append writing IO to two zones on a device. The device reply with written LBAs for the IOs. If the offsets of the returned addresses from the beginning of the zone are different, then it results in different logical addresses.

We need fine-grained logical to physical mapping to support such separated physical address issue. Since it should require additional metadata type, disable non-SINGLE profiles for now.

---
 fs/btrfs/block-group.c |  15 ++++
 fs/btrfs/block-group.h |   6 ++
 fs/btrfs/zoned.c       | 153 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |   6 ++
 4 files changed, 180 insertions(+)
fs/btrfs/block-group.c +15 -0
index e989c66aa764..920b2708c7f2 100644
@@ -15,6 +15,7 @@
 #include "delalloc-space.h"
 #include "discard.h"
 #include "raid56.h"
+#include "zoned.h"
 
 /*
  * Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1935,6 +1936,13 @@ static int read_one_block_group(struct btrfs_fs_info *info,
 			goto error;
 	}
 
+	ret = btrfs_load_block_group_zone_info(cache);
+	if (ret) {
+		btrfs_err(info, "failed to load zone info of bg %llu",
+			  cache->start);
+		goto error;
+	}
+
 	/*
 	 * We need to exclude the super stripes now so that the space info has
 	 * super bytes accounted for, otherwise we'll think we have more space
@@ -2161,6 +2169,13 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans, u64 bytes_used,
 	cache->last_byte_to_unpin = (u64)-1;
 	cache->cached = BTRFS_CACHE_FINISHED;
 	cache->needs_free_space = 1;
+
+	ret = btrfs_load_block_group_zone_info(cache);
+	if (ret) {
+		btrfs_put_block_group(cache);
+		return ret;
+	}
+
 	ret = exclude_super_stripes(cache);
 	if (ret) {
 		/* We may have excluded something, so call this just in case */
fs/btrfs/block-group.h +6 -0
index adfd7583a17b..14e3043c9ce7 100644
@@ -183,6 +183,12 @@ struct btrfs_block_group {
 
 	/* Record locked full stripes for RAID5/6 block group */
 	struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
+
+	/*
+	 * Allocation offset for the block group to implement sequential
+	 * allocation. This is used only with ZONED mode enabled.
+	 */
+	u64 alloc_offset;
 };
 
 static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/zoned.c +153 -0

170 lines not on this page · raw message

fs/btrfs/zoned.h +6 -0
index 24dd0c9561f9..90ed43a25595 100644
@@ -39,6 +39,7 @@ u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
 int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
 			    u64 length, u64 *bytes);
 int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -99,6 +100,11 @@ static inline int btrfs_ensure_empty_zones(struct btrfs_device *device,
 {
 	return 0;
 }
+static inline int btrfs_load_block_group_zone_info(
+	struct btrfs_block_group *cache)
+{
+	return 0;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
#48 lore raw

Re: [PATCH v9 14/41] btrfs: load zone's alloction offset

Quoted 5 lines · Naohiro Aota

Zoned btrfs must allocate blocks at the zones' write pointer. The device's write pointer position can be mapped to a logical address within a block group. This commit adds "alloc_offset" to track the logical address.

This logical address is populated in btrfs_load_block-group_zone_info()

btrfs_load_block_group_zone_info()

Quoted 219 lines

from write pointers of corresponding zones.

For now, zoned btrfs only support the SINGLE profile. Supporting non-SINGLE profile with zone append writing is not trivial. For example, in the DUP profile, we send a zone append writing IO to two zones on a device. The device reply with written LBAs for the IOs. If the offsets of the returned addresses from the beginning of the zone are different, then it results in different logical addresses.

We need fine-grained logical to physical mapping to support such separated physical address issue. Since it should require additional metadata type, disable non-SINGLE profiles for now.

---
  fs/btrfs/block-group.c |  15 ++++
  fs/btrfs/block-group.h |   6 ++
  fs/btrfs/zoned.c       | 153 +++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/zoned.h       |   6 ++
  4 files changed, 180 insertions(+)
fs/btrfs/block-group.c +15 -0
index e989c66aa764..920b2708c7f2 100644
@@ -15,6 +15,7 @@
  #include "delalloc-space.h"
  #include "discard.h"
  #include "raid56.h"
+#include "zoned.h"
 
  /*
   * Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1935,6 +1936,13 @@ static int read_one_block_group(struct btrfs_fs_info *info,
  			goto error;
  	}
 
+	ret = btrfs_load_block_group_zone_info(cache);
+	if (ret) {
+		btrfs_err(info, "failed to load zone info of bg %llu",
+			  cache->start);
+		goto error;
+	}
+
  	/*
  	 * We need to exclude the super stripes now so that the space info has
  	 * super bytes accounted for, otherwise we'll think we have more space
@@ -2161,6 +2169,13 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans, u64 bytes_used,
  	cache->last_byte_to_unpin = (u64)-1;
  	cache->cached = BTRFS_CACHE_FINISHED;
  	cache->needs_free_space = 1;
+
+	ret = btrfs_load_block_group_zone_info(cache);
+	if (ret) {
+		btrfs_put_block_group(cache);
+		return ret;
+	}
+
  	ret = exclude_super_stripes(cache);
  	if (ret) {
  		/* We may have excluded something, so call this just in case */
fs/btrfs/block-group.h +6 -0
index adfd7583a17b..14e3043c9ce7 100644
@@ -183,6 +183,12 @@ struct btrfs_block_group {
 
  	/* Record locked full stripes for RAID5/6 block group */
  	struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
+
+	/*
+	 * Allocation offset for the block group to implement sequential
+	 * allocation. This is used only with ZONED mode enabled.
+	 */
+	u64 alloc_offset;
  };
 
  static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/zoned.c +118 -0
index 4411d786597a..0aa821893a51 100644
@@ -3,14 +3,20 @@
  #include <linux/bitops.h>
  #include <linux/slab.h>
  #include <linux/blkdev.h>
+#include <linux/sched/mm.h>
  #include "ctree.h"
  #include "volumes.h"
  #include "zoned.h"
  #include "rcu-string.h"
  #include "disk-io.h"
+#include "block-group.h"
 
  /* Maximum number of zones to report per blkdev_report_zones() call */
  #define BTRFS_REPORT_NR_ZONES   4096
+/* Invalid allocation pointer value for missing devices */
+#define WP_MISSING_DEV ((u64)-1)
+/* Pseudo write pointer value for conventional zone */
+#define WP_CONVENTIONAL ((u64)-2)
 
  static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
  			     void *data)
@@ -733,3 +739,150 @@ int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
 
  	return 0;
  }
+
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct extent_map_tree *em_tree = &fs_info->mapping_tree;
+	struct extent_map *em;
+	struct map_lookup *map;
+	struct btrfs_device *device;
+	u64 logical = cache->start;
+	u64 length = cache->length;
+	u64 physical = 0;
+	int ret;
+	int i;
+	unsigned int nofs_flag;
+	u64 *alloc_offsets = NULL;
+	u32 num_sequential = 0, num_conventional = 0;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	/* Sanity check */
+	if (!IS_ALIGNED(length, fs_info->zone_size)) {
+		btrfs_err(fs_info, "unaligned block group at %llu + %llu",
+			  logical, length);
+		return -EIO;
+	}
+
+	/* Get the chunk mapping */
+	read_lock(&em_tree->lock);
+	em = lookup_extent_mapping(em_tree, logical, length);
+	read_unlock(&em_tree->lock);
+
+	if (!em)
+		return -EINVAL;
+
+	map = em->map_lookup;
+
+	/*
+	 * Get the zone type: if the group is mapped to a non-sequential zone,
+	 * there is no need for the allocation offset (fit allocation is OK).
+	 */
+	alloc_offsets = kcalloc(map->num_stripes, sizeof(*alloc_offsets),
+				GFP_NOFS);
+	if (!alloc_offsets) {
+		free_extent_map(em);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < map->num_stripes; i++) {
+		bool is_sequential;
+		struct blk_zone zone;
+
+		device = map->stripes[i].dev;
+		physical = map->stripes[i].physical;
+
+		if (device->bdev == NULL) {
+			alloc_offsets[i] = WP_MISSING_DEV;
+			continue;
+		}
+
+		is_sequential = btrfs_dev_is_sequential(device, physical);
+		if (is_sequential)
+			num_sequential++;
+		else
+			num_conventional++;
+
+		if (!is_sequential) {
+			alloc_offsets[i] = WP_CONVENTIONAL;
+			continue;
+		}
+
+		/*
+		 * This zone will be used for allocation, so mark this
+		 * zone non-empty.
+		 */
+		btrfs_dev_clear_zone_empty(device, physical);
+
+		/*
+		 * The group is mapped to a sequential zone. Get the zone write
+		 * pointer to determine the allocation offset within the zone.
+		 */
+		WARN_ON(!IS_ALIGNED(physical, fs_info->zone_size));
+		nofs_flag = memalloc_nofs_save();
+		ret = btrfs_get_dev_zone(device, physical, &zone);
+		memalloc_nofs_restore(nofs_flag);
+		if (ret == -EIO || ret == -EOPNOTSUPP) {
+			ret = 0;
+			alloc_offsets[i] = WP_MISSING_DEV;
+			continue;
+		} else if (ret) {
+			goto out;
+		}
+
+		switch (zone.cond) {
+		case BLK_ZONE_COND_OFFLINE:
+		case BLK_ZONE_COND_READONLY:
+			btrfs_err(fs_info, "Offline/readonly zone %llu",
+				  physical >> device->zone_info->zone_size_shift);
+			alloc_offsets[i] = WP_MISSING_DEV;
+			break;
+		case BLK_ZONE_COND_EMPTY:
+			alloc_offsets[i] = 0;
+			break;
+		case BLK_ZONE_COND_FULL:
+			alloc_offsets[i] = fs_info->zone_size;
+			break;
+		default:
+			/* Partially used zone */
+			alloc_offsets[i] =
+				((zone.wp - zone.start) << SECTOR_SHIFT);
+			break;
+		}

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 863

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	}
+
+	if (num_conventional > 0) {
+		/*
+		 * Since conventional zones does not have write pointer, we
+		 * cannot determine alloc_offset from the pointer
+		 */
+		ret = -EINVAL;
+		goto out;
+	}

Does this mean we can't have zoned with a device that has conventional and sequential zones? I thought such things existed currently? Thanks,

Josef

Reviewed-by
#49 lore raw

Re: [PATCH v9 14/41] btrfs: load zone's alloction offset

Quoted 228 lines · Josef Bacik
Quoted 5 lines · Naohiro Aota

Zoned btrfs must allocate blocks at the zones' write pointer. The device's write pointer position can be mapped to a logical address within a block group. This commit adds "alloc_offset" to track the logical address.

This logical address is populated in btrfs_load_block-group_zone_info()

btrfs_load_block_group_zone_info()

Quoted 219 lines

from write pointers of corresponding zones.

For now, zoned btrfs only support the SINGLE profile. Supporting non-SINGLE profile with zone append writing is not trivial. For example, in the DUP profile, we send a zone append writing IO to two zones on a device. The device reply with written LBAs for the IOs. If the offsets of the returned addresses from the beginning of the zone are different, then it results in different logical addresses.

We need fine-grained logical to physical mapping to support such separated physical address issue. Since it should require additional metadata type, disable non-SINGLE profiles for now.

---
  fs/btrfs/block-group.c |  15 ++++
  fs/btrfs/block-group.h |   6 ++
  fs/btrfs/zoned.c       | 153 +++++++++++++++++++++++++++++++++++++++++ fs/btrfs/zoned.h       |   6 ++

  4 files changed, 180 insertions(+)
fs/btrfs/block-group.c +15 -0
index e989c66aa764..920b2708c7f2 100644
@@ -15,6 +15,7 @@
   #include "delalloc-space.h"
   #include "discard.h"
   #include "raid56.h"
+#include "zoned.h"
   /*
    * Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1935,6 +1936,13 @@ static int read_one_block_group(struct btrfs_fs_info 
 *info,
               goto error;
       }
+    ret = btrfs_load_block_group_zone_info(cache);
+    if (ret) {
+        btrfs_err(info, "failed to load zone info of bg %llu",
+              cache->start);
+        goto error;
+    }
+
       /*
        * We need to exclude the super stripes now so that the space info has
        * super bytes accounted for, otherwise we'll think we have more space
@@ -2161,6 +2169,13 @@ int btrfs_make_block_group(struct btrfs_trans_handle 
 *trans, u64 bytes_used,
       cache->last_byte_to_unpin = (u64)-1;
       cache->cached = BTRFS_CACHE_FINISHED;
       cache->needs_free_space = 1;
+
+    ret = btrfs_load_block_group_zone_info(cache);
+    if (ret) {
+        btrfs_put_block_group(cache);
+        return ret;
+    }
+
       ret = exclude_super_stripes(cache);
       if (ret) {

          /* We may have excluded something, so call this just in case */

fs/btrfs/block-group.h +6 -0
index adfd7583a17b..14e3043c9ce7 100644
@@ -183,6 +183,12 @@ struct btrfs_block_group {
       /* Record locked full stripes for RAID5/6 block group */
       struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
+
+    /*
+     * Allocation offset for the block group to implement sequential
+     * allocation. This is used only with ZONED mode enabled.
+     */
+    u64 alloc_offset;
   };
   static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/zoned.c +122 -0
index 4411d786597a..0aa821893a51 100644
@@ -3,14 +3,20 @@
   #include <linux/bitops.h>
   #include <linux/slab.h>
   #include <linux/blkdev.h>
+#include <linux/sched/mm.h>
   #include "ctree.h"
   #include "volumes.h"
   #include "zoned.h"
   #include "rcu-string.h"
   #include "disk-io.h"
+#include "block-group.h"
   /* Maximum number of zones to report per blkdev_report_zones() call */
   #define BTRFS_REPORT_NR_ZONES   4096
+/* Invalid allocation pointer value for missing devices */
+#define WP_MISSING_DEV ((u64)-1)
+/* Pseudo write pointer value for conventional zone */
+#define WP_CONVENTIONAL ((u64)-2)
   static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
                    void *data)
@@ -733,3 +739,150 @@ int btrfs_ensure_empty_zones(struct btrfs_device 
 *device, u64 start, u64 size)
       return 0;
   }
+
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
+{
+    struct btrfs_fs_info *fs_info = cache->fs_info;
+    struct extent_map_tree *em_tree = &fs_info->mapping_tree;
+    struct extent_map *em;
+    struct map_lookup *map;
+    struct btrfs_device *device;
+    u64 logical = cache->start;
+    u64 length = cache->length;
+    u64 physical = 0;
+    int ret;
+    int i;
+    unsigned int nofs_flag;
+    u64 *alloc_offsets = NULL;
+    u32 num_sequential = 0, num_conventional = 0;
+
+    if (!btrfs_is_zoned(fs_info))
+        return 0;
+
+    /* Sanity check */
+    if (!IS_ALIGNED(length, fs_info->zone_size)) {
+        btrfs_err(fs_info, "unaligned block group at %llu + %llu",
+              logical, length);
+        return -EIO;
+    }
+
+    /* Get the chunk mapping */
+    read_lock(&em_tree->lock);
+    em = lookup_extent_mapping(em_tree, logical, length);
+    read_unlock(&em_tree->lock);
+
+    if (!em)
+        return -EINVAL;
+
+    map = em->map_lookup;
+
+    /*
+     * Get the zone type: if the group is mapped to a non-sequential zone,
+     * there is no need for the allocation offset (fit allocation is OK).
+     */
+    alloc_offsets = kcalloc(map->num_stripes, sizeof(*alloc_offsets),
+                GFP_NOFS);
+    if (!alloc_offsets) {
+        free_extent_map(em);
+        return -ENOMEM;
+    }
+
+    for (i = 0; i < map->num_stripes; i++) {
+        bool is_sequential;
+        struct blk_zone zone;
+
+        device = map->stripes[i].dev;
+        physical = map->stripes[i].physical;
+
+        if (device->bdev == NULL) {
+            alloc_offsets[i] = WP_MISSING_DEV;
+            continue;
+        }
+
+        is_sequential = btrfs_dev_is_sequential(device, physical);
+        if (is_sequential)
+            num_sequential++;
+        else
+            num_conventional++;
+
+        if (!is_sequential) {
+            alloc_offsets[i] = WP_CONVENTIONAL;
+            continue;
+        }
+
+        /*
+         * This zone will be used for allocation, so mark this
+         * zone non-empty.
+         */
+        btrfs_dev_clear_zone_empty(device, physical);
+
+        /*
+         * The group is mapped to a sequential zone. Get the zone write
+         * pointer to determine the allocation offset within the zone.
+         */
+        WARN_ON(!IS_ALIGNED(physical, fs_info->zone_size));
+        nofs_flag = memalloc_nofs_save();
+        ret = btrfs_get_dev_zone(device, physical, &zone);
+        memalloc_nofs_restore(nofs_flag);
+        if (ret == -EIO || ret == -EOPNOTSUPP) {
+            ret = 0;
+            alloc_offsets[i] = WP_MISSING_DEV;
+            continue;
+        } else if (ret) {
+            goto out;
+        }
+
+        switch (zone.cond) {
+        case BLK_ZONE_COND_OFFLINE:
+        case BLK_ZONE_COND_READONLY:
+            btrfs_err(fs_info, "Offline/readonly zone %llu",
+                  physical >> device->zone_info->zone_size_shift);
+            alloc_offsets[i] = WP_MISSING_DEV;
+            break;
+        case BLK_ZONE_COND_EMPTY:
+            alloc_offsets[i] = 0;
+            break;
+        case BLK_ZONE_COND_FULL:
+            alloc_offsets[i] = fs_info->zone_size;
+            break;
+        default:
+            /* Partially used zone */
+            alloc_offsets[i] =
+                ((zone.wp - zone.start) << SECTOR_SHIFT);
+            break;
+        }
+    }
+
+    if (num_conventional > 0) {
+        /*
Quoted 8 lines

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 863

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+         * Since conventional zones does not have write pointer, we
+         * cannot determine alloc_offset from the pointer
+         */
+        ret = -EINVAL;
+        goto out;
+    }

Does this mean we can't have zoned with a device that has conventional and sequential zones?  I thought such things existed currently?  Thanks,

I see this changes in a follow up patch, ignore me, you can add

Thanks,

Josef

#50 lore raw

Re: [PATCH v9 14/41] btrfs: load zone's alloction offset

Quoted 238 lines · Josef Bacik
Quoted 228 lines · Josef Bacik
Quoted 5 lines · Naohiro Aota

Zoned btrfs must allocate blocks at the zones' write pointer. The device's write pointer position can be mapped to a logical address within a block group. This commit adds "alloc_offset" to track the logical address.

This logical address is populated in btrfs_load_block-group_zone_info()

btrfs_load_block_group_zone_info()

Quoted 219 lines

from write pointers of corresponding zones.

For now, zoned btrfs only support the SINGLE profile. Supporting non-SINGLE profile with zone append writing is not trivial. For example, in the DUP profile, we send a zone append writing IO to two zones on a device. The device reply with written LBAs for the IOs. If the offsets of the returned addresses from the beginning of the zone are different, then it results in different logical addresses.

We need fine-grained logical to physical mapping to support such separated physical address issue. Since it should require additional metadata type, disable non-SINGLE profiles for now.

---
  fs/btrfs/block-group.c |  15 ++++
  fs/btrfs/block-group.h |   6 ++
  fs/btrfs/zoned.c       | 153 +++++++++++++++++++++++++++++++++++++++++ fs/btrfs/zoned.h       |   6 ++

  4 files changed, 180 insertions(+)
fs/btrfs/block-group.c +15 -0
index e989c66aa764..920b2708c7f2 100644
@@ -15,6 +15,7 @@
   #include "delalloc-space.h"
   #include "discard.h"
   #include "raid56.h"
+#include "zoned.h"
   /*
    * Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1935,6 +1936,13 @@ static int read_one_block_group(struct 
 btrfs_fs_info *info,
               goto error;
       }
+    ret = btrfs_load_block_group_zone_info(cache);
+    if (ret) {
+        btrfs_err(info, "failed to load zone info of bg %llu",
+              cache->start);
+        goto error;
+    }
+
       /*
        * We need to exclude the super stripes now so that the space info has
        * super bytes accounted for, otherwise we'll think we have more space
@@ -2161,6 +2169,13 @@ int btrfs_make_block_group(struct 
 btrfs_trans_handle *trans, u64 bytes_used,
       cache->last_byte_to_unpin = (u64)-1;
       cache->cached = BTRFS_CACHE_FINISHED;
       cache->needs_free_space = 1;
+
+    ret = btrfs_load_block_group_zone_info(cache);
+    if (ret) {
+        btrfs_put_block_group(cache);
+        return ret;
+    }
+
       ret = exclude_super_stripes(cache);
       if (ret) {

          /* We may have excluded something, so call this just in case */

fs/btrfs/block-group.h +6 -0
index adfd7583a17b..14e3043c9ce7 100644
@@ -183,6 +183,12 @@ struct btrfs_block_group {
       /* Record locked full stripes for RAID5/6 block group */
       struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
+
+    /*
+     * Allocation offset for the block group to implement sequential
+     * allocation. This is used only with ZONED mode enabled.
+     */
+    u64 alloc_offset;
   };
   static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/zoned.c +122 -0
index 4411d786597a..0aa821893a51 100644
@@ -3,14 +3,20 @@
   #include <linux/bitops.h>
   #include <linux/slab.h>
   #include <linux/blkdev.h>
+#include <linux/sched/mm.h>
   #include "ctree.h"
   #include "volumes.h"
   #include "zoned.h"
   #include "rcu-string.h"
   #include "disk-io.h"
+#include "block-group.h"
   /* Maximum number of zones to report per blkdev_report_zones() call */
   #define BTRFS_REPORT_NR_ZONES   4096
+/* Invalid allocation pointer value for missing devices */
+#define WP_MISSING_DEV ((u64)-1)
+/* Pseudo write pointer value for conventional zone */
+#define WP_CONVENTIONAL ((u64)-2)
   static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
                    void *data)
@@ -733,3 +739,150 @@ int btrfs_ensure_empty_zones(struct 
 btrfs_device *device, u64 start, u64 size)
       return 0;
   }
+
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
+{
+    struct btrfs_fs_info *fs_info = cache->fs_info;
+    struct extent_map_tree *em_tree = &fs_info->mapping_tree;
+    struct extent_map *em;
+    struct map_lookup *map;
+    struct btrfs_device *device;
+    u64 logical = cache->start;
+    u64 length = cache->length;
+    u64 physical = 0;
+    int ret;
+    int i;
+    unsigned int nofs_flag;
+    u64 *alloc_offsets = NULL;
+    u32 num_sequential = 0, num_conventional = 0;
+
+    if (!btrfs_is_zoned(fs_info))
+        return 0;
+
+    /* Sanity check */
+    if (!IS_ALIGNED(length, fs_info->zone_size)) {
+        btrfs_err(fs_info, "unaligned block group at %llu + %llu",
+              logical, length);
+        return -EIO;
+    }
+
+    /* Get the chunk mapping */
+    read_lock(&em_tree->lock);
+    em = lookup_extent_mapping(em_tree, logical, length);
+    read_unlock(&em_tree->lock);
+
+    if (!em)
+        return -EINVAL;
+
+    map = em->map_lookup;
+
+    /*
+     * Get the zone type: if the group is mapped to a non-sequential zone,
+     * there is no need for the allocation offset (fit allocation is OK).
+     */
+    alloc_offsets = kcalloc(map->num_stripes, sizeof(*alloc_offsets),
+                GFP_NOFS);
+    if (!alloc_offsets) {
+        free_extent_map(em);
+        return -ENOMEM;
+    }
+
+    for (i = 0; i < map->num_stripes; i++) {
+        bool is_sequential;
+        struct blk_zone zone;
+
+        device = map->stripes[i].dev;
+        physical = map->stripes[i].physical;
+
+        if (device->bdev == NULL) {
+            alloc_offsets[i] = WP_MISSING_DEV;
+            continue;
+        }
+
+        is_sequential = btrfs_dev_is_sequential(device, physical);
+        if (is_sequential)
+            num_sequential++;
+        else
+            num_conventional++;
+
+        if (!is_sequential) {
+            alloc_offsets[i] = WP_CONVENTIONAL;
+            continue;
+        }
+
+        /*
+         * This zone will be used for allocation, so mark this
+         * zone non-empty.
+         */
+        btrfs_dev_clear_zone_empty(device, physical);
+
+        /*
+         * The group is mapped to a sequential zone. Get the zone write
+         * pointer to determine the allocation offset within the zone.
+         */
+        WARN_ON(!IS_ALIGNED(physical, fs_info->zone_size));
+        nofs_flag = memalloc_nofs_save();
+        ret = btrfs_get_dev_zone(device, physical, &zone);
+        memalloc_nofs_restore(nofs_flag);
+        if (ret == -EIO || ret == -EOPNOTSUPP) {
+            ret = 0;
+            alloc_offsets[i] = WP_MISSING_DEV;
+            continue;
+        } else if (ret) {
+            goto out;
+        }
+
+        switch (zone.cond) {
+        case BLK_ZONE_COND_OFFLINE:
+        case BLK_ZONE_COND_READONLY:
+            btrfs_err(fs_info, "Offline/readonly zone %llu",
+                  physical >> device->zone_info->zone_size_shift);
+            alloc_offsets[i] = WP_MISSING_DEV;
+            break;
+        case BLK_ZONE_COND_EMPTY:
+            alloc_offsets[i] = 0;
+            break;
+        case BLK_ZONE_COND_FULL:
+            alloc_offsets[i] = fs_info->zone_size;
+            break;
+        default:
+            /* Partially used zone */
+            alloc_offsets[i] =
+                ((zone.wp - zone.start) << SECTOR_SHIFT);
+            break;
+        }
+    }
+
+    if (num_conventional > 0) {
+        /*
Quoted 8 lines

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 863

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+         * Since conventional zones does not have write pointer, we
+         * cannot determine alloc_offset from the pointer
+         */
+        ret = -EINVAL;
+        goto out;
+    }

Does this mean we can't have zoned with a device that has conventional and sequential zones?  I thought such things existed currently?  Thanks,

Quoted 3 lines

I see this changes in a follow up patch, ignore me, you can add

Thanks. I have added a comment mentioning that the next patch will handle the case, just in case.

Quoted 4 lines

Thanks,

Josef

#51 lore raw

Re: [PATCH v9 14/41] btrfs: load zone's alloction offset

In reply to Josef Bacik
Quoted 9 lines · Josef Bacik
Quoted 5 lines · Naohiro Aota

Zoned btrfs must allocate blocks at the zones' write pointer. The device's write pointer position can be mapped to a logical address within a block group. This commit adds "alloc_offset" to track the logical address.

This logical address is populated in btrfs_load_block-group_zone_info()

btrfs_load_block_group_zone_info()

Fixed. Thanks.

Quoted 226 lines
Quoted 212 lines

from write pointers of corresponding zones.

For now, zoned btrfs only support the SINGLE profile. Supporting non-SINGLE profile with zone append writing is not trivial. For example, in the DUP profile, we send a zone append writing IO to two zones on a device. The device reply with written LBAs for the IOs. If the offsets of the returned addresses from the beginning of the zone are different, then it results in different logical addresses.

We need fine-grained logical to physical mapping to support such separated physical address issue. Since it should require additional metadata type, disable non-SINGLE profiles for now.

---
 fs/btrfs/block-group.c |  15 ++++
 fs/btrfs/block-group.h |   6 ++
 fs/btrfs/zoned.c       | 153 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |   6 ++
 4 files changed, 180 insertions(+)
fs/btrfs/block-group.c +15 -0
index e989c66aa764..920b2708c7f2 100644
@@ -15,6 +15,7 @@
 #include "delalloc-space.h"
 #include "discard.h"
 #include "raid56.h"
+#include "zoned.h"
 /*
  * Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1935,6 +1936,13 @@ static int read_one_block_group(struct btrfs_fs_info *info,
 			goto error;
 	}
+	ret = btrfs_load_block_group_zone_info(cache);
+	if (ret) {
+		btrfs_err(info, "failed to load zone info of bg %llu",
+			  cache->start);
+		goto error;
+	}
+
 	/*
 	 * We need to exclude the super stripes now so that the space info has
 	 * super bytes accounted for, otherwise we'll think we have more space
@@ -2161,6 +2169,13 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans, u64 bytes_used,
 	cache->last_byte_to_unpin = (u64)-1;
 	cache->cached = BTRFS_CACHE_FINISHED;
 	cache->needs_free_space = 1;
+
+	ret = btrfs_load_block_group_zone_info(cache);
+	if (ret) {
+		btrfs_put_block_group(cache);
+		return ret;
+	}
+
 	ret = exclude_super_stripes(cache);
 	if (ret) {
 		/* We may have excluded something, so call this just in case */
fs/btrfs/block-group.h +6 -0
index adfd7583a17b..14e3043c9ce7 100644
@@ -183,6 +183,12 @@ struct btrfs_block_group {
 	/* Record locked full stripes for RAID5/6 block group */
 	struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
+
+	/*
+	 * Allocation offset for the block group to implement sequential
+	 * allocation. This is used only with ZONED mode enabled.
+	 */
+	u64 alloc_offset;
 };
 static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/zoned.c +118 -0
index 4411d786597a..0aa821893a51 100644
@@ -3,14 +3,20 @@
 #include <linux/bitops.h>
 #include <linux/slab.h>
 #include <linux/blkdev.h>
+#include <linux/sched/mm.h>
 #include "ctree.h"
 #include "volumes.h"
 #include "zoned.h"
 #include "rcu-string.h"
 #include "disk-io.h"
+#include "block-group.h"
 /* Maximum number of zones to report per blkdev_report_zones() call */
 #define BTRFS_REPORT_NR_ZONES   4096
+/* Invalid allocation pointer value for missing devices */
+#define WP_MISSING_DEV ((u64)-1)
+/* Pseudo write pointer value for conventional zone */
+#define WP_CONVENTIONAL ((u64)-2)
 static int copy_zone_info_cb(struct blk_zone *zone, unsigned int idx,
 			     void *data)
@@ -733,3 +739,150 @@ int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
 	return 0;
 }
+
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct extent_map_tree *em_tree = &fs_info->mapping_tree;
+	struct extent_map *em;
+	struct map_lookup *map;
+	struct btrfs_device *device;
+	u64 logical = cache->start;
+	u64 length = cache->length;
+	u64 physical = 0;
+	int ret;
+	int i;
+	unsigned int nofs_flag;
+	u64 *alloc_offsets = NULL;
+	u32 num_sequential = 0, num_conventional = 0;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	/* Sanity check */
+	if (!IS_ALIGNED(length, fs_info->zone_size)) {
+		btrfs_err(fs_info, "unaligned block group at %llu + %llu",
+			  logical, length);
+		return -EIO;
+	}
+
+	/* Get the chunk mapping */
+	read_lock(&em_tree->lock);
+	em = lookup_extent_mapping(em_tree, logical, length);
+	read_unlock(&em_tree->lock);
+
+	if (!em)
+		return -EINVAL;
+
+	map = em->map_lookup;
+
+	/*
+	 * Get the zone type: if the group is mapped to a non-sequential zone,
+	 * there is no need for the allocation offset (fit allocation is OK).
+	 */
+	alloc_offsets = kcalloc(map->num_stripes, sizeof(*alloc_offsets),
+				GFP_NOFS);
+	if (!alloc_offsets) {
+		free_extent_map(em);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < map->num_stripes; i++) {
+		bool is_sequential;
+		struct blk_zone zone;
+
+		device = map->stripes[i].dev;
+		physical = map->stripes[i].physical;
+
+		if (device->bdev == NULL) {
+			alloc_offsets[i] = WP_MISSING_DEV;
+			continue;
+		}
+
+		is_sequential = btrfs_dev_is_sequential(device, physical);
+		if (is_sequential)
+			num_sequential++;
+		else
+			num_conventional++;
+
+		if (!is_sequential) {
+			alloc_offsets[i] = WP_CONVENTIONAL;
+			continue;
+		}
+
+		/*
+		 * This zone will be used for allocation, so mark this
+		 * zone non-empty.
+		 */
+		btrfs_dev_clear_zone_empty(device, physical);
+
+		/*
+		 * The group is mapped to a sequential zone. Get the zone write
+		 * pointer to determine the allocation offset within the zone.
+		 */
+		WARN_ON(!IS_ALIGNED(physical, fs_info->zone_size));
+		nofs_flag = memalloc_nofs_save();
+		ret = btrfs_get_dev_zone(device, physical, &zone);
+		memalloc_nofs_restore(nofs_flag);
+		if (ret == -EIO || ret == -EOPNOTSUPP) {
+			ret = 0;
+			alloc_offsets[i] = WP_MISSING_DEV;
+			continue;
+		} else if (ret) {
+			goto out;
+		}
+
+		switch (zone.cond) {
+		case BLK_ZONE_COND_OFFLINE:
+		case BLK_ZONE_COND_READONLY:
+			btrfs_err(fs_info, "Offline/readonly zone %llu",
+				  physical >> device->zone_info->zone_size_shift);
+			alloc_offsets[i] = WP_MISSING_DEV;
+			break;
+		case BLK_ZONE_COND_EMPTY:
+			alloc_offsets[i] = 0;
+			break;
+		case BLK_ZONE_COND_FULL:
+			alloc_offsets[i] = fs_info->zone_size;
+			break;
+		default:
+			/* Partially used zone */
+			alloc_offsets[i] =
+				((zone.wp - zone.start) << SECTOR_SHIFT);
+			break;
+		}

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 862

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	}
+
+	if (num_conventional > 0) {
+		/*
+		 * Since conventional zones does not have write pointer, we
+		 * cannot determine alloc_offset from the pointer
+		 */
+		ret = -EINVAL;
+		goto out;
+	}

Does this mean we can't have zoned with a device that has conventional and sequential zones? I thought such things existed currently? Thanks,

Josef

#52 lore raw

Re: [PATCH v9 14/41] btrfs: load zone's alloction offset

In reply to Naohiro Aota
Quoted 17 lines · Naohiro Aota
fs/btrfs/zoned.c +13 -0
@@ -733,3 +739,150 @@ int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
 
 	return 0;
 }
+
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct extent_map_tree *em_tree = &fs_info->mapping_tree;
+	struct extent_map *em;
+	struct map_lookup *map;
+	struct btrfs_device *device;
+	u64 logical = cache->start;
+	u64 length = cache->length;
+	u64 physical = 0;
+	int ret;
+	int i;

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 764

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	unsigned int nofs_flag;
+	u64 *alloc_offsets = NULL;
+	u32 num_sequential = 0, num_conventional = 0;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	/* Sanity check */
+	if (!IS_ALIGNED(length, fs_info->zone_size)) {
+		btrfs_err(fs_info, "unaligned block group at %llu + %llu",

"zoned: block group %llu len %llu unaligned to zone size %u"

Quoted 64 lines
fs/btrfs/zoned.c +64 -0
+			  logical, length);
+		return -EIO;
+	}
+
+	/* Get the chunk mapping */
+	read_lock(&em_tree->lock);
+	em = lookup_extent_mapping(em_tree, logical, length);
+	read_unlock(&em_tree->lock);
+
+	if (!em)
+		return -EINVAL;
+
+	map = em->map_lookup;
+
+	/*
+	 * Get the zone type: if the group is mapped to a non-sequential zone,
+	 * there is no need for the allocation offset (fit allocation is OK).
+	 */
+	alloc_offsets = kcalloc(map->num_stripes, sizeof(*alloc_offsets),
+				GFP_NOFS);
+	if (!alloc_offsets) {
+		free_extent_map(em);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < map->num_stripes; i++) {
+		bool is_sequential;
+		struct blk_zone zone;
+
+		device = map->stripes[i].dev;
+		physical = map->stripes[i].physical;
+
+		if (device->bdev == NULL) {
+			alloc_offsets[i] = WP_MISSING_DEV;
+			continue;
+		}
+
+		is_sequential = btrfs_dev_is_sequential(device, physical);
+		if (is_sequential)
+			num_sequential++;
+		else
+			num_conventional++;
+
+		if (!is_sequential) {
+			alloc_offsets[i] = WP_CONVENTIONAL;
+			continue;
+		}
+
+		/*
+		 * This zone will be used for allocation, so mark this
+		 * zone non-empty.
+		 */
+		btrfs_dev_clear_zone_empty(device, physical);
+
+		/*
+		 * The group is mapped to a sequential zone. Get the zone write
+		 * pointer to determine the allocation offset within the zone.
+		 */
+		WARN_ON(!IS_ALIGNED(physical, fs_info->zone_size));
+		nofs_flag = memalloc_nofs_save();
+		ret = btrfs_get_dev_zone(device, physical, &zone);
+		memalloc_nofs_restore(nofs_flag);
+		if (ret == -EIO || ret == -EOPNOTSUPP) {
+			ret = 0;

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 838

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+			alloc_offsets[i] = WP_MISSING_DEV;
+			continue;
+		} else if (ret) {
+			goto out;
+		}
+
+		switch (zone.cond) {
+		case BLK_ZONE_COND_OFFLINE:
+		case BLK_ZONE_COND_READONLY:
+			btrfs_err(fs_info, "Offline/readonly zone %llu",

"zoned: offline/readonly zone %llu on device %s (devid %llu)"

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+				  physical >> device->zone_info->zone_size_shift);
+			alloc_offsets[i] = WP_MISSING_DEV;
+			break;
+		case BLK_ZONE_COND_EMPTY:
+			alloc_offsets[i] = 0;
+			break;
+		case BLK_ZONE_COND_FULL:
+			alloc_offsets[i] = fs_info->zone_size;
+			break;
+		default:

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 858

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+			/* Partially used zone */
+			alloc_offsets[i] =
+				((zone.wp - zone.start) << SECTOR_SHIFT);
+			break;
+		}
+	}
+
+	if (num_conventional > 0) {
+		/*
+		 * Since conventional zones does not have write pointer, we
				  ... zones do not ...
Quoted 9 lines
fs/btrfs/zoned.c +9 -0
+		 * cannot determine alloc_offset from the pointer
+		 */
+		ret = -EINVAL;
+		goto out;
+	}
+
+	switch (map->type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {
+	case 0: /* single */
+		cache->alloc_offset = alloc_offsets[0];

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 877

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		break;
+	case BTRFS_BLOCK_GROUP_DUP:
+	case BTRFS_BLOCK_GROUP_RAID1:
+	case BTRFS_BLOCK_GROUP_RAID0:
+	case BTRFS_BLOCK_GROUP_RAID10:
+	case BTRFS_BLOCK_GROUP_RAID5:
+	case BTRFS_BLOCK_GROUP_RAID6:
+		/* non-SINGLE profiles are not supported yet */
+	default:
+		btrfs_err(fs_info, "Unsupported profile on ZONED %s",

"zoned: profile %s not supported"

Quoted 24 lines
fs/btrfs/zoned.c +11 -0
+			  btrfs_bg_type_to_raid_name(map->type));
+		ret = -EINVAL;
+		goto out;
+	}
+
+out:
+	kfree(alloc_offsets);
+	free_extent_map(em);
+
+	return ret;
+}
fs/btrfs/zoned.h +1 -0
index 24dd0c9561f9..90ed43a25595 100644
@@ -39,6 +39,7 @@ u64 btrfs_find_allocatable_zones(struct btrfs_device *device, u64 hole_start,
 int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
 			    u64 length, u64 *bytes);
 int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
+int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)

fs/btrfs/zoned.h static inline int btrfs_ensure_empty_zones(struct btrfs_device *device, · line 102

Quoted 3 lines
fs/btrfs/zoned.h +0 -0
@@ -99,6 +100,11 @@ static inline int btrfs_ensure_empty_zones(struct btrfs_device *device,
 {
 	return 0;
 }

newline

fs/btrfs/zoned.h static inline int btrfs_ensure_empty_zones(struct btrfs_device *device, · line 107

Quoted 5 lines
fs/btrfs/zoned.h +5 -0
+static inline int btrfs_load_block_group_zone_info(
+	struct btrfs_block_group *cache)
+{
+	return 0;
+}

newline

Quoted 5 lines
fs/btrfs/zoned.h +0 -0
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
#53 lore raw

PATCH v9 15/41 btrfs: emulate write pointer for conventional zones

In reply to Naohiro Aota

Conventional zones do not have a write pointer, so we cannot use it to determine the allocation offset if a block group contains a conventional zone.

But instead, we can consider the end of the last allocated extent in the block group as an allocation offset.

---
 fs/btrfs/zoned.c | 119 ++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 113 insertions(+), 6 deletions(-)
fs/btrfs/zoned.c +113 -6
index 0aa821893a51..8f58d0853cc3 100644
@@ -740,6 +740,104 @@ int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
 	return 0;
 }
 
+static int emulate_write_pointer(struct btrfs_block_group *cache,
+				 u64 *offset_ret)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct btrfs_root *root = fs_info->extent_root;
+	struct btrfs_path *path;
+	struct extent_buffer *leaf;
+	struct btrfs_key search_key;
+	struct btrfs_key found_key;
+	int slot;
+	int ret;
+	u64 length;
+
+	path = btrfs_alloc_path();
+	if (!path)
+		return -ENOMEM;
+
+	search_key.objectid = cache->start + cache->length;
+	search_key.type = 0;
+	search_key.offset = 0;
+
+	ret = btrfs_search_slot(NULL, root, &search_key, path, 0, 0);
+	if (ret < 0)
+		goto out;
+	ASSERT(ret != 0);
+	slot = path->slots[0];
+	leaf = path->nodes[0];
+	ASSERT(slot != 0);
+	slot--;
+	btrfs_item_key_to_cpu(leaf, &found_key, slot);
+
+	if (found_key.objectid < cache->start) {
+		*offset_ret = 0;
+	} else if (found_key.type == BTRFS_BLOCK_GROUP_ITEM_KEY) {
+		struct btrfs_key extent_item_key;
+
+		if (found_key.objectid != cache->start) {
+			ret = -EUCLEAN;
+			goto out;
+		}
+
+		length = 0;
+
+		/* metadata may have METADATA_ITEM_KEY */
+		if (slot == 0) {
+			btrfs_set_path_blocking(path);
+			ret = btrfs_prev_leaf(root, path);
+			if (ret < 0)
+				goto out;
+			if (ret == 0) {
+				slot = btrfs_header_nritems(leaf) - 1;
+				btrfs_item_key_to_cpu(leaf, &extent_item_key,
+						      slot);
+			}
+		} else {
+			btrfs_item_key_to_cpu(leaf, &extent_item_key, slot - 1);
+			ret = 0;
+		}
+
+		if (ret == 0 &&
+		    extent_item_key.objectid == cache->start) {
+			if (extent_item_key.type == BTRFS_METADATA_ITEM_KEY)
+				length = fs_info->nodesize;
+			else if (extent_item_key.type == BTRFS_EXTENT_ITEM_KEY)
+				length = extent_item_key.offset;
+			else {
+				ret = -EUCLEAN;
+				goto out;
+			}
+		}
+
+		*offset_ret = length;
+	} else if (found_key.type == BTRFS_EXTENT_ITEM_KEY ||
+		   found_key.type == BTRFS_METADATA_ITEM_KEY) {
+
+		if (found_key.type == BTRFS_EXTENT_ITEM_KEY)
+			length = found_key.offset;
+		else
+			length = fs_info->nodesize;
+
+		if (!(found_key.objectid >= cache->start &&
+		       found_key.objectid + length <=
+		       cache->start + cache->length)) {
+			ret = -EUCLEAN;
+			goto out;
+		}
+		*offset_ret = found_key.objectid + length - cache->start;
+	} else {
+		ret = -EUCLEAN;
+		goto out;
+	}
+	ret = 0;
+
+out:
+	btrfs_free_path(path);
+	return ret;
+}
+
 int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 {
 	struct btrfs_fs_info *fs_info = cache->fs_info;
@@ -754,6 +852,7 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	int i;
 	unsigned int nofs_flag;
 	u64 *alloc_offsets = NULL;
+	u64 emulated_offset = 0;
 	u32 num_sequential = 0, num_conventional = 0;
 
 	if (!btrfs_is_zoned(fs_info))
@@ -854,12 +953,12 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	}
 
 	if (num_conventional > 0) {
-		/*
-		 * Since conventional zones does not have write pointer, we
-		 * cannot determine alloc_offset from the pointer
-		 */
-		ret = -EINVAL;
-		goto out;
+		ret = emulate_write_pointer(cache, &emulated_offset);
+		if (ret || map->num_stripes == num_conventional) {
+			if (!ret)
+				cache->alloc_offset = emulated_offset;
+			goto out;
+		}
 	}
 
 	switch (map->type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {
@@ -881,6 +980,14 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	}
 
 out:
+	/* an extent is allocated after the write pointer */
+	if (num_conventional && emulated_offset > cache->alloc_offset) {
+		btrfs_err(fs_info,
+			  "got wrong write pointer in BG %llu: %llu > %llu",
+			  logical, emulated_offset, cache->alloc_offset);
+		ret = -EIO;
+	}
+
 	kfree(alloc_offsets);
 	free_extent_map(em);
 
Signature · 1 line
2.27.0
#54 lore raw

Re: [PATCH v9 15/41] btrfs: emulate write pointer for conventional zones

Quoted 31 lines · Naohiro Aota

Conventional zones do not have a write pointer, so we cannot use it to determine the allocation offset if a block group contains a conventional zone.

But instead, we can consider the end of the last allocated extent in the block group as an allocation offset.

---
  fs/btrfs/zoned.c | 119 ++++++++++++++++++++++++++++++++++++++++++++---
  1 file changed, 113 insertions(+), 6 deletions(-)
fs/btrfs/zoned.c +11 -0
index 0aa821893a51..8f58d0853cc3 100644
@@ -740,6 +740,104 @@ int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
  	return 0;
  }
 
+static int emulate_write_pointer(struct btrfs_block_group *cache,
+				 u64 *offset_ret)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct btrfs_root *root = fs_info->extent_root;
+	struct btrfs_path *path;
+	struct extent_buffer *leaf;
+	struct btrfs_key search_key;
+	struct btrfs_key found_key;
+	int slot;
+	int ret;

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 763

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+	u64 length;
+
+	path = btrfs_alloc_path();
+	if (!path)
+		return -ENOMEM;
+
+	search_key.objectid = cache->start + cache->length;
+	search_key.type = 0;
+	search_key.offset = 0;
+

You can just use 'key', don't have to use 'search_key'.

Also you don't check for things like BTRFS_TREE_BLOCK_REF_KEY or whatever in the case that we don't have an inline extent ref, so this could error out with a fs with lots of snapshots and different references. What you need is to search back until you hit an BTRFS_METADATA_ITEM_KEY or a BTRFS_EXTENT_ITEM_KEY, and then check the offset of that thing. Otherwise this will screw up. Thanks,

Josef

#55 lore raw

Re: [PATCH v9 15/41] btrfs: emulate write pointer for conventional zones

Quoted 35 lines · Josef Bacik
Quoted 34 lines · Naohiro Aota

Conventional zones do not have a write pointer, so we cannot use it to determine the allocation offset if a block group contains a conventional zone.

But instead, we can consider the end of the last allocated extent in the block group as an allocation offset.

---
 fs/btrfs/zoned.c | 119 ++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 113 insertions(+), 6 deletions(-)
fs/btrfs/zoned.c +15 -0
index 0aa821893a51..8f58d0853cc3 100644
@@ -740,6 +740,104 @@ int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size)
 	return 0;
 }
+static int emulate_write_pointer(struct btrfs_block_group *cache,
+				 u64 *offset_ret)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct btrfs_root *root = fs_info->extent_root;
+	struct btrfs_path *path;
+	struct extent_buffer *leaf;
+	struct btrfs_key search_key;
+	struct btrfs_key found_key;
+	int slot;
+	int ret;
+	u64 length;
+
+	path = btrfs_alloc_path();
+	if (!path)
Quoted 8 lines

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 763

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+		return -ENOMEM;
+
+	search_key.objectid = cache->start + cache->length;
+	search_key.type = 0;
+	search_key.offset = 0;
+

You can just use 'key', don't have to use 'search_key'.

Fixed.

Quoted 4 lines

Also you don't check for things like BTRFS_TREE_BLOCK_REF_KEY or whatever in the case that we don't have an inline extent ref, so this could error out with a fs with lots of snapshots and different references. What you need is to search back until you hit an BTRFS_METADATA_ITEM_KEY or a BTRFS_EXTENT_ITEM_KEY, and then check the offset of that thing. Otherwise this will screw up. Thanks,

Josef

I overlooked that case. And, I just noticed we can use btrfs_previous_extent_item() here. It looks much simpler now. Thanks.

#56 lore raw

Re: [PATCH v9 15/41] btrfs: emulate write pointer for conventional zones

In reply to Naohiro Aota
Quoted 7 lines · Naohiro Aota
fs/btrfs/zoned.c +7 -0
+	struct btrfs_key found_key;
+	int slot;
+	int ret;
+	u64 length;
+
+	path = btrfs_alloc_path();
+	if (!path)

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 767

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		return -ENOMEM;
+
+	search_key.objectid = cache->start + cache->length;
+	search_key.type = 0;
+	search_key.offset = 0;
+
+	ret = btrfs_search_slot(NULL, root, &search_key, path, 0, 0);
+	if (ret < 0)
+		goto out;
+	ASSERT(ret != 0);

This should be a runtime check not an assert, if this happens it's likely a corruption

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 770

Quoted 3 lines
fs/btrfs/zoned.c +3 -0
+	slot = path->slots[0];
+	leaf = path->nodes[0];
+	ASSERT(slot != 0);

Same

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+	slot--;
+	btrfs_item_key_to_cpu(leaf, &found_key, slot);
+
+	if (found_key.objectid < cache->start) {
+		*offset_ret = 0;
+	} else if (found_key.type == BTRFS_BLOCK_GROUP_ITEM_KEY) {

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 786

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		struct btrfs_key extent_item_key;
+
+		if (found_key.objectid != cache->start) {
+			ret = -EUCLEAN;
+			goto out;
+		}
+
+		length = 0;
+
+		/* metadata may have METADATA_ITEM_KEY */
		/* Metadata ... */
Quoted 8 lines
fs/btrfs/zoned.c +8 -0
+		if (slot == 0) {
+			btrfs_set_path_blocking(path);
+			ret = btrfs_prev_leaf(root, path);
+			if (ret < 0)
+				goto out;
+			if (ret == 0) {
+				slot = btrfs_header_nritems(leaf) - 1;
+				btrfs_item_key_to_cpu(leaf, &extent_item_key,

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 804

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+						      slot);
+			}
+		} else {
+			btrfs_item_key_to_cpu(leaf, &extent_item_key, slot - 1);
+			ret = 0;
+		}
+
+		if (ret == 0 &&
+		    extent_item_key.objectid == cache->start) {
+			if (extent_item_key.type == BTRFS_METADATA_ITEM_KEY)
			if (...) {

Review

Quoted 1 line
(quoted diff) +1 -0
+				length = fs_info->nodesize;
			} else if (...) {

fs/btrfs/zoned.c int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size) · line 807

Quoted 2 lines
fs/btrfs/zoned.c +2 -0
+			else if (extent_item_key.type == BTRFS_EXTENT_ITEM_KEY)
+				length = extent_item_key.offset;
			} else {
Quoted 64 lines
fs/btrfs/zoned.c +40 -6
+			else {
+				ret = -EUCLEAN;
+				goto out;
+			}
+		}
+
+		*offset_ret = length;
+	} else if (found_key.type == BTRFS_EXTENT_ITEM_KEY ||
+		   found_key.type == BTRFS_METADATA_ITEM_KEY) {
+
+		if (found_key.type == BTRFS_EXTENT_ITEM_KEY)
+			length = found_key.offset;
+		else
+			length = fs_info->nodesize;
+
+		if (!(found_key.objectid >= cache->start &&
+		       found_key.objectid + length <=
+		       cache->start + cache->length)) {
+			ret = -EUCLEAN;
+			goto out;
+		}
+		*offset_ret = found_key.objectid + length - cache->start;
+	} else {
+		ret = -EUCLEAN;
+		goto out;
+	}
+	ret = 0;
+
+out:
+	btrfs_free_path(path);
+	return ret;
+}
+
 int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 {
 	struct btrfs_fs_info *fs_info = cache->fs_info;
@@ -754,6 +852,7 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	int i;
 	unsigned int nofs_flag;
 	u64 *alloc_offsets = NULL;
+	u64 emulated_offset = 0;
 	u32 num_sequential = 0, num_conventional = 0;
 
 	if (!btrfs_is_zoned(fs_info))
@@ -854,12 +953,12 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	}
 
 	if (num_conventional > 0) {
-		/*
-		 * Since conventional zones does not have write pointer, we
-		 * cannot determine alloc_offset from the pointer
-		 */
-		ret = -EINVAL;
-		goto out;
+		ret = emulate_write_pointer(cache, &emulated_offset);
+		if (ret || map->num_stripes == num_conventional) {
+			if (!ret)
+				cache->alloc_offset = emulated_offset;
+			goto out;
+		}
 	}
 
 	switch (map->type & BTRFS_BLOCK_GROUP_PROFILE_MASK) {

fs/btrfs/zoned.c int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache) · line 983

Quoted 4 lines
fs/btrfs/zoned.c +1 -0
@@ -881,6 +980,14 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	}
 
 out:
+	/* an extent is allocated after the write pointer */
	/* An ... */

fs/btrfs/zoned.c int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache) · line 986

Quoted 3 lines
fs/btrfs/zoned.c +3 -0
+	if (num_conventional && emulated_offset > cache->alloc_offset) {
+		btrfs_err(fs_info,
+			  "got wrong write pointer in BG %llu: %llu > %llu",

"zoned: got wrong write pointer in block group %llu found %llu emulated %llu"

Quoted 9 lines
fs/btrfs/zoned.c +4 -0
+			  logical, emulated_offset, cache->alloc_offset);
+		ret = -EIO;
+	}
+
 	kfree(alloc_offsets);
 	free_extent_map(em);
 
Signature · 1 line
2.27.0
#57 lore raw

PATCH v9 16/41 btrfs: track unusable bytes for zones

In reply to Naohiro Aota

In zoned btrfs a region that was once written then freed is not usable until we reset the underlying zone. So we need to distinguish such unusable space from usable free space.

Therefore we need to introduce the "zone_unusable" field to the block group structure, and "bytes_zone_unusable" to the space_info structure to track the unusable space.

Pinned bytes are always reclaimed to the unusable space. But, when an allocated region is returned before using e.g., the block group becomes read-only between allocation time and reservation time, we can safely return the region to the block group. For the situation, this commit introduces "btrfs_add_free_space_unused". This behaves the same as btrfs_add_free_space() on regular btrfs. On zoned btrfs, it rewinds the allocation offset.

---
 fs/btrfs/block-group.c      | 19 +++++++++-----
 fs/btrfs/block-group.h      |  1 +
 fs/btrfs/extent-tree.c      | 15 ++++++++---
 fs/btrfs/free-space-cache.c | 52 +++++++++++++++++++++++++++++++++++++
 fs/btrfs/free-space-cache.h |  4 +++
 fs/btrfs/space-info.c       | 13 ++++++----
 fs/btrfs/space-info.h       |  4 ++-
 fs/btrfs/sysfs.c            |  2 ++
 fs/btrfs/zoned.c            | 22 ++++++++++++++++
 fs/btrfs/zoned.h            |  2 ++
 10 files changed, 118 insertions(+), 16 deletions(-)
fs/btrfs/block-group.c +13 -6
index 920b2708c7f2..c34bd2dbdf82 100644
@@ -1080,12 +1080,15 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans,
 		WARN_ON(block_group->space_info->total_bytes
 			< block_group->length);
 		WARN_ON(block_group->space_info->bytes_readonly
-			< block_group->length);
+			< block_group->length - block_group->zone_unusable);
+		WARN_ON(block_group->space_info->bytes_zone_unusable
+			< block_group->zone_unusable);
 		WARN_ON(block_group->space_info->disk_total
 			< block_group->length * factor);
 	}
 	block_group->space_info->total_bytes -= block_group->length;
-	block_group->space_info->bytes_readonly -= block_group->length;
+	block_group->space_info->bytes_readonly -=
+		(block_group->length - block_group->zone_unusable);
 	block_group->space_info->disk_total -= block_group->length * factor;
 
 	spin_unlock(&block_group->space_info->lock);
@@ -1229,7 +1232,7 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)
 	}
 
 	num_bytes = cache->length - cache->reserved - cache->pinned -
-		    cache->bytes_super - cache->used;
+		    cache->bytes_super - cache->zone_unusable - cache->used;
 
 	/*
 	 * Data never overcommits, even in mixed mode, so do just the straight
@@ -1973,6 +1976,8 @@ static int read_one_block_group(struct btrfs_fs_info *info,
 		btrfs_free_excluded_extents(cache);
 	}
 
+	btrfs_calc_zone_unusable(cache);
+
 	ret = btrfs_add_block_group_cache(info, cache);
 	if (ret) {
 		btrfs_remove_free_space_cache(cache);
@@ -1980,7 +1985,8 @@ static int read_one_block_group(struct btrfs_fs_info *info,
 	}
 	trace_btrfs_add_block_group(info, cache, 0);
 	btrfs_update_space_info(info, cache->flags, cache->length,
-				cache->used, cache->bytes_super, &space_info);
+				cache->used, cache->bytes_super,
+				cache->zone_unusable, &space_info);
 
 	cache->space_info = space_info;
 
@@ -2217,7 +2223,7 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans, u64 bytes_used,
 	 */
 	trace_btrfs_add_block_group(fs_info, cache, 1);
 	btrfs_update_space_info(fs_info, cache->flags, size, bytes_used,
-				cache->bytes_super, &cache->space_info);
+				cache->bytes_super, 0, &cache->space_info);
 	btrfs_update_global_block_rsv(fs_info);
 
 	link_block_group(cache);
@@ -2325,7 +2331,8 @@ void btrfs_dec_block_group_ro(struct btrfs_block_group *cache)
 	spin_lock(&cache->lock);
 	if (!--cache->ro) {
 		num_bytes = cache->length - cache->reserved -
-			    cache->pinned - cache->bytes_super - cache->used;
+			    cache->pinned - cache->bytes_super -
+			    cache->zone_unusable - cache->used;
 		sinfo->bytes_readonly -= num_bytes;
 		list_del_init(&cache->ro_list);
 	}
fs/btrfs/block-group.h +1 -0
index 14e3043c9ce7..5be47f4bfea7 100644
@@ -189,6 +189,7 @@ struct btrfs_block_group {
 	 * allocation. This is used only with ZONED mode enabled.
 	 */
 	u64 alloc_offset;
+	u64 zone_unusable;
 };
 
 static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/extent-tree.c +11 -4
index 3b21fee13e77..fad53c702d8a 100644
@@ -34,6 +34,7 @@
 #include "block-group.h"
 #include "discard.h"
 #include "rcu-string.h"
+#include "zoned.h"
 
 #undef SCRAMBLE_DELAYED_REFS
 
@@ -2807,9 +2808,11 @@ static int unpin_extent_range(struct btrfs_fs_info *fs_info,
 			cache = btrfs_lookup_block_group(fs_info, start);
 			BUG_ON(!cache); /* Logic error */
 
-			cluster = fetch_cluster_info(fs_info,
-						     cache->space_info,
-						     &empty_cluster);
+			if (!btrfs_is_zoned(fs_info))
+				cluster = fetch_cluster_info(fs_info,
+							     cache->space_info,
+							     &empty_cluster);
+
 			empty_cluster <<= 1;
 		}
 
@@ -2846,7 +2849,11 @@ static int unpin_extent_range(struct btrfs_fs_info *fs_info,
 		space_info->max_extent_size = 0;
 		percpu_counter_add_batch(&space_info->total_bytes_pinned,
 			    -len, BTRFS_TOTAL_BYTES_PINNED_BATCH);
-		if (cache->ro) {
+		if (btrfs_is_zoned(fs_info)) {
+			/* need reset before reusing in zoned Block Group */
+			space_info->bytes_zone_unusable += len;
+			readonly = true;
+		} else if (cache->ro) {
 			space_info->bytes_readonly += len;
 			readonly = true;
 		}
fs/btrfs/free-space-cache.c +52 -0

87 lines not on this page · raw message

fs/btrfs/free-space-cache.h +4 -0
index e3d5e0ad8f8e..7081216257a8 100644
@@ -114,8 +114,12 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
 			   struct btrfs_free_space_ctl *ctl,
 			   u64 bytenr, u64 size,
 			   enum btrfs_trim_state trim_state);
+int __btrfs_add_free_space_zoned(struct btrfs_block_group *block_group,
+				 u64 bytenr, u64 size, bool used);
 int btrfs_add_free_space(struct btrfs_block_group *block_group,
 			 u64 bytenr, u64 size);
+int btrfs_add_free_space_unused(struct btrfs_block_group *block_group,
+				u64 bytenr, u64 size);
 int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
 				       u64 bytenr, u64 size);
 int btrfs_remove_free_space(struct btrfs_block_group *block_group,
fs/btrfs/space-info.c +8 -5
index 64099565ab8f..bbbf3c1412a4 100644
@@ -163,6 +163,7 @@ u64 __pure btrfs_space_info_used(struct btrfs_space_info *s_info,
 	ASSERT(s_info);
 	return s_info->bytes_used + s_info->bytes_reserved +
 		s_info->bytes_pinned + s_info->bytes_readonly +
+		s_info->bytes_zone_unusable +
 		(may_use_included ? s_info->bytes_may_use : 0);
 }
 
@@ -257,7 +258,7 @@ int btrfs_init_space_info(struct btrfs_fs_info *fs_info)
 
 void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
 			     u64 total_bytes, u64 bytes_used,
-			     u64 bytes_readonly,
+			     u64 bytes_readonly, u64 bytes_zone_unusable,
 			     struct btrfs_space_info **space_info)
 {
 	struct btrfs_space_info *found;
@@ -273,6 +274,7 @@ void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
 	found->bytes_used += bytes_used;
 	found->disk_used += bytes_used * factor;
 	found->bytes_readonly += bytes_readonly;
+	found->bytes_zone_unusable += bytes_zone_unusable;
 	if (total_bytes > 0)
 		found->full = 0;
 	btrfs_try_granting_tickets(info, found);
@@ -422,10 +424,10 @@ static void __btrfs_dump_space_info(struct btrfs_fs_info *fs_info,
 		   info->total_bytes - btrfs_space_info_used(info, true),
 		   info->full ? "" : "not ");
 	btrfs_info(fs_info,
-		"space_info total=%llu, used=%llu, pinned=%llu, reserved=%llu, may_use=%llu, readonly=%llu",
+		"space_info total=%llu, used=%llu, pinned=%llu, reserved=%llu, may_use=%llu, readonly=%llu zone_unusable=%llu",
 		info->total_bytes, info->bytes_used, info->bytes_pinned,
 		info->bytes_reserved, info->bytes_may_use,
-		info->bytes_readonly);
+		info->bytes_readonly, info->bytes_zone_unusable);
 
 	DUMP_BLOCK_RSV(fs_info, global_block_rsv);
 	DUMP_BLOCK_RSV(fs_info, trans_block_rsv);
@@ -454,9 +456,10 @@ void btrfs_dump_space_info(struct btrfs_fs_info *fs_info,
 	list_for_each_entry(cache, &info->block_groups[index], list) {
 		spin_lock(&cache->lock);
 		btrfs_info(fs_info,
-			"block group %llu has %llu bytes, %llu used %llu pinned %llu reserved %s",
+			"block group %llu has %llu bytes, %llu used %llu pinned %llu reserved %llu zone_unusable %s",
 			cache->start, cache->length, cache->used, cache->pinned,
-			cache->reserved, cache->ro ? "[readonly]" : "");
+			cache->reserved, cache->zone_unusable,
+			cache->ro ? "[readonly]" : "");
 		spin_unlock(&cache->lock);
 		btrfs_dump_free_space(cache, bytes);
 	}
fs/btrfs/space-info.h +3 -1
index 5646393b928c..ee003ffba956 100644
@@ -17,6 +17,8 @@ struct btrfs_space_info {
 	u64 bytes_may_use;	/* number of bytes that may be used for
 				   delalloc/allocations */
 	u64 bytes_readonly;	/* total bytes that are read only */
+	u64 bytes_zone_unusable;	/* total bytes that are unusable until
+					   resetting the device zone */
 
 	u64 max_extent_size;	/* This will hold the maximum extent size of
 				   the space info if we had an ENOSPC in the
@@ -119,7 +121,7 @@ DECLARE_SPACE_INFO_UPDATE(bytes_pinned, "pinned");
 int btrfs_init_space_info(struct btrfs_fs_info *fs_info);
 void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
 			     u64 total_bytes, u64 bytes_used,
-			     u64 bytes_readonly,
+			     u64 bytes_readonly, u64 bytes_zone_unusable,
 			     struct btrfs_space_info **space_info);
 struct btrfs_space_info *btrfs_find_space_info(struct btrfs_fs_info *info,
 					       u64 flags);
fs/btrfs/sysfs.c +2 -0
index 828006020bbd..ea679803da9b 100644
@@ -635,6 +635,7 @@ SPACE_INFO_ATTR(bytes_pinned);
 SPACE_INFO_ATTR(bytes_reserved);
 SPACE_INFO_ATTR(bytes_may_use);
 SPACE_INFO_ATTR(bytes_readonly);
+SPACE_INFO_ATTR(bytes_zone_unusable);
 SPACE_INFO_ATTR(disk_used);
 SPACE_INFO_ATTR(disk_total);
 BTRFS_ATTR(space_info, total_bytes_pinned,
@@ -648,6 +649,7 @@ static struct attribute *space_info_attrs[] = {
 	BTRFS_ATTR_PTR(space_info, bytes_reserved),
 	BTRFS_ATTR_PTR(space_info, bytes_may_use),
 	BTRFS_ATTR_PTR(space_info, bytes_readonly),
+	BTRFS_ATTR_PTR(space_info, bytes_zone_unusable),
 	BTRFS_ATTR_PTR(space_info, disk_used),
 	BTRFS_ATTR_PTR(space_info, disk_total),
 	BTRFS_ATTR_PTR(space_info, total_bytes_pinned),
fs/btrfs/zoned.c +22 -0
index 8f58d0853cc3..d94a2c363a47 100644
@@ -993,3 +993,25 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 
 	return ret;
 }
+
+void btrfs_calc_zone_unusable(struct btrfs_block_group *cache)
+{
+	u64 unusable, free;
+
+	if (!btrfs_is_zoned(cache->fs_info))
+		return;
+
+	WARN_ON(cache->bytes_super != 0);
+	unusable = cache->alloc_offset - cache->used;
+	free = cache->length - cache->alloc_offset;
+	/* we only need ->free_space in ALLOC_SEQ BGs */
+	cache->last_byte_to_unpin = (u64)-1;
+	cache->cached = BTRFS_CACHE_FINISHED;
+	cache->free_space_ctl->free_space = free;
+	cache->zone_unusable = unusable;
+	/*
+	 * Should not have any excluded extents. Just
+	 * in case, though.
+	 */
+	btrfs_free_excluded_extents(cache);
+}
fs/btrfs/zoned.h +2 -0
index 90ed43a25595..42b1a4217c6b 100644
@@ -40,6 +40,7 @@ int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
 			    u64 length, u64 *bytes);
 int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
 int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache);
+void btrfs_calc_zone_unusable(struct btrfs_block_group *cache);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -105,6 +106,7 @@ static inline int btrfs_load_block_group_zone_info(
 {
 	return 0;
 }
+static inline void btrfs_calc_zone_unusable(struct btrfs_block_group *cache) { }
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
#58 lore raw

Re: [PATCH v9 16/41] btrfs: track unusable bytes for zones

Quoted 124 lines · Naohiro Aota

In zoned btrfs a region that was once written then freed is not usable until we reset the underlying zone. So we need to distinguish such unusable space from usable free space.

Therefore we need to introduce the "zone_unusable" field to the block group structure, and "bytes_zone_unusable" to the space_info structure to track the unusable space.

Pinned bytes are always reclaimed to the unusable space. But, when an allocated region is returned before using e.g., the block group becomes read-only between allocation time and reservation time, we can safely return the region to the block group. For the situation, this commit introduces "btrfs_add_free_space_unused". This behaves the same as btrfs_add_free_space() on regular btrfs. On zoned btrfs, it rewinds the allocation offset.

---
  fs/btrfs/block-group.c      | 19 +++++++++-----
  fs/btrfs/block-group.h      |  1 +
  fs/btrfs/extent-tree.c      | 15 ++++++++---
  fs/btrfs/free-space-cache.c | 52 +++++++++++++++++++++++++++++++++++++
  fs/btrfs/free-space-cache.h |  4 +++
  fs/btrfs/space-info.c       | 13 ++++++----
  fs/btrfs/space-info.h       |  4 ++-
  fs/btrfs/sysfs.c            |  2 ++
  fs/btrfs/zoned.c            | 22 ++++++++++++++++
  fs/btrfs/zoned.h            |  2 ++
  10 files changed, 118 insertions(+), 16 deletions(-)
fs/btrfs/block-group.c +13 -6
index 920b2708c7f2..c34bd2dbdf82 100644
@@ -1080,12 +1080,15 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans,
  		WARN_ON(block_group->space_info->total_bytes
  			< block_group->length);
  		WARN_ON(block_group->space_info->bytes_readonly
-			< block_group->length);
+			< block_group->length - block_group->zone_unusable);
+		WARN_ON(block_group->space_info->bytes_zone_unusable
+			< block_group->zone_unusable);
  		WARN_ON(block_group->space_info->disk_total
  			< block_group->length * factor);
  	}
  	block_group->space_info->total_bytes -= block_group->length;
-	block_group->space_info->bytes_readonly -= block_group->length;
+	block_group->space_info->bytes_readonly -=
+		(block_group->length - block_group->zone_unusable);
  	block_group->space_info->disk_total -= block_group->length * factor;
 
  	spin_unlock(&block_group->space_info->lock);
@@ -1229,7 +1232,7 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)
  	}
 
  	num_bytes = cache->length - cache->reserved - cache->pinned -
-		    cache->bytes_super - cache->used;
+		    cache->bytes_super - cache->zone_unusable - cache->used;
 
  	/*
  	 * Data never overcommits, even in mixed mode, so do just the straight
@@ -1973,6 +1976,8 @@ static int read_one_block_group(struct btrfs_fs_info *info,
  		btrfs_free_excluded_extents(cache);
  	}
 
+	btrfs_calc_zone_unusable(cache);
+
  	ret = btrfs_add_block_group_cache(info, cache);
  	if (ret) {
  		btrfs_remove_free_space_cache(cache);
@@ -1980,7 +1985,8 @@ static int read_one_block_group(struct btrfs_fs_info *info,
  	}
  	trace_btrfs_add_block_group(info, cache, 0);
  	btrfs_update_space_info(info, cache->flags, cache->length,
-				cache->used, cache->bytes_super, &space_info);
+				cache->used, cache->bytes_super,
+				cache->zone_unusable, &space_info);
 
  	cache->space_info = space_info;
 
@@ -2217,7 +2223,7 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans, u64 bytes_used,
  	 */
  	trace_btrfs_add_block_group(fs_info, cache, 1);
  	btrfs_update_space_info(fs_info, cache->flags, size, bytes_used,
-				cache->bytes_super, &cache->space_info);
+				cache->bytes_super, 0, &cache->space_info);
  	btrfs_update_global_block_rsv(fs_info);
 
  	link_block_group(cache);
@@ -2325,7 +2331,8 @@ void btrfs_dec_block_group_ro(struct btrfs_block_group *cache)
  	spin_lock(&cache->lock);
  	if (!--cache->ro) {
  		num_bytes = cache->length - cache->reserved -
-			    cache->pinned - cache->bytes_super - cache->used;
+			    cache->pinned - cache->bytes_super -
+			    cache->zone_unusable - cache->used;
  		sinfo->bytes_readonly -= num_bytes;
  		list_del_init(&cache->ro_list);
  	}
fs/btrfs/block-group.h +1 -0
index 14e3043c9ce7..5be47f4bfea7 100644
@@ -189,6 +189,7 @@ struct btrfs_block_group {
  	 * allocation. This is used only with ZONED mode enabled.
  	 */
  	u64 alloc_offset;
+	u64 zone_unusable;
  };
 
  static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/extent-tree.c +1 -0
index 3b21fee13e77..fad53c702d8a 100644
@@ -34,6 +34,7 @@
  #include "block-group.h"
  #include "discard.h"
  #include "rcu-string.h"
+#include "zoned.h"
 
  #undef SCRAMBLE_DELAYED_REFS
 

fs/btrfs/extent-tree.c static int unpin_extent_range(struct btrfs_fs_info *fs_info, · line 2814

Quoted 10 lines
fs/btrfs/extent-tree.c +4 -3
@@ -2807,9 +2808,11 @@ static int unpin_extent_range(struct btrfs_fs_info *fs_info,
  			cache = btrfs_lookup_block_group(fs_info, start);
  			BUG_ON(!cache); /* Logic error */
 
-			cluster = fetch_cluster_info(fs_info,
-						     cache->space_info,
-						     &empty_cluster);
+			if (!btrfs_is_zoned(fs_info))
+				cluster = fetch_cluster_info(fs_info,
+							     cache->space_info,
+							     &empty_cluster);

Move the !btrfs_is_zoned() check into fetch_cluster_info so we don't have to indent even more.

Quoted 31 lines
fs/btrfs/extent-tree.c +6 -1
+
  			empty_cluster <<= 1;
  		}
 
@@ -2846,7 +2849,11 @@ static int unpin_extent_range(struct btrfs_fs_info *fs_info,
  		space_info->max_extent_size = 0;
  		percpu_counter_add_batch(&space_info->total_bytes_pinned,
  			    -len, BTRFS_TOTAL_BYTES_PINNED_BATCH);
-		if (cache->ro) {
+		if (btrfs_is_zoned(fs_info)) {
+			/* need reset before reusing in zoned Block Group */
+			space_info->bytes_zone_unusable += len;
+			readonly = true;
+		} else if (cache->ro) {
  			space_info->bytes_readonly += len;
  			readonly = true;
  		}
fs/btrfs/free-space-cache.c +2 -0
index af0013d3df63..cfa466319166 100644
@@ -2467,6 +2467,8 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
  	int ret = 0;
  	u64 filter_bytes = bytes;
 
+	ASSERT(!btrfs_is_zoned(fs_info));
+
  	info = kmem_cache_zalloc(btrfs_free_space_cachep, GFP_NOFS);
  	if (!info)
  		return -ENOMEM;

fs/btrfs/free-space-cache.c int __btrfs_add_free_space(struct btrfs_fs_info *fs_info, · line 2531

Quoted 6 lines
fs/btrfs/free-space-cache.c +3 -0
@@ -2524,11 +2526,44 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
  	return ret;
  }
 
+int __btrfs_add_free_space_zoned(struct btrfs_block_group *block_group,
+				 u64 bytenr, u64 size, bool used)
+{

This is only used in free-space-cache.c, it can be made static.

Quoted 199 lines
fs/btrfs/free-space-cache.c +47 -0
+	struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
+	u64 offset = bytenr - block_group->start;
+	u64 to_free, to_unusable;
+
+	spin_lock(&ctl->tree_lock);
+	if (!used)
+		to_free = size;
+	else if (offset >= block_group->alloc_offset)
+		to_free = size;
+	else if (offset + size <= block_group->alloc_offset)
+		to_free = 0;
+	else
+		to_free = offset + size - block_group->alloc_offset;
+	to_unusable = size - to_free;
+
+	ctl->free_space += to_free;
+	block_group->zone_unusable += to_unusable;
+	spin_unlock(&ctl->tree_lock);
+	if (!used) {
+		spin_lock(&block_group->lock);
+		block_group->alloc_offset -= size;
+		spin_unlock(&block_group->lock);
+	}
+	return 0;
+}
+
  int btrfs_add_free_space(struct btrfs_block_group *block_group,
  			 u64 bytenr, u64 size)
  {
  	enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
 
+	if (btrfs_is_zoned(block_group->fs_info))
+		return __btrfs_add_free_space_zoned(block_group, bytenr, size,
+						    true);
+
  	if (btrfs_test_opt(block_group->fs_info, DISCARD_SYNC))
  		trim_state = BTRFS_TRIM_STATE_TRIMMED;
 
@@ -2537,6 +2572,16 @@ int btrfs_add_free_space(struct btrfs_block_group *block_group,
  				      bytenr, size, trim_state);
  }
 
+int btrfs_add_free_space_unused(struct btrfs_block_group *block_group,
+				u64 bytenr, u64 size)
+{
+	if (btrfs_is_zoned(block_group->fs_info))
+		return __btrfs_add_free_space_zoned(block_group, bytenr, size,
+						    false);
+
+	return btrfs_add_free_space(block_group, bytenr, size);
+}
+
  /*
   * This is a subtle distinction because when adding free space back in general,
   * we want it to be added as untrimmed for async. But in the case where we add
@@ -2547,6 +2592,10 @@ int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
  {
  	enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
 
+	if (btrfs_is_zoned(block_group->fs_info))
+		return __btrfs_add_free_space_zoned(block_group, bytenr, size,
+						    true);
+
  	if (btrfs_test_opt(block_group->fs_info, DISCARD_SYNC) ||
  	    btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
  		trim_state = BTRFS_TRIM_STATE_TRIMMED;
@@ -2564,6 +2613,9 @@ int btrfs_remove_free_space(struct btrfs_block_group *block_group,
  	int ret;
  	bool re_search = false;
 
+	if (btrfs_is_zoned(block_group->fs_info))
+		return 0;
+
  	spin_lock(&ctl->tree_lock);
 
  again:
fs/btrfs/free-space-cache.h +4 -0
index e3d5e0ad8f8e..7081216257a8 100644
@@ -114,8 +114,12 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
  			   struct btrfs_free_space_ctl *ctl,
  			   u64 bytenr, u64 size,
  			   enum btrfs_trim_state trim_state);
+int __btrfs_add_free_space_zoned(struct btrfs_block_group *block_group,
+				 u64 bytenr, u64 size, bool used);
  int btrfs_add_free_space(struct btrfs_block_group *block_group,
  			 u64 bytenr, u64 size);
+int btrfs_add_free_space_unused(struct btrfs_block_group *block_group,
+				u64 bytenr, u64 size);
  int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
  				       u64 bytenr, u64 size);
  int btrfs_remove_free_space(struct btrfs_block_group *block_group,
fs/btrfs/space-info.c +8 -5
index 64099565ab8f..bbbf3c1412a4 100644
@@ -163,6 +163,7 @@ u64 __pure btrfs_space_info_used(struct btrfs_space_info *s_info,
  	ASSERT(s_info);
  	return s_info->bytes_used + s_info->bytes_reserved +
  		s_info->bytes_pinned + s_info->bytes_readonly +
+		s_info->bytes_zone_unusable +
  		(may_use_included ? s_info->bytes_may_use : 0);
  }
 
@@ -257,7 +258,7 @@ int btrfs_init_space_info(struct btrfs_fs_info *fs_info)
 
  void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
  			     u64 total_bytes, u64 bytes_used,
-			     u64 bytes_readonly,
+			     u64 bytes_readonly, u64 bytes_zone_unusable,
  			     struct btrfs_space_info **space_info)
  {
  	struct btrfs_space_info *found;
@@ -273,6 +274,7 @@ void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
  	found->bytes_used += bytes_used;
  	found->disk_used += bytes_used * factor;
  	found->bytes_readonly += bytes_readonly;
+	found->bytes_zone_unusable += bytes_zone_unusable;
  	if (total_bytes > 0)
  		found->full = 0;
  	btrfs_try_granting_tickets(info, found);
@@ -422,10 +424,10 @@ static void __btrfs_dump_space_info(struct btrfs_fs_info *fs_info,
  		   info->total_bytes - btrfs_space_info_used(info, true),
  		   info->full ? "" : "not ");
  	btrfs_info(fs_info,
-		"space_info total=%llu, used=%llu, pinned=%llu, reserved=%llu, may_use=%llu, readonly=%llu",
+		"space_info total=%llu, used=%llu, pinned=%llu, reserved=%llu, may_use=%llu, readonly=%llu zone_unusable=%llu",
  		info->total_bytes, info->bytes_used, info->bytes_pinned,
  		info->bytes_reserved, info->bytes_may_use,
-		info->bytes_readonly);
+		info->bytes_readonly, info->bytes_zone_unusable);
 
  	DUMP_BLOCK_RSV(fs_info, global_block_rsv);
  	DUMP_BLOCK_RSV(fs_info, trans_block_rsv);
@@ -454,9 +456,10 @@ void btrfs_dump_space_info(struct btrfs_fs_info *fs_info,
  	list_for_each_entry(cache, &info->block_groups[index], list) {
  		spin_lock(&cache->lock);
  		btrfs_info(fs_info,
-			"block group %llu has %llu bytes, %llu used %llu pinned %llu reserved %s",
+			"block group %llu has %llu bytes, %llu used %llu pinned %llu reserved %llu zone_unusable %s",
  			cache->start, cache->length, cache->used, cache->pinned,
-			cache->reserved, cache->ro ? "[readonly]" : "");
+			cache->reserved, cache->zone_unusable,
+			cache->ro ? "[readonly]" : "");
  		spin_unlock(&cache->lock);
  		btrfs_dump_free_space(cache, bytes);
  	}
fs/btrfs/space-info.h +3 -1
index 5646393b928c..ee003ffba956 100644
@@ -17,6 +17,8 @@ struct btrfs_space_info {
  	u64 bytes_may_use;	/* number of bytes that may be used for
  				   delalloc/allocations */
  	u64 bytes_readonly;	/* total bytes that are read only */
+	u64 bytes_zone_unusable;	/* total bytes that are unusable until
+					   resetting the device zone */
 
  	u64 max_extent_size;	/* This will hold the maximum extent size of
  				   the space info if we had an ENOSPC in the
@@ -119,7 +121,7 @@ DECLARE_SPACE_INFO_UPDATE(bytes_pinned, "pinned");
  int btrfs_init_space_info(struct btrfs_fs_info *fs_info);
  void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
  			     u64 total_bytes, u64 bytes_used,
-			     u64 bytes_readonly,
+			     u64 bytes_readonly, u64 bytes_zone_unusable,
  			     struct btrfs_space_info **space_info);
  struct btrfs_space_info *btrfs_find_space_info(struct btrfs_fs_info *info,
  					       u64 flags);
fs/btrfs/sysfs.c +2 -0
index 828006020bbd..ea679803da9b 100644
@@ -635,6 +635,7 @@ SPACE_INFO_ATTR(bytes_pinned);
  SPACE_INFO_ATTR(bytes_reserved);
  SPACE_INFO_ATTR(bytes_may_use);
  SPACE_INFO_ATTR(bytes_readonly);
+SPACE_INFO_ATTR(bytes_zone_unusable);
  SPACE_INFO_ATTR(disk_used);
  SPACE_INFO_ATTR(disk_total);
  BTRFS_ATTR(space_info, total_bytes_pinned,
@@ -648,6 +649,7 @@ static struct attribute *space_info_attrs[] = {
  	BTRFS_ATTR_PTR(space_info, bytes_reserved),
  	BTRFS_ATTR_PTR(space_info, bytes_may_use),
  	BTRFS_ATTR_PTR(space_info, bytes_readonly),
+	BTRFS_ATTR_PTR(space_info, bytes_zone_unusable),
  	BTRFS_ATTR_PTR(space_info, disk_used),
  	BTRFS_ATTR_PTR(space_info, disk_total),
  	BTRFS_ATTR_PTR(space_info, total_bytes_pinned),
fs/btrfs/zoned.c +1 -0
index 8f58d0853cc3..d94a2c363a47 100644
@@ -993,3 +993,25 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 
  	return ret;
  }
+

fs/btrfs/zoned.c int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache) · line 1006

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+void btrfs_calc_zone_unusable(struct btrfs_block_group *cache)
+{
+	u64 unusable, free;
+
+	if (!btrfs_is_zoned(cache->fs_info))
+		return;
+
+	WARN_ON(cache->bytes_super != 0);
+	unusable = cache->alloc_offset - cache->used;
+	free = cache->length - cache->alloc_offset;

newline.

fs/btrfs/zoned.c int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache) · line 1011

Quoted 5 lines
fs/btrfs/zoned.c +5 -0
+	/* we only need ->free_space in ALLOC_SEQ BGs */
+	cache->last_byte_to_unpin = (u64)-1;
+	cache->cached = BTRFS_CACHE_FINISHED;
+	cache->free_space_ctl->free_space = free;
+	cache->zone_unusable = unusable;

newline.

fs/btrfs/zoned.c int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache) · line 1017

Quoted 6 lines
fs/btrfs/zoned.c +6 -0
+	/*
+	 * Should not have any excluded extents. Just
+	 * in case, though.
+	 */
+	btrfs_free_excluded_extents(cache);
+}

Thanks,

Josef

#59 lore raw

PATCH v9 17/41 btrfs: do sequential extent allocation in ZONED mode

In reply to Naohiro Aota

This commit implements a sequential extent allocator for the ZONED mode. This allocator just needs to check if there is enough space in the block group. Therefor the allocator never manages bitmaps or clusters. Also add ASSERTs to the corresponding functions.

Actually, with zone append writing, it is unnecessary to track the allocation offset. It only needs to check space availability. But, by tracking the offset and returning the offset as an allocated region, we can skip modification of ordered extents and checksum information when there is no IO reordering.

---
 fs/btrfs/block-group.c      |  4 ++
 fs/btrfs/extent-tree.c      | 85 ++++++++++++++++++++++++++++++++++---
 fs/btrfs/free-space-cache.c |  6 +++
 3 files changed, 89 insertions(+), 6 deletions(-)
fs/btrfs/block-group.c +4 -0
index c34bd2dbdf82..d67f9cabe5c1 100644
@@ -683,6 +683,10 @@ int btrfs_cache_block_group(struct btrfs_block_group *cache, int load_cache_only
 	struct btrfs_caching_control *caching_ctl;
 	int ret = 0;
 
+	/* Allocator for ZONED btrfs do not use the cache at all */
+	if (btrfs_is_zoned(fs_info))
+		return 0;
+
 	caching_ctl = kzalloc(sizeof(*caching_ctl), GFP_NOFS);
 	if (!caching_ctl)
 		return -ENOMEM;
fs/btrfs/extent-tree.c +79 -6

147 lines not on this page · raw message

fs/btrfs/free-space-cache.c +6 -0
index cfa466319166..7ad046d33c7e 100644
@@ -2903,6 +2903,8 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
 	u64 align_gap_len = 0;
 	enum btrfs_trim_state align_gap_trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
 
+	ASSERT(!btrfs_is_zoned(block_group->fs_info));
+
 	spin_lock(&ctl->tree_lock);
 	entry = find_free_space(ctl, &offset, &bytes_search,
 				block_group->full_stripe_len, max_extent_size);
@@ -3034,6 +3036,8 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group *block_group,
 	struct rb_node *node;
 	u64 ret = 0;
 
+	ASSERT(!btrfs_is_zoned(block_group->fs_info));
+
 	spin_lock(&cluster->lock);
 	if (bytes > cluster->max_size)
 		goto out;
@@ -3810,6 +3814,8 @@ int btrfs_trim_block_group(struct btrfs_block_group *block_group,
 	int ret;
 	u64 rem = 0;
 
+	ASSERT(!btrfs_is_zoned(block_group->fs_info));
+
 	*trimmed = 0;
 
 	spin_lock(&block_group->lock);
Signature · 1 line
2.27.0
Reviewed-by on v9 17/41
#60 lore raw

Re: [PATCH v9 17/41] btrfs: do sequential extent allocation in ZONED mode

Quoted 5 lines · Naohiro Aota

This commit implements a sequential extent allocator for the ZONED mode. This allocator just needs to check if there is enough space in the block group. Therefor the allocator never manages bitmaps or clusters. Also add ASSERTs to the corresponding functions.

Quoted 7 lines

Actually, with zone append writing, it is unnecessary to track the allocation offset. It only needs to check space availability. But, by tracking the offset and returning the offset as an allocated region, we can skip modification of ordered extents and checksum information when there is no IO reordering.

Thanks,

Josef

#61 lore raw

PATCH v9 18/41 btrfs: reset zones of unused block groups

In reply to Naohiro Aota

For an ZONED volume, a block group maps to a zone of the device. For deleted unused block groups, the zone of the block group can be reset to rewind the zone write pointer at the start of the zone.

---
 fs/btrfs/block-group.c |  8 ++++++--
 fs/btrfs/extent-tree.c | 17 ++++++++++++-----
 fs/btrfs/zoned.h       | 16 ++++++++++++++++
 3 files changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/block-group.c +6 -2
index d67f9cabe5c1..82d556368c85 100644
@@ -1468,8 +1468,12 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
 		if (!async_trim_enabled && btrfs_test_opt(fs_info, DISCARD_ASYNC))
 			goto flip_async;
 
-		/* DISCARD can flip during remount */
-		trimming = btrfs_test_opt(fs_info, DISCARD_SYNC);
+		/*
+		 * DISCARD can flip during remount. In ZONED mode, we need
+		 * to reset sequential required zones.
+		 */
+		trimming = btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+				btrfs_is_zoned(fs_info);
 
 		/* Implicit trim during transaction commit. */
 		if (trimming)
fs/btrfs/extent-tree.c +12 -5
index 5e6b4d1712f2..c134746d7417 100644
@@ -1331,6 +1331,9 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 
 		stripe = bbio->stripes;
 		for (i = 0; i < bbio->num_stripes; i++, stripe++) {
+			struct btrfs_device *dev = stripe->dev;
+			u64 physical = stripe->physical;
+			u64 length = stripe->length;
 			u64 bytes;
 			struct request_queue *req_q;
 
@@ -1338,14 +1341,18 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 				ASSERT(btrfs_test_opt(fs_info, DEGRADED));
 				continue;
 			}
+
 			req_q = bdev_get_queue(stripe->dev->bdev);
-			if (!blk_queue_discard(req_q))
+			/* zone reset in ZONED mode */
+			if (btrfs_can_zone_reset(dev, physical, length))
+				ret = btrfs_reset_device_zone(dev, physical,
+							      length, &bytes);
+			else if (blk_queue_discard(req_q))
+				ret = btrfs_issue_discard(dev->bdev, physical,
+							  length, &bytes);
+			else
 				continue;
 
-			ret = btrfs_issue_discard(stripe->dev->bdev,
-						  stripe->physical,
-						  stripe->length,
-						  &bytes);
 			if (!ret) {
 				discarded_bytes += bytes;
 			} else if (ret != -EOPNOTSUPP) {
fs/btrfs/zoned.h +16 -0
index 42b1a4217c6b..d13bc6d70ea4 100644
@@ -193,4 +193,20 @@ static inline u64 btrfs_zone_align(struct btrfs_device *device, u64 pos)
 	return ALIGN(pos, device->zone_info->zone_size);
 }
 
+static inline bool btrfs_can_zone_reset(struct btrfs_device *device,
+					u64 physical, u64 length)
+{
+	u64 zone_size;
+
+	if (!btrfs_dev_is_sequential(device, physical))
+		return false;
+
+	zone_size = device->zone_info->zone_size;
+	if (!IS_ALIGNED(physical, zone_size) ||
+	    !IS_ALIGNED(length, zone_size))
+		return false;
+
+	return true;
+}
+
 #endif
Signature · 1 line
2.27.0
#62 lore raw

Re: [PATCH v9 18/41] btrfs: reset zones of unused block groups

Quoted 55 lines · Naohiro Aota

For an ZONED volume, a block group maps to a zone of the device. For deleted unused block groups, the zone of the block group can be reset to rewind the zone write pointer at the start of the zone.

---
  fs/btrfs/block-group.c |  8 ++++++--
  fs/btrfs/extent-tree.c | 17 ++++++++++++-----
  fs/btrfs/zoned.h       | 16 ++++++++++++++++
  3 files changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/block-group.c +6 -2
index d67f9cabe5c1..82d556368c85 100644
@@ -1468,8 +1468,12 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
  		if (!async_trim_enabled && btrfs_test_opt(fs_info, DISCARD_ASYNC))
  			goto flip_async;
 
-		/* DISCARD can flip during remount */
-		trimming = btrfs_test_opt(fs_info, DISCARD_SYNC);
+		/*
+		 * DISCARD can flip during remount. In ZONED mode, we need
+		 * to reset sequential required zones.
+		 */
+		trimming = btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+				btrfs_is_zoned(fs_info);
 
  		/* Implicit trim during transaction commit. */
  		if (trimming)
fs/btrfs/extent-tree.c +8 -1
index 5e6b4d1712f2..c134746d7417 100644
@@ -1331,6 +1331,9 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 
  		stripe = bbio->stripes;
  		for (i = 0; i < bbio->num_stripes; i++, stripe++) {
+			struct btrfs_device *dev = stripe->dev;
+			u64 physical = stripe->physical;
+			u64 length = stripe->length;
  			u64 bytes;
  			struct request_queue *req_q;
 
@@ -1338,14 +1341,18 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
  				ASSERT(btrfs_test_opt(fs_info, DEGRADED));
  				continue;
  			}
+
  			req_q = bdev_get_queue(stripe->dev->bdev);
-			if (!blk_queue_discard(req_q))
+			/* zone reset in ZONED mode */
+			if (btrfs_can_zone_reset(dev, physical, length))
+				ret = btrfs_reset_device_zone(dev, physical,
+							      length, &bytes);

fs/btrfs/extent-tree.c int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, · line 1348

Quoted 10 lines
fs/btrfs/extent-tree.c +4 -4
+			else if (blk_queue_discard(req_q))
+				ret = btrfs_issue_discard(dev->bdev, physical,
+							  length, &bytes);
+			else
  				continue;
 
-			ret = btrfs_issue_discard(stripe->dev->bdev,
-						  stripe->physical,
-						  stripe->length,
-						  &bytes);

The problem is you have

if (btrfs_test_opt(fs_info, DISCARD_SYNC))

	ret = btrfs_discard_extent(fs_info, start,
				   end + 1 - start, NULL);

in btrfs_finish_extent_commit, so even if you add support here, you aren't actually discarding anything because the transaction commit won't call discard for this range.

You're going to have to rework this logic to allow for discard to be called everywhere it checks DISCARD_SYNC, but then you also need to go and make sure you don't actually allow discards to happen at non-bg aligned ranges. Thanks,

Josef

#63 lore raw

Re: [PATCH v9 18/41] btrfs: reset zones of unused block groups

Quoted 64 lines · Josef Bacik
Quoted 50 lines · Naohiro Aota

For an ZONED volume, a block group maps to a zone of the device. For deleted unused block groups, the zone of the block group can be reset to rewind the zone write pointer at the start of the zone.

---
 fs/btrfs/block-group.c |  8 ++++++--
 fs/btrfs/extent-tree.c | 17 ++++++++++++-----
 fs/btrfs/zoned.h       | 16 ++++++++++++++++
 3 files changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/block-group.c +6 -2
index d67f9cabe5c1..82d556368c85 100644
@@ -1468,8 +1468,12 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
 		if (!async_trim_enabled && btrfs_test_opt(fs_info, DISCARD_ASYNC))
 			goto flip_async;
-		/* DISCARD can flip during remount */
-		trimming = btrfs_test_opt(fs_info, DISCARD_SYNC);
+		/*
+		 * DISCARD can flip during remount. In ZONED mode, we need
+		 * to reset sequential required zones.
+		 */
+		trimming = btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+				btrfs_is_zoned(fs_info);
 		/* Implicit trim during transaction commit. */
 		if (trimming)
fs/btrfs/extent-tree.c +7 -1
index 5e6b4d1712f2..c134746d7417 100644
@@ -1331,6 +1331,9 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 		stripe = bbio->stripes;
 		for (i = 0; i < bbio->num_stripes; i++, stripe++) {
+			struct btrfs_device *dev = stripe->dev;
+			u64 physical = stripe->physical;
+			u64 length = stripe->length;
 			u64 bytes;
 			struct request_queue *req_q;
@@ -1338,14 +1341,18 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 				ASSERT(btrfs_test_opt(fs_info, DEGRADED));
 				continue;
 			}
+
 			req_q = bdev_get_queue(stripe->dev->bdev);
-			if (!blk_queue_discard(req_q))
+			/* zone reset in ZONED mode */
+			if (btrfs_can_zone_reset(dev, physical, length))
+				ret = btrfs_reset_device_zone(dev, physical,

fs/btrfs/extent-tree.c int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, · line 1347

Quoted 10 lines
fs/btrfs/extent-tree.c +5 -4
+							      length, &bytes);
+			else if (blk_queue_discard(req_q))
+				ret = btrfs_issue_discard(dev->bdev, physical,
+							  length, &bytes);
+			else
 				continue;
-			ret = btrfs_issue_discard(stripe->dev->bdev,
-						  stripe->physical,
-						  stripe->length,
-						  &bytes);

The problem is you have

Quoted 5 lines

if (btrfs_test_opt(fs_info, DISCARD_SYNC))

	ret = btrfs_discard_extent(fs_info, start,
				   end + 1 - start, NULL);

in btrfs_finish_extent_commit, so even if you add support here, you aren't actually discarding anything because the transaction commit won't call discard for this range.

btrfs_discard_extent() is called for each deleted_bg on the BG area, regardless of the above pinned_extent's range.

Quoted 4 lines

You're going to have to rework this logic to allow for discard to be called everywhere it checks DISCARD_SYNC, but then you also need to go and make sure you don't actually allow discards to happen at non-bg aligned ranges. Thanks,

Josef

We already ignore btrfs_discard_extent() which is not aligned to zone. So, I confirmed it worked fine with DISCARD_SYNC.

#64 lore raw

PATCH v9 19/41 btrfs: redirty released extent buffers in ZONED mode

In reply to Naohiro Aota

Tree manipulating operations like merging nodes often release once-allocated tree nodes. Btrfs cleans such nodes so that pages in the node are not uselessly written out. On ZONED volumes, however, such optimization blocks the following IOs as the cancellation of the write out of the freed blocks breaks the sequential write sequence expected by the device.

This patch introduces a list of clean and unwritten extent buffers that have been released in a transaction. Btrfs redirty the buffer so that btree_write_cache_pages() can send proper bios to the devices.

Besides it clears the entire content of the extent buffer not to confuse raw block scanners e.g. btrfsck. By clearing the content, csum_dirty_buffer() complains about bytenr mismatch, so avoid the checking and checksum using newly introduced buffer flag EXTENT_BUFFER_NO_CHECK.

---
 fs/btrfs/disk-io.c     |  8 ++++++++
 fs/btrfs/extent-tree.c | 12 +++++++++++-
 fs/btrfs/extent_io.c   |  4 ++++
 fs/btrfs/extent_io.h   |  2 ++
 fs/btrfs/transaction.c | 10 ++++++++++
 fs/btrfs/transaction.h |  3 +++
 fs/btrfs/tree-log.c    |  6 ++++++
 fs/btrfs/zoned.c       | 37 +++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       |  6 ++++++
 9 files changed, 87 insertions(+), 1 deletion(-)
fs/btrfs/disk-io.c +8 -0
index fd8b970ee92c..9750b4e6d538 100644
@@ -462,6 +462,12 @@ static int csum_dirty_buffer(struct btrfs_fs_info *fs_info, struct page *page)
 		return 0;
 
 	found_start = btrfs_header_bytenr(eb);
+
+	if (test_bit(EXTENT_BUFFER_NO_CHECK, &eb->bflags)) {
+		WARN_ON(found_start != 0);
+		return 0;
+	}
+
 	/*
 	 * Please do not consolidate these warnings into a single if.
 	 * It is useful to know what went wrong.
@@ -4616,6 +4622,8 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,
 				     EXTENT_DIRTY);
 	btrfs_destroy_pinned_extent(fs_info, &cur_trans->pinned_extents);
 
+	btrfs_free_redirty_list(cur_trans);
+
 	cur_trans->state =TRANS_STATE_COMPLETED;
 	wake_up(&cur_trans->commit_wait);
 }
fs/btrfs/extent-tree.c +11 -1
index c134746d7417..57454ef4c91e 100644
@@ -3421,8 +3421,10 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
 
 		if (root->root_key.objectid != BTRFS_TREE_LOG_OBJECTID) {
 			ret = check_ref_cleanup(trans, buf->start);
-			if (!ret)
+			if (!ret) {
+				btrfs_redirty_list_add(trans->transaction, buf);
 				goto out;
+			}
 		}
 
 		pin = 0;
@@ -3434,6 +3436,13 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
 			goto out;
 		}
 
+		if (btrfs_is_zoned(fs_info)) {
+			btrfs_redirty_list_add(trans->transaction, buf);
+			pin_down_extent(trans, cache, buf->start, buf->len, 1);
+			btrfs_put_block_group(cache);
+			goto out;
+		}
+
 		WARN_ON(test_bit(EXTENT_BUFFER_DIRTY, &buf->bflags));
 
 		btrfs_add_free_space(cache, buf->start, buf->len);
@@ -4770,6 +4779,7 @@ btrfs_init_new_buffer(struct btrfs_trans_handle *trans, struct btrfs_root *root,
 	__btrfs_tree_lock(buf, nest);
 	btrfs_clean_tree_block(buf);
 	clear_bit(EXTENT_BUFFER_STALE, &buf->bflags);
+	clear_bit(EXTENT_BUFFER_NO_CHECK, &buf->bflags);
 
 	btrfs_set_lock_blocking_write(buf);
 	set_extent_buffer_uptodate(buf);
fs/btrfs/extent_io.c +4 -0
index 60f5f68d892d..e91c504fe973 100644
@@ -24,6 +24,7 @@
 #include "rcu-string.h"
 #include "backref.h"
 #include "disk-io.h"
+#include "zoned.h"
 
 static struct kmem_cache *extent_state_cache;
 static struct kmem_cache *extent_buffer_cache;
@@ -4959,6 +4960,7 @@ __alloc_extent_buffer(struct btrfs_fs_info *fs_info, u64 start,
 
 	btrfs_leak_debug_add(&fs_info->eb_leak_lock, &eb->leak_list,
 			     &fs_info->allocated_ebs);
+	INIT_LIST_HEAD(&eb->release_list);
 
 	spin_lock_init(&eb->refs_lock);
 	atomic_set(&eb->refs, 1);
@@ -5744,6 +5746,8 @@ void write_extent_buffer(const struct extent_buffer *eb, const void *srcv,
 	char *src = (char *)srcv;
 	unsigned long i = start >> PAGE_SHIFT;
 
+	WARN_ON(test_bit(EXTENT_BUFFER_NO_CHECK, &eb->bflags));
+
 	if (check_eb_range(eb, start, len))
 		return;
 
fs/btrfs/extent_io.h +2 -0
index f39d02e7f7ef..5f2ccfd0205e 100644
@@ -30,6 +30,7 @@ enum {
 	EXTENT_BUFFER_IN_TREE,
 	/* write IO error */
 	EXTENT_BUFFER_WRITE_ERR,
+	EXTENT_BUFFER_NO_CHECK,
 };
 
 /* these are flags for __process_pages_contig */
@@ -107,6 +108,7 @@ struct extent_buffer {
 	 */
 	wait_queue_head_t read_lock_wq;
 	struct page *pages[INLINE_EXTENT_BUFFER_PAGES];
+	struct list_head release_list;
 #ifdef CONFIG_BTRFS_DEBUG
 	int spinning_writers;
 	atomic_t spinning_readers;
fs/btrfs/transaction.c +10 -0
index 52ada47aff50..5c561cdcce42 100644
@@ -22,6 +22,7 @@
 #include "qgroup.h"
 #include "block-group.h"
 #include "space-info.h"
+#include "zoned.h"
 
 #define BTRFS_ROOT_TRANS_TAG 0
 
@@ -336,6 +337,8 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,
 	spin_lock_init(&cur_trans->dirty_bgs_lock);
 	INIT_LIST_HEAD(&cur_trans->deleted_bgs);
 	spin_lock_init(&cur_trans->dropped_roots_lock);
+	INIT_LIST_HEAD(&cur_trans->releasing_ebs);
+	spin_lock_init(&cur_trans->releasing_ebs_lock);
 	list_add_tail(&cur_trans->list, &fs_info->trans_list);
 	extent_io_tree_init(fs_info, &cur_trans->dirty_pages,
 			IO_TREE_TRANS_DIRTY_PAGES, fs_info->btree_inode);
@@ -2345,6 +2348,13 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans)
 		goto scrub_continue;
 	}
 
+	/*
+	 * At this point, we should have written the all tree blocks
+	 * allocated in this transaction. So it's now safe to free the
+	 * redirtyied extent buffers.
+	 */
+	btrfs_free_redirty_list(cur_trans);
+
 	ret = write_all_supers(fs_info, 0);
 	/*
 	 * the super is written, we can safely allow the tree-loggers
fs/btrfs/transaction.h +3 -0
index 858d9153a1cd..380e0aaa15b3 100644
@@ -92,6 +92,9 @@ struct btrfs_transaction {
 	 */
 	atomic_t pending_ordered;
 	wait_queue_head_t pending_wait;
+
+	spinlock_t releasing_ebs_lock;
+	struct list_head releasing_ebs;
 };
 
 #define __TRANS_FREEZABLE	(1U << 0)
fs/btrfs/tree-log.c +6 -0
index 56cbc1706b6f..5f585cf57383 100644
@@ -20,6 +20,7 @@
 #include "inode-map.h"
 #include "block-group.h"
 #include "space-info.h"
+#include "zoned.h"
 
 /* magic values for the inode_only field in btrfs_log_inode:
  *
@@ -2742,6 +2743,8 @@ static noinline int walk_down_log_tree(struct btrfs_trans_handle *trans,
 						free_extent_buffer(next);
 						return ret;
 					}
+					btrfs_redirty_list_add(
+						trans->transaction, next);
 				} else {
 					if (test_and_clear_bit(EXTENT_BUFFER_DIRTY, &next->bflags))
 						clear_extent_buffer_dirty(next);
@@ -3277,6 +3280,9 @@ static void free_log_tree(struct btrfs_trans_handle *trans,
 	clear_extent_bits(&log->dirty_log_pages, 0, (u64)-1,
 			  EXTENT_DIRTY | EXTENT_NEW | EXTENT_NEED_WAIT);
 	extent_io_tree_release(&log->log_csum_range);
+
+	if (trans && log->node)
+		btrfs_redirty_list_add(trans->transaction, log->node);
 	btrfs_put_root(log);
 }
 
fs/btrfs/zoned.c +37 -0
index d94a2c363a47..b45ca33282d9 100644
@@ -10,6 +10,7 @@
 #include "rcu-string.h"
 #include "disk-io.h"
 #include "block-group.h"
+#include "transaction.h"
 
 /* Maximum number of zones to report per blkdev_report_zones() call */
 #define BTRFS_REPORT_NR_ZONES   4096
@@ -1015,3 +1016,39 @@ void btrfs_calc_zone_unusable(struct btrfs_block_group *cache)
 	 */
 	btrfs_free_excluded_extents(cache);
 }
+
+void btrfs_redirty_list_add(struct btrfs_transaction *trans,
+			    struct extent_buffer *eb)
+{
+	struct btrfs_fs_info *fs_info = eb->fs_info;
+
+	if (!btrfs_is_zoned(fs_info) ||
+	    btrfs_header_flag(eb, BTRFS_HEADER_FLAG_WRITTEN) ||
+	    !list_empty(&eb->release_list))
+		return;
+
+	set_extent_buffer_dirty(eb);
+	set_extent_bits_nowait(&trans->dirty_pages, eb->start,
+			       eb->start + eb->len - 1, EXTENT_DIRTY);
+	memzero_extent_buffer(eb, 0, eb->len);
+	set_bit(EXTENT_BUFFER_NO_CHECK, &eb->bflags);
+
+	spin_lock(&trans->releasing_ebs_lock);
+	list_add_tail(&eb->release_list, &trans->releasing_ebs);
+	spin_unlock(&trans->releasing_ebs_lock);
+	atomic_inc(&eb->refs);
+}
+
+void btrfs_free_redirty_list(struct btrfs_transaction *trans)
+{
+	spin_lock(&trans->releasing_ebs_lock);
+	while (!list_empty(&trans->releasing_ebs)) {
+		struct extent_buffer *eb;
+
+		eb = list_first_entry(&trans->releasing_ebs,
+				      struct extent_buffer, release_list);
+		list_del_init(&eb->release_list);
+		free_extent_buffer(eb);
+	}
+	spin_unlock(&trans->releasing_ebs_lock);
+}
fs/btrfs/zoned.h +6 -0
index d13bc6d70ea4..845623932fa5 100644
@@ -41,6 +41,9 @@ int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
 int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
 int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache);
 void btrfs_calc_zone_unusable(struct btrfs_block_group *cache);
+void btrfs_redirty_list_add(struct btrfs_transaction *trans,
+			    struct extent_buffer *eb);
+void btrfs_free_redirty_list(struct btrfs_transaction *trans);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -107,6 +110,9 @@ static inline int btrfs_load_block_group_zone_info(
 	return 0;
 }
 static inline void btrfs_calc_zone_unusable(struct btrfs_block_group *cache) { }
+static inline void btrfs_redirty_list_add(struct btrfs_transaction *trans,
+					  struct extent_buffer *eb) { }
+static inline void btrfs_free_redirty_list(struct btrfs_transaction *trans) { }
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
#65 lore raw

Re: [PATCH v9 19/41] btrfs: redirty released extent buffers in ZONED mode

Quoted 11 lines · Naohiro Aota

Tree manipulating operations like merging nodes often release once-allocated tree nodes. Btrfs cleans such nodes so that pages in the node are not uselessly written out. On ZONED volumes, however, such optimization blocks the following IOs as the cancellation of the write out of the freed blocks breaks the sequential write sequence expected by the device.

This patch introduces a list of clean and unwritten extent buffers that have been released in a transaction. Btrfs redirty the buffer so that btree_write_cache_pages() can send proper bios to the devices.

Quoted 6 lines

Besides it clears the entire content of the extent buffer not to confuse raw block scanners e.g. btrfsck. By clearing the content, csum_dirty_buffer() complains about bytenr mismatch, so avoid the checking and checksum using newly introduced buffer flag EXTENT_BUFFER_NO_CHECK.

This is a lot of work when you could just add

if (btrfs_is_zoned(fs_info))

	return;

to btrfs_clean_tree_block(). The dirty secret is we don't actually unset the bits in the transaction io tree because it would require memory allocation sometimes, so you don't even need to mess with ->dirty_pages in the first place.
The only thing you need is to keep from clearing the EB dirty. In fact you
could just do

if (btrfs_is_zoned(fs_info)) {
	memzero_extent_buffer(eb, 0, eb->len);
	set_bit(EXTENT_BUFFER_NO_CHECK, &eb->bflags);
}

to btrfs_clean_tree_block() and then in btrfs_free_tree_block() make sure we always pin the extent if we're zoned. Thanks,

Josef

#66 lore raw

Re: [PATCH v9 19/41] btrfs: redirty released extent buffers in ZONED mode

Quoted 11 lines · Josef Bacik

This is a lot of work when you could just add

if (btrfs_is_zoned(fs_info))

	return;

to btrfs_clean_tree_block(). The dirty secret is we don't actually unset the bits in the transaction io tree because it would require memory allocation sometimes, so you don't even need to mess with ->dirty_pages in the first place.
The only thing you need is to keep from clearing the EB dirty. In fact you
could just do

Quoted 7 lines
if (btrfs_is_zoned(fs_info)) {
	memzero_extent_buffer(eb, 0, eb->len);
	set_bit(EXTENT_BUFFER_NO_CHECK, &eb->bflags);
}

to btrfs_clean_tree_block() and then in btrfs_free_tree_block() make sure we always pin the extent if we're zoned. Thanks,

As much as I'd love the simple solution you described it unfortunately didn't work in our testing [1]. So unless I did something completely stupid [2] (which always is an option) I don't think we can go with the easy solution here, unfortunately.

[1]
btrfs/001 [ 16.212869] BTRFS error (device nullb1): unable to find ref byte nr 805388288 parent 805388288 root 5 owner 0 offset 0

[   16.214314] BTRFS: error (device nullb1) in __btrfs_free_extent:3196: errno=-2 No such entry
[   16.215270] BTRFS: error (device nullb1) in btrfs_run_delayed_refs:2215: errno=-2 No such entry                                                                                    
[   16.266987] BTRFS error (device nullb1): unable to find ref byte nr 805388288 parent 805388288 root 5  owner 0 offset 0                                
[   16.268788] BTRFS: error (device nullb1) in __btrfs_free_extent:3196: errno=-2 No such entry                                           
[   16.269973] BTRFS: error (device nullb1) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
[   16.271210] BTRFS: error (device nullb1) in cleanup_transaction:1904: errno=-2 No such entry                                                  
[   16.297358] BTRFS error (device nullb1): unable to find ref byte nr 805502976 parent 805502976 root 1  owner 0 offset 0                       
[   16.299128] BTRFS: error (device nullb1) in __btrfs_free_extent:3196: errno=-2 No such entry   
[   16.300324] BTRFS: error (device nullb1) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
[   16.335600] BTRFS error (device nullb1): unable to find ref byte nr 805388288 parent 805388288 root 5  owner 0 offset 0
[   16.337600] BTRFS: error (device nullb1) in __btrfs_free_extent:3196: errno=-2 No such entry
[   16.338785] BTRFS: error (device nullb1) in btrfs_run_delayed_refs:2215: errno=-2 No such entry                                                                                    
[   16.340026] BTRFS: error (device nullb1) in cleanup_transaction:1904: errno=-2 No such entry            
[   16.366461] BTRFS error (device nullb0): unable to find ref byte nr 805502976 parent 805502976 root 1  owner 0 offset 0
[   16.368304] BTRFS: error (device nullb0) in __btrfs_free_extent:3196: errno=-2 No such entry
[   16.369480] BTRFS: error (device nullb0) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
[   16.370689] BTRFS: error (device nullb0) in cleanup_transaction:1904: errno=-2 No such entry
[   16.392890] BTRFS error (device nullb1): unable to find ref byte nr 805388288 parent 805388288 root 5  owner 0 offset 0
[   16.394531] BTRFS: error (device nullb1) in __btrfs_free_extent:3196: errno=-2 No such entry
[   16.395612] BTRFS: error (device nullb1) in btrfs_run_delayed_refs:2215: errno=-2 No such entry                                                        
[   16.396648] BTRFS: error (device nullb1) in cleanup_transaction:1904: errno=-2 No such entry
[   16.422315] BTRFS error (device nullb1): unable to find ref byte nr 805502976 parent 805502976 root 1  owner 0 offset 0                
[   16.424147] BTRFS: error (device nullb1) in __btrfs_free_extent:3196: errno=-2 No such entry
[   16.425320] BTRFS: error (device nullb1) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
[   16.426532] BTRFS: error (device nullb1) in cleanup_transaction:1904: errno=-2 No such entry

_check_dmesg: something found in dmesg (see /home/johannes/src/xfstests-dev/results//btrfs/001.dmesg)
- output mismatch (see /home/johannes/src/xfstests-dev/results//btrfs/001.out.bad)
--- tests/btrfs/001.out 2020-01-07 15:49:53.000000000 +0000
+++ /home/johannes/src/xfstests-dev/results//btrfs/001.out.bad 2020-11-05 16:17:37.266632915 +0000
@@ -3,38 +3,29 @@
List root dir
foo
Creating snapshot of root dir
+ERROR: cannot snapshot '/mnt/scratch': No such file or directory
Create a snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap'
List root dir after snapshot
-foo
...
(Run 'diff -u /home/johannes/src/xfstests-dev/tests/btrfs/001.out /home/johannes/src/xfstests-dev/results//btrfs/001.out.bad' to see the entire diff)

[2]

fs/btrfs/disk-io.c +5 -3
index e896dd564434..e1bf3b561b45 100644
@@ -1013,7 +1013,11 @@ struct extent_buffer *read_tree_block(struct btrfs_fs_info *fs_info, u64 bytenr,
 void btrfs_clean_tree_block(struct extent_buffer *buf)
 {
        struct btrfs_fs_info *fs_info = buf->fs_info;
-       if (btrfs_header_generation(buf) ==
+
+       if (btrfs_is_zoned(fs_info)) {
+               memzero_extent_buffer(buf, 0, buf->len);
+               set_bit(EXTENT_BUFFER_NO_CHECK, &buf->bflags);
+       } else if (btrfs_header_generation(buf) ==
            fs_info->running_transaction->transid) {
                btrfs_assert_tree_locked(buf);
 
@@ -4639,8 +4643,6 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,
                                     EXTENT_DIRTY);
        btrfs_destroy_pinned_extent(fs_info, &cur_trans->pinned_extents);
 
-       btrfs_free_redirty_list(cur_trans);
-
        cur_trans->state =TRANS_STATE_COMPLETED;
        wake_up(&cur_trans->commit_wait);
 }
fs/btrfs/extent-tree.c +11 -10
index bfd7f3082037..f9f9c329fa69 100644
@@ -3432,15 +3432,23 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
                pin = old_ref_mod >= 0 && new_ref_mod < 0;
        }
 
+       if (btrfs_is_zoned(fs_info)) {
+               struct btrfs_block_group *cache;
+
+               cache = btrfs_lookup_block_group(fs_info, buf->start);
+               pin_down_extent(trans, cache, buf->start, buf->len, 1);
+               btrfs_put_block_group(cache);
+               pin = 0;
+               goto out;
+       }
+
        if (last_ref && btrfs_header_generation(buf) == trans->transid) {
                struct btrfs_block_group *cache;
 
                if (root->root_key.objectid != BTRFS_TREE_LOG_OBJECTID) {
                        ret = check_ref_cleanup(trans, buf->start);
-                       if (!ret) {
-                               btrfs_redirty_list_add(trans->transaction, buf);
+                       if (!ret)
                                goto out;
-                       }
                }
 
                pin = 0;
@@ -3452,13 +3460,6 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
                        goto out;
                }
 
-               if (btrfs_is_zoned(fs_info)) {
-                       btrfs_redirty_list_add(trans->transaction, buf);
-                       pin_down_extent(trans, cache, buf->start, buf->len, 1);
-                       btrfs_put_block_group(cache);
-                       goto out;
-               }
-
                WARN_ON(test_bit(EXTENT_BUFFER_DIRTY, &buf->bflags));
 
                btrfs_add_free_space(cache, buf->start, buf->len);
fs/btrfs/transaction.c +0 -7
index a8561536cd0d..4acc96969959 100644
@@ -2348,13 +2348,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans)
                goto scrub_continue;
        }
 
-       /*
-        * At this point, we should have written all the tree blocks
-        * allocated in this transaction. So it's now safe to free the
-        * redirtyied extent buffers.
-        */
-       btrfs_free_redirty_list(cur_trans);
-
        ret = write_all_supers(fs_info, 0);
        /*
         * the super is written, we can safely allow the tree-loggers
fs/btrfs/tree-log.c +0 -4
index 15f9e8a461ee..e91bdfbb03da 100644
@@ -2763,8 +2763,6 @@ static noinline int walk_down_log_tree(struct btrfs_trans_handle *trans,
                                                free_extent_buffer(next);
                                                return ret;
                                        }
-                                       btrfs_redirty_list_add(
-                                               trans->transaction, next);
                                } else {
                                        if (test_and_clear_bit(EXTENT_BUFFER_DIRTY, &next->bflags))
                                                clear_extent_buffer_dirty(next);
@@ -3313,8 +3311,6 @@ static void free_log_tree(struct btrfs_trans_handle *trans,
                          EXTENT_DIRTY | EXTENT_NEW | EXTENT_NEED_WAIT);
        extent_io_tree_release(&log->log_csum_range);
 
-       if (trans && log->node)
-               btrfs_redirty_list_add(trans->transaction, log->node);
        btrfs_put_root(log);
 }
 
fs/btrfs/zoned.h +0 -7
index 5b61500a0aa9..3140c39d3728 100644
@@ -42,9 +42,6 @@ int btrfs_reset_device_zone(struct btrfs_device *device, u64 physical,
 int btrfs_ensure_empty_zones(struct btrfs_device *device, u64 start, u64 size);
 int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache);
 void btrfs_calc_zone_unusable(struct btrfs_block_group *cache);
-void btrfs_redirty_list_add(struct btrfs_transaction *trans,
-                           struct extent_buffer *eb);
-void btrfs_free_redirty_list(struct btrfs_transaction *trans);
 void btrfs_record_physical_zoned(struct inode *inode, u64 file_offset,
                                 struct bio *bio);
 void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered);
@@ -137,10 +134,6 @@ static inline int btrfs_load_block_group_zone_info(
 
 static inline void btrfs_calc_zone_unusable(struct btrfs_block_group *cache) { }
 
-static inline void btrfs_redirty_list_add(struct btrfs_transaction *trans,
-                                         struct extent_buffer *eb) { }
-static inline void btrfs_free_redirty_list(struct btrfs_transaction *trans) { }
-
 static inline void btrfs_record_physical_zoned(struct inode *inode,
                                               u64 file_offset, struct bio *bio)
 {
#67 lore raw

Re: [PATCH v9 19/41] btrfs: redirty released extent buffers in ZONED mode

Quoted 17 lines · Johannes Thumshirn
Quoted 16 lines · Josef Bacik

This is a lot of work when you could just add

if (btrfs_is_zoned(fs_info))

	return;

to btrfs_clean_tree_block(). The dirty secret is we don't actually unset the bits in the transaction io tree because it would require memory allocation sometimes, so you don't even need to mess with ->dirty_pages in the first place.
The only thing you need is to keep from clearing the EB dirty. In fact you
could just do

if (btrfs_is_zoned(fs_info)) {
	memzero_extent_buffer(eb, 0, eb->len);
	set_bit(EXTENT_BUFFER_NO_CHECK, &eb->bflags);
}
Quoted 7 lines
Quoted 2 lines

to btrfs_clean_tree_block() and then in btrfs_free_tree_block() make sure we always pin the extent if we're zoned. Thanks,

As much as I'd love the simple solution you described it unfortunately didn't work in our testing [1]. So unless I did something completely stupid [2] (which always is an option) I don't think we can go with the easy solution here, unfortunately.

Actually it's because we're calling btrfs_clean_tree_block() in btrfs_init_new_buffer(), so any new block is now getting marked as BUFFER_NO_CHECk, hence everything blowing up.

I think first you push btrfs_clean_tree_block() into btrfs_free_tree_block() and kill all other callers, because we're just marking it no longer dirty. In fact I'd rename it as btrfs_mark_extent_buffer_clean() or something like that. Then your patch should work just fine. Thanks,

Josef

#68 lore raw

PATCH v9 20/41 btrfs: extract page adding function

In reply to Naohiro Aota

This commit extract page adding to bio part from submit_extent_page(). The page is added only when bio_flags are the same, contiguous and the added page fits in the same stripe as pages in the bio.

Condition checkings are reordered to allow early return to avoid possibly heavy btrfs_bio_fits_in_stripe() calling.

---
 fs/btrfs/extent_io.c | 55 ++++++++++++++++++++++++++++++++------------
 1 file changed, 40 insertions(+), 15 deletions(-)
fs/btrfs/extent_io.c +40 -15
index e91c504fe973..17285048fb5a 100644
@@ -3012,6 +3012,43 @@ struct bio *btrfs_bio_clone_partial(struct bio *orig, int offset, int size)
 	return bio;
 }
 
+/**
+ * btrfs_bio_add_page	-	attempt to add a page to bio
+ * @bio:	destination bio
+ * @page:	page to add to the bio
+ * @logical:	offset of the new bio or to check whether we are adding
+ *              a contiguous page to the previous one
+ * @pg_offset:	starting offset in the page
+ * @size:	portion of page that we want to write
+ * @prev_bio_flags:  flags of previous bio to see if we can merge the current one
+ * @bio_flags:	flags of the current bio to see if we can merge them
+ *
+ * Attempt to add a page to bio considering stripe alignment etc. Return
+ * true if successfully page added. Otherwise, return false.
+ */
+bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
+			unsigned int size, unsigned int pg_offset,
+			unsigned long prev_bio_flags, unsigned long bio_flags)
+{
+	sector_t sector = logical >> SECTOR_SHIFT;
+	bool contig;
+
+	if (prev_bio_flags != bio_flags)
+		return false;
+
+	if (prev_bio_flags & EXTENT_BIO_COMPRESSED)
+		contig = bio->bi_iter.bi_sector == sector;
+	else
+		contig = bio_end_sector(bio) == sector;
+	if (!contig)
+		return false;
+
+	if (btrfs_bio_fits_in_stripe(page, size, bio, bio_flags))
+		return false;
+
+	return bio_add_page(bio, page, size, pg_offset) == size;
+}
+
 /*
  * @opf:	bio REQ_OP_* and REQ_* flags as one value
  * @wbc:	optional writeback control for io accounting
@@ -3040,27 +3077,15 @@ static int submit_extent_page(unsigned int opf,
 	int ret = 0;
 	struct bio *bio;
 	size_t page_size = min_t(size_t, size, PAGE_SIZE);
-	sector_t sector = offset >> 9;
 	struct extent_io_tree *tree = &BTRFS_I(page->mapping->host)->io_tree;
 
 	ASSERT(bio_ret);
 
 	if (*bio_ret) {
-		bool contig;
-		bool can_merge = true;
-
 		bio = *bio_ret;
-		if (prev_bio_flags & EXTENT_BIO_COMPRESSED)
-			contig = bio->bi_iter.bi_sector == sector;
-		else
-			contig = bio_end_sector(bio) == sector;
-
-		if (btrfs_bio_fits_in_stripe(page, page_size, bio, bio_flags))
-			can_merge = false;
-
-		if (prev_bio_flags != bio_flags || !contig || !can_merge ||
-		    force_bio_submit ||
-		    bio_add_page(bio, page, page_size, pg_offset) < page_size) {
+		if (force_bio_submit ||
+		    !btrfs_bio_add_page(bio, page, offset, page_size, pg_offset,
+					prev_bio_flags, bio_flags)) {
 			ret = submit_one_bio(bio, mirror_num, prev_bio_flags);
 			if (ret < 0) {
 				*bio_ret = NULL;
Signature · 1 line
2.27.0
Reviewed-by on v9 20/41
#69 lore raw

Re: [PATCH v9 20/41] btrfs: extract page adding function

Quoted 27 lines · Naohiro Aota

This commit extract page adding to bio part from submit_extent_page(). The page is added only when bio_flags are the same, contiguous and the added page fits in the same stripe as pages in the bio.

Condition checkings are reordered to allow early return to avoid possibly heavy btrfs_bio_fits_in_stripe() calling.

---
  fs/btrfs/extent_io.c | 55 ++++++++++++++++++++++++++++++++------------
  1 file changed, 40 insertions(+), 15 deletions(-)
fs/btrfs/extent_io.c +7 -0
index e91c504fe973..17285048fb5a 100644
@@ -3012,6 +3012,43 @@ struct bio *btrfs_bio_clone_partial(struct bio *orig, int offset, int size)
  	return bio;
  }
 
+/**
+ * btrfs_bio_add_page	-	attempt to add a page to bio
+ * @bio:	destination bio
+ * @page:	page to add to the bio
+ * @logical:	offset of the new bio or to check whether we are adding
+ *              a contiguous page to the previous one
+ * @pg_offset:	starting offset in the page

fs/btrfs/extent_io.c struct bio *btrfs_bio_clone_partial(struct bio *orig, int offset, int size) · line 3031

Quoted 10 lines
fs/btrfs/extent_io.c +10 -0
+ * @size:	portion of page that we want to write
+ * @prev_bio_flags:  flags of previous bio to see if we can merge the current one
+ * @bio_flags:	flags of the current bio to see if we can merge them
+ *
+ * Attempt to add a page to bio considering stripe alignment etc. Return
+ * true if successfully page added. Otherwise, return false.
+ */
+bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
+			unsigned int size, unsigned int pg_offset,
+			unsigned long prev_bio_flags, unsigned long bio_flags)

This should be static, once you change that you can add

Thanks,

Josef

#70 lore raw

PATCH v9 21/41 btrfs: use bio_add_zone_append_page for zoned btrfs

In reply to Naohiro Aota

Zoned device has its own hardware restrictions e.g. max_zone_append_size when using REQ_OP_ZONE_APPEND. To follow the restrictions, use bio_add_zone_append_page() instead of bio_add_page(). We need target device to use bio_add_zone_append_page(), so this commit reads the chunk information to memoize the target device to btrfs_io_bio(bio)->device.

Currently, zoned btrfs only supports SINGLE profile. In the feature, btrfs_io_bio can hold extent_map and check the restrictions for all the devices the bio will be mapped.

---
 fs/btrfs/extent_io.c | 37 ++++++++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +34 -3
index 17285048fb5a..764257eb658f 100644
@@ -3032,6 +3032,7 @@ bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
 {
 	sector_t sector = logical >> SECTOR_SHIFT;
 	bool contig;
+	int ret;
 
 	if (prev_bio_flags != bio_flags)
 		return false;
@@ -3046,7 +3047,19 @@ bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
 	if (btrfs_bio_fits_in_stripe(page, size, bio, bio_flags))
 		return false;
 
-	return bio_add_page(bio, page, size, pg_offset) == size;
+	if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+		struct bio orig_bio;
+
+		memset(&orig_bio, 0, sizeof(orig_bio));
+		bio_copy_dev(&orig_bio, bio);
+		bio_set_dev(bio, btrfs_io_bio(bio)->device->bdev);
+		ret = bio_add_zone_append_page(bio, page, size, pg_offset);
+		bio_copy_dev(bio, &orig_bio);
+	} else {
+		ret = bio_add_page(bio, page, size, pg_offset);
+	}
+
+	return ret == size;
 }
 
 /*
@@ -3077,7 +3090,9 @@ static int submit_extent_page(unsigned int opf,
 	int ret = 0;
 	struct bio *bio;
 	size_t page_size = min_t(size_t, size, PAGE_SIZE);
-	struct extent_io_tree *tree = &BTRFS_I(page->mapping->host)->io_tree;
+	struct btrfs_inode *inode = BTRFS_I(page->mapping->host);
+	struct extent_io_tree *tree = &inode->io_tree;
+	struct btrfs_fs_info *fs_info = inode->root->fs_info;
 
 	ASSERT(bio_ret);
 
@@ -3108,11 +3123,27 @@ static int submit_extent_page(unsigned int opf,
 	if (wbc) {
 		struct block_device *bdev;
 
-		bdev = BTRFS_I(page->mapping->host)->root->fs_info->fs_devices->latest_bdev;
+		bdev = fs_info->fs_devices->latest_bdev;
 		bio_set_dev(bio, bdev);
 		wbc_init_bio(wbc, bio);
 		wbc_account_cgroup_owner(wbc, page, page_size);
 	}
+	if (btrfs_is_zoned(fs_info) &&
+	    bio_op(bio) == REQ_OP_ZONE_APPEND) {
+		struct extent_map *em;
+		struct map_lookup *map;
+
+		em = btrfs_get_chunk_map(fs_info, offset, page_size);
+		if (IS_ERR(em))
+			return PTR_ERR(em);
+
+		map = em->map_lookup;
+		/* only support SINGLE profile for now */
+		ASSERT(map->num_stripes == 1);
+		btrfs_io_bio(bio)->device = map->stripes[0].dev;
+
+		free_extent_map(em);
+	}
 
 	*bio_ret = bio;
 
Signature · 1 line
2.27.0
#71 lore raw

Re: [PATCH v9 21/41] btrfs: use bio_add_zone_append_page for zoned btrfs

Quoted 30 lines · Naohiro Aota

Zoned device has its own hardware restrictions e.g. max_zone_append_size when using REQ_OP_ZONE_APPEND. To follow the restrictions, use bio_add_zone_append_page() instead of bio_add_page(). We need target device to use bio_add_zone_append_page(), so this commit reads the chunk information to memoize the target device to btrfs_io_bio(bio)->device.

Currently, zoned btrfs only supports SINGLE profile. In the feature, btrfs_io_bio can hold extent_map and check the restrictions for all the devices the bio will be mapped.

---
  fs/btrfs/extent_io.c | 37 ++++++++++++++++++++++++++++++++++---
  1 file changed, 34 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +1 -0
index 17285048fb5a..764257eb658f 100644
@@ -3032,6 +3032,7 @@ bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
  {
  	sector_t sector = logical >> SECTOR_SHIFT;
  	bool contig;
+	int ret;
 
  	if (prev_bio_flags != bio_flags)
  		return false;
@@ -3046,7 +3047,19 @@ bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
  	if (btrfs_bio_fits_in_stripe(page, size, bio, bio_flags))
  		return false;

fs/btrfs/extent_io.c bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical, · line 3057

Quoted 10 lines
fs/btrfs/extent_io.c +8 -1
 
-	return bio_add_page(bio, page, size, pg_offset) == size;
+	if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+		struct bio orig_bio;
+
+		memset(&orig_bio, 0, sizeof(orig_bio));
+		bio_copy_dev(&orig_bio, bio);
+		bio_set_dev(bio, btrfs_io_bio(bio)->device->bdev);
+		ret = bio_add_zone_append_page(bio, page, size, pg_offset);
+		bio_copy_dev(bio, &orig_bio);

Why do we need this in the first place, since we're only supporting single right now? latest_bdev should be the same, so this serves no purpose, right?

And if it does, we need to figure out another solution, because right now this leaks references to the bio->bi_blkg, each copy inc's the refcount on the blkg for that bio so we're going to leak blkg's like crazy here. Thanks,

Josef

#72 lore raw

Re: [PATCH v9 21/41] btrfs: use bio_add_zone_append_page for zoned btrfs

Quoted 42 lines · Josef Bacik
Quoted 28 lines · Naohiro Aota

Zoned device has its own hardware restrictions e.g. max_zone_append_size when using REQ_OP_ZONE_APPEND. To follow the restrictions, use bio_add_zone_append_page() instead of bio_add_page(). We need target device to use bio_add_zone_append_page(), so this commit reads the chunk information to memoize the target device to btrfs_io_bio(bio)->device.

Currently, zoned btrfs only supports SINGLE profile. In the feature, btrfs_io_bio can hold extent_map and check the restrictions for all the devices the bio will be mapped.

---
 fs/btrfs/extent_io.c | 37 ++++++++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)
fs/btrfs/extent_io.c +1 -0
index 17285048fb5a..764257eb658f 100644
@@ -3032,6 +3032,7 @@ bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
 {
 	sector_t sector = logical >> SECTOR_SHIFT;
 	bool contig;
+	int ret;
 	if (prev_bio_flags != bio_flags)
 		return false;
@@ -3046,7 +3047,19 @@ bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical,
 	if (btrfs_bio_fits_in_stripe(page, size, bio, bio_flags))

fs/btrfs/extent_io.c bool btrfs_bio_add_page(struct bio *bio, struct page *page, u64 logical, · line 3056

Quoted 10 lines
fs/btrfs/extent_io.c +8 -1
 		return false;
-	return bio_add_page(bio, page, size, pg_offset) == size;
+	if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+		struct bio orig_bio;
+
+		memset(&orig_bio, 0, sizeof(orig_bio));
+		bio_copy_dev(&orig_bio, bio);
+		bio_set_dev(bio, btrfs_io_bio(bio)->device->bdev);
+		ret = bio_add_zone_append_page(bio, page, size, pg_offset);
+		bio_copy_dev(bio, &orig_bio);

Why do we need this in the first place, since we're only supporting single right now? latest_bdev should be the same, so this serves no purpose, right?

Quoted 3 lines

And if it does, we need to figure out another solution, because right now this leaks references to the bio->bi_blkg, each copy inc's the refcount on the blkg for that bio so we're going to leak blkg's like crazy here. Thanks,

Josef

True. I just tried to make it multipe device ready as much as possible.

I'll drop this part for now, and will fix this when we add multipe device support.

#73 lore raw

PATCH v9 22/41 btrfs: handle REQ_OP_ZONE_APPEND as writing

In reply to Naohiro Aota

ZONED btrfs uses REQ_OP_ZONE_APPEND bios for writing to actual devices. Let btrfs_end_bio() and btrfs_op be aware of it.

---
 fs/btrfs/disk-io.c |  4 ++--
 fs/btrfs/inode.c   | 10 +++++-----
 fs/btrfs/volumes.c |  8 ++++----
 fs/btrfs/volumes.h |  1 +
 4 files changed, 12 insertions(+), 11 deletions(-)
fs/btrfs/disk-io.c +2 -2
index 9750b4e6d538..778716e223ff 100644
@@ -652,7 +652,7 @@ static void end_workqueue_bio(struct bio *bio)
 	fs_info = end_io_wq->info;
 	end_io_wq->status = bio->bi_status;
 
-	if (bio_op(bio) == REQ_OP_WRITE) {
+	if (btrfs_op(bio) == BTRFS_MAP_WRITE) {
 		if (end_io_wq->metadata == BTRFS_WQ_ENDIO_METADATA)
 			wq = fs_info->endio_meta_write_workers;
 		else if (end_io_wq->metadata == BTRFS_WQ_ENDIO_FREE_SPACE)
@@ -827,7 +827,7 @@ blk_status_t btrfs_submit_metadata_bio(struct inode *inode, struct bio *bio,
 	int async = check_async_write(fs_info, BTRFS_I(inode));
 	blk_status_t ret;
 
-	if (bio_op(bio) != REQ_OP_WRITE) {
+	if (btrfs_op(bio) != BTRFS_MAP_WRITE) {
 		/*
 		 * called for a read, do the setup so that checksum validation
 		 * can happen in the async kernel threads
fs/btrfs/inode.c +5 -5
index 936c3137c646..591ca539e444 100644
@@ -2192,7 +2192,7 @@ blk_status_t btrfs_submit_data_bio(struct inode *inode, struct bio *bio,
 	if (btrfs_is_free_space_inode(BTRFS_I(inode)))
 		metadata = BTRFS_WQ_ENDIO_FREE_SPACE;
 
-	if (bio_op(bio) != REQ_OP_WRITE) {
+	if (btrfs_op(bio) != BTRFS_MAP_WRITE) {
 		ret = btrfs_bio_wq_end_io(fs_info, bio, metadata);
 		if (ret)
 			goto out;
@@ -7526,7 +7526,7 @@ static void btrfs_dio_private_put(struct btrfs_dio_private *dip)
 	if (!refcount_dec_and_test(&dip->refs))
 		return;
 
-	if (bio_op(dip->dio_bio) == REQ_OP_WRITE) {
+	if (btrfs_op(dip->dio_bio) == BTRFS_MAP_WRITE) {
 		__endio_write_update_ordered(BTRFS_I(dip->inode),
 					     dip->logical_offset,
 					     dip->bytes,
@@ -7695,7 +7695,7 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,
 {
 	struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
 	struct btrfs_dio_private *dip = bio->bi_private;
-	bool write = bio_op(bio) == REQ_OP_WRITE;
+	bool write = btrfs_op(bio) == BTRFS_MAP_WRITE;
 	blk_status_t ret;
 
 	/* Check btrfs_submit_bio_hook() for rules about async submit. */
@@ -7746,7 +7746,7 @@ static struct btrfs_dio_private *btrfs_create_dio_private(struct bio *dio_bio,
 							  struct inode *inode,
 							  loff_t file_offset)
 {
-	const bool write = (bio_op(dio_bio) == REQ_OP_WRITE);
+	const bool write = (btrfs_op(dio_bio) == BTRFS_MAP_WRITE);
 	const bool csum = !(BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM);
 	size_t dip_size;
 	struct btrfs_dio_private *dip;
@@ -7777,7 +7777,7 @@ static struct btrfs_dio_private *btrfs_create_dio_private(struct bio *dio_bio,
 static blk_qc_t btrfs_submit_direct(struct inode *inode, struct iomap *iomap,
 		struct bio *dio_bio, loff_t file_offset)
 {
-	const bool write = (bio_op(dio_bio) == REQ_OP_WRITE);
+	const bool write = (btrfs_op(dio_bio) == BTRFS_MAP_WRITE);
 	const bool csum = !(BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM);
 	struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
 	const bool raid56 = (btrfs_data_alloc_profile(fs_info) &
fs/btrfs/volumes.c +4 -4
index 9e69222086ae..a0056b205964 100644
@@ -6446,7 +6446,7 @@ static void btrfs_end_bio(struct bio *bio)
 			struct btrfs_device *dev = btrfs_io_bio(bio)->device;
 
 			ASSERT(dev->bdev);
-			if (bio_op(bio) == REQ_OP_WRITE)
+			if (btrfs_op(bio) == BTRFS_MAP_WRITE)
 				btrfs_dev_stat_inc_and_print(dev,
 						BTRFS_DEV_STAT_WRITE_ERRS);
 			else if (!(bio->bi_opf & REQ_RAHEAD))
@@ -6559,10 +6559,10 @@ blk_status_t btrfs_map_bio(struct btrfs_fs_info *fs_info, struct bio *bio,
 	atomic_set(&bbio->stripes_pending, bbio->num_stripes);
 
 	if ((bbio->map_type & BTRFS_BLOCK_GROUP_RAID56_MASK) &&
-	    ((bio_op(bio) == REQ_OP_WRITE) || (mirror_num > 1))) {
+	    ((btrfs_op(bio) == BTRFS_MAP_WRITE) || (mirror_num > 1))) {
 		/* In this case, map_length has been set to the length of
 		   a single stripe; not the whole write */
-		if (bio_op(bio) == REQ_OP_WRITE) {
+		if (btrfs_op(bio) == BTRFS_MAP_WRITE) {
 			ret = raid56_parity_write(fs_info, bio, bbio,
 						  map_length);
 		} else {
@@ -6585,7 +6585,7 @@ blk_status_t btrfs_map_bio(struct btrfs_fs_info *fs_info, struct bio *bio,
 		dev = bbio->stripes[dev_nr].dev;
 		if (!dev || !dev->bdev || test_bit(BTRFS_DEV_STATE_MISSING,
 						   &dev->dev_state) ||
-		    (bio_op(first_bio) == REQ_OP_WRITE &&
+		    (btrfs_op(first_bio) == BTRFS_MAP_WRITE &&
 		    !test_bit(BTRFS_DEV_STATE_WRITEABLE, &dev->dev_state))) {
 			bbio_error(bbio, first_bio, logical);
 			continue;
fs/btrfs/volumes.h +1 -0
index 0249aca668fb..cff1f7689eac 100644
@@ -410,6 +410,7 @@ static inline enum btrfs_map_op btrfs_op(struct bio *bio)
 	case REQ_OP_DISCARD:
 		return BTRFS_MAP_DISCARD;
 	case REQ_OP_WRITE:
+	case REQ_OP_ZONE_APPEND:
 		return BTRFS_MAP_WRITE;
 	default:
 		WARN_ON_ONCE(1);
Signature · 1 line
2.27.0
#75 lore raw

PATCH v9 23/41 btrfs: split ordered extent when bio is sent

In reply to Naohiro Aota

For a zone append write, the device decides the location the data is written to. Therefore we cannot ensure that two bios are written consecutively on the device. In order to ensure that a ordered extent maps to a contiguous region on disk, we need to maintain a "one bio == one ordered extent" rule.

This commit implements the splitting of an ordered extent and extent map on bio submission to adhere to the rule.

---
 fs/btrfs/inode.c        | 89 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/ordered-data.c | 76 +++++++++++++++++++++++++++++++++++
 fs/btrfs/ordered-data.h |  2 +
 3 files changed, 167 insertions(+)
fs/btrfs/inode.c +89 -0
index 591ca539e444..6b2569dfc3bd 100644
@@ -2158,6 +2158,86 @@ static blk_status_t btrfs_submit_bio_start(void *private_data, struct bio *bio,
 	return btrfs_csum_one_bio(BTRFS_I(inode), bio, 0, 0);
 }
 
+int extract_ordered_extent(struct inode *inode, struct bio *bio,
+			   loff_t file_offset)
+{
+	struct btrfs_ordered_extent *ordered;
+	struct extent_map *em = NULL, *em_new = NULL;
+	struct extent_map_tree *em_tree = &BTRFS_I(inode)->extent_tree;
+	u64 start = (u64)bio->bi_iter.bi_sector << SECTOR_SHIFT;
+	u64 len = bio->bi_iter.bi_size;
+	u64 end = start + len;
+	u64 ordered_end;
+	u64 pre, post;
+	int ret = 0;
+
+	ordered = btrfs_lookup_ordered_extent(BTRFS_I(inode), file_offset);
+	if (WARN_ON_ONCE(!ordered))
+		return -EIO;
+
+	/* no need to split */
+	if (ordered->disk_num_bytes == len)
+		goto out;
+
+	/* cannot split once end_bio'd ordered extent */
+	if (WARN_ON_ONCE(ordered->bytes_left != ordered->disk_num_bytes)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* we cannot split compressed ordered extent */
+	if (WARN_ON_ONCE(ordered->disk_num_bytes != ordered->num_bytes)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* cannot split waietd ordered extent */
+	if (WARN_ON_ONCE(wq_has_sleeper(&ordered->wait))) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ordered_end = ordered->disk_bytenr + ordered->disk_num_bytes;
+	/* bio must be in one ordered extent */
+	if (WARN_ON_ONCE(start < ordered->disk_bytenr || end > ordered_end)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* checksum list should be empty */
+	if (WARN_ON_ONCE(!list_empty(&ordered->list))) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	pre = start - ordered->disk_bytenr;
+	post = ordered_end - end;
+
+	btrfs_split_ordered_extent(ordered, pre, post);
+
+	read_lock(&em_tree->lock);
+	em = lookup_extent_mapping(em_tree, ordered->file_offset, len);
+	if (!em) {
+		read_unlock(&em_tree->lock);
+		ret = -EIO;
+		goto out;
+	}
+	read_unlock(&em_tree->lock);
+
+	ASSERT(!test_bit(EXTENT_FLAG_COMPRESSED, &em->flags));
+	em_new = create_io_em(BTRFS_I(inode), em->start + pre, len,
+			      em->start + pre, em->block_start + pre, len,
+			      len, len, BTRFS_COMPRESS_NONE,
+			      BTRFS_ORDERED_REGULAR);
+	free_extent_map(em_new);
+
+out:
+	free_extent_map(em);
+	btrfs_put_ordered_extent(ordered);
+
+	return ret;
+}
+
 /*
  * extent_io.c submission hook. This does the right thing for csum calculation
  * on write, or reading the csums from the tree before a read.
@@ -2192,6 +2272,15 @@ blk_status_t btrfs_submit_data_bio(struct inode *inode, struct bio *bio,
 	if (btrfs_is_free_space_inode(BTRFS_I(inode)))
 		metadata = BTRFS_WQ_ENDIO_FREE_SPACE;
 
+	if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+		struct page *page = bio_first_bvec_all(bio)->bv_page;
+		loff_t file_offset = page_offset(page);
+
+		ret = extract_ordered_extent(inode, bio, file_offset);
+		if (ret)
+			goto out;
+	}
+
 	if (btrfs_op(bio) != BTRFS_MAP_WRITE) {
 		ret = btrfs_bio_wq_end_io(fs_info, bio, metadata);
 		if (ret)
fs/btrfs/ordered-data.c +76 -0

82 lines not on this page · raw message

fs/btrfs/ordered-data.h +2 -0
index c3a2325e64a4..e346b03bd66a 100644
@@ -193,6 +193,8 @@ void btrfs_wait_ordered_roots(struct btrfs_fs_info *fs_info, u64 nr,
 void btrfs_lock_and_flush_ordered_range(struct btrfs_inode *inode, u64 start,
 					u64 end,
 					struct extent_state **cached_state);
+void btrfs_split_ordered_extent(struct btrfs_ordered_extent *ordered, u64 pre,
+				u64 post);
 int __init ordered_data_init(void);
 void __cold ordered_data_exit(void);
 
Signature · 1 line
2.27.0
#76 lore raw

Re: [PATCH v9 23/41] btrfs: split ordered extent when bio is sent

Quoted 52 lines · Naohiro Aota

For a zone append write, the device decides the location the data is written to. Therefore we cannot ensure that two bios are written consecutively on the device. In order to ensure that a ordered extent maps to a contiguous region on disk, we need to maintain a "one bio == one ordered extent" rule.

This commit implements the splitting of an ordered extent and extent map on bio submission to adhere to the rule.

---
  fs/btrfs/inode.c        | 89 +++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/ordered-data.c | 76 +++++++++++++++++++++++++++++++++++
  fs/btrfs/ordered-data.h |  2 +
  3 files changed, 167 insertions(+)
fs/btrfs/inode.c +28 -0
index 591ca539e444..6b2569dfc3bd 100644
@@ -2158,6 +2158,86 @@ static blk_status_t btrfs_submit_bio_start(void *private_data, struct bio *bio,
  	return btrfs_csum_one_bio(BTRFS_I(inode), bio, 0, 0);
  }
 
+int extract_ordered_extent(struct inode *inode, struct bio *bio,
+			   loff_t file_offset)
+{
+	struct btrfs_ordered_extent *ordered;
+	struct extent_map *em = NULL, *em_new = NULL;
+	struct extent_map_tree *em_tree = &BTRFS_I(inode)->extent_tree;
+	u64 start = (u64)bio->bi_iter.bi_sector << SECTOR_SHIFT;
+	u64 len = bio->bi_iter.bi_size;
+	u64 end = start + len;
+	u64 ordered_end;
+	u64 pre, post;
+	int ret = 0;
+
+	ordered = btrfs_lookup_ordered_extent(BTRFS_I(inode), file_offset);
+	if (WARN_ON_ONCE(!ordered))
+		return -EIO;
+
+	/* no need to split */
+	if (ordered->disk_num_bytes == len)
+		goto out;
+
+	/* cannot split once end_bio'd ordered extent */
+	if (WARN_ON_ONCE(ordered->bytes_left != ordered->disk_num_bytes)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* we cannot split compressed ordered extent */

fs/btrfs/inode.c static blk_status_t btrfs_submit_bio_start(void *private_data, struct bio *bio, · line 2198

Quoted 10 lines
fs/btrfs/inode.c +10 -0
+	if (WARN_ON_ONCE(ordered->disk_num_bytes != ordered->num_bytes)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* cannot split waietd ordered extent */
+	if (WARN_ON_ONCE(wq_has_sleeper(&ordered->wait))) {
+		ret = -EINVAL;
+		goto out;
+	}

This is bad, we could choose any moment to wait on an ordered extent, and then this will break.

In fact I'm not a fan of any of this code. I assume we only know at bio_add_zone_append_page time how much we'll be able to shove into a bio? Then I think the best/cleanest approach here is going to be to add something like what compressed does, an entire alternate way to allocate and submit extents.

It would look something like

->lock pages
->reserve space

loop until all pages are submitted

	->build bio
	->add ordered extent for the bio

->unlock pages

Then the ordered extents are their correct size and you don't have to worry about arbitrary waiters on ordered extents screwing things up, and you don't have to split ordered extents after the fact. Thanks,

Josef

#77 lore raw

PATCH v9 24/41 btrfs: extend btrfs_rmap_block for specifying a device

In reply to Naohiro Aota

btrfs_rmap_block currently reverse-maps the physical addresses on all devices to the corresponding logical addresses.

This commit extends the function to match to a specified device. The old functionality of querying all devices is left intact by specifying NULL as target device.

We pass block_device instead of btrfs_device to __btrfs_rmap_block. This function is intended to reverse-map the result of bio, which only have block_device.

This commit also exports the function for later use.

---
 fs/btrfs/block-group.c | 28 ++++++++++++++++++++++------
 fs/btrfs/block-group.h |  3 +++
 2 files changed, 25 insertions(+), 6 deletions(-)
fs/btrfs/block-group.c +22 -6
index 82d556368c85..21e40046dce1 100644
@@ -1644,8 +1644,11 @@ static void set_avail_alloc_bits(struct btrfs_fs_info *fs_info, u64 flags)
 }
 
 /**
- * btrfs_rmap_block - Map a physical disk address to a list of logical addresses
+ * __btrfs_rmap_block - Map a physical disk address to a list of logical
+ *                      addresses
  * @chunk_start:   logical address of block group
+ * @bdev:	   physical device to resolve. Can be NULL to indicate any
+ *                 device.
  * @physical:	   physical address to map to logical addresses
  * @logical:	   return array of logical addresses which map to @physical
  * @naddrs:	   length of @logical
@@ -1655,9 +1658,9 @@ static void set_avail_alloc_bits(struct btrfs_fs_info *fs_info, u64 flags)
  * Used primarily to exclude those portions of a block group that contain super
  * block copies.
  */
-EXPORT_FOR_TESTS
-int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
-		     u64 physical, u64 **logical, int *naddrs, int *stripe_len)
+int __btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
+		       struct block_device *bdev, u64 physical, u64 **logical,
+		       int *naddrs, int *stripe_len)
 {
 	struct extent_map *em;
 	struct map_lookup *map;
@@ -1675,6 +1678,7 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 	map = em->map_lookup;
 	data_stripe_length = em->orig_block_len;
 	io_stripe_size = map->stripe_len;
+	chunk_start = em->start;
 
 	/* For RAID5/6 adjust to a full IO stripe length */
 	if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK)
@@ -1689,14 +1693,18 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 	for (i = 0; i < map->num_stripes; i++) {
 		bool already_inserted = false;
 		u64 stripe_nr;
+		u64 offset;
 		int j;
 
 		if (!in_range(physical, map->stripes[i].physical,
 			      data_stripe_length))
 			continue;
 
+		if (bdev && map->stripes[i].dev->bdev != bdev)
+			continue;
+
 		stripe_nr = physical - map->stripes[i].physical;
-		stripe_nr = div64_u64(stripe_nr, map->stripe_len);
+		stripe_nr = div64_u64_rem(stripe_nr, map->stripe_len, &offset);
 
 		if (map->type & BTRFS_BLOCK_GROUP_RAID10) {
 			stripe_nr = stripe_nr * map->num_stripes + i;
@@ -1710,7 +1718,7 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 		 * instead of map->stripe_len
 		 */
 
-		bytenr = chunk_start + stripe_nr * io_stripe_size;
+		bytenr = chunk_start + stripe_nr * io_stripe_size + offset;
 
 		/* Ensure we don't add duplicate addresses */
 		for (j = 0; j < nr; j++) {
@@ -1732,6 +1740,14 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
 	return ret;
 }
 
+EXPORT_FOR_TESTS
+int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
+		     u64 physical, u64 **logical, int *naddrs, int *stripe_len)
+{
+	return __btrfs_rmap_block(fs_info, chunk_start, NULL, physical, logical,
+				  naddrs, stripe_len);
+}
+
 static int exclude_super_stripes(struct btrfs_block_group *cache)
 {
 	struct btrfs_fs_info *fs_info = cache->fs_info;
fs/btrfs/block-group.h +3 -0
index 5be47f4bfea7..401e9bcefaec 100644
@@ -275,6 +275,9 @@ void check_system_chunk(struct btrfs_trans_handle *trans, const u64 type);
 u64 btrfs_get_alloc_profile(struct btrfs_fs_info *fs_info, u64 orig_flags);
 void btrfs_put_block_group_cache(struct btrfs_fs_info *info);
 int btrfs_free_block_groups(struct btrfs_fs_info *info);
+int __btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start,
+		       struct block_device *bdev, u64 physical, u64 **logical,
+		       int *naddrs, int *stripe_len);
 
 static inline u64 btrfs_data_alloc_profile(struct btrfs_fs_info *fs_info)
 {
Signature · 1 line
2.27.0
#78 lore raw

Re: [PATCH v9 24/41] btrfs: extend btrfs_rmap_block for specifying a device

Quoted 7 lines · Naohiro Aota

btrfs_rmap_block currently reverse-maps the physical addresses on all devices to the corresponding logical addresses.

This commit extends the function to match to a specified device. The old functionality of querying all devices is left intact by specifying NULL as target device.

Quoted 7 lines

We pass block_device instead of btrfs_device to __btrfs_rmap_block. This function is intended to reverse-map the result of bio, which only have block_device.

This commit also exports the function for later use.

Since there's only one caller of btrfs_rmap_block() in this file, and the rest are for tests, you might as well just make btrfs_rmap_block() exported with the device, and switch the existing callers to use NULL for the device. Thanks,

Josef

#79 lore raw

Re: [PATCH v9 24/41] btrfs: extend btrfs_rmap_block for specifying a device

On 03/11/2020 16:32, Josef Bacik wrote:
[...]

Quoted 3 lines

Since there's only one caller of btrfs_rmap_block() in this file, and the rest are for tests, you might as well just make btrfs_rmap_block() exported with the device, and switch the existing callers to use NULL for the device. Thanks,

Fixed

#80 lore raw

PATCH v9 25/41 btrfs: use ZONE_APPEND write for ZONED btrfs

In reply to Naohiro Aota

This commit enables zone append writing for zoned btrfs. When using zone append, a bio is issued to the start of a target zone and the device decides to place it inside the zone. Upon completion the device reports the actual written position back to the host.

Three parts are necessary to enable zone append in btrfs. First, modify the bio to use REQ_OP_ZONE_APPEND in btrfs_submit_bio_hook() and adjust the bi_sector to point the beginning of the zone.

Secondly, records the returned physical address (and disk/partno) to the ordered extent in end_bio_extent_writepage() after the bio has been completed. We cannot resolve the physical address to the logical address because we can neither take locks nor allocate a buffer in this end_bio context. So, we need to record the physical address to resolve it later in btrfs_finish_ordered_io().

And finally, rewrites the logical addresses of the extent mapping and checksum data according to the physical address (using __btrfs_rmap_block). If the returned address matches the originally allocated address, we can skip this rewriting process.

---
 fs/btrfs/extent_io.c    | 12 +++++++-
 fs/btrfs/file.c         |  2 +-
 fs/btrfs/inode.c        |  4 +++
 fs/btrfs/ordered-data.c |  3 ++
 fs/btrfs/ordered-data.h |  8 +++++
 fs/btrfs/volumes.c      |  9 ++++++
 fs/btrfs/zoned.c        | 68 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h        |  9 ++++++
 8 files changed, 113 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +11 -1
index 764257eb658f..3f49febafc69 100644
@@ -2743,6 +2743,7 @@ static void end_bio_extent_writepage(struct bio *bio)
 	u64 start;
 	u64 end;
 	struct bvec_iter_all iter_all;
+	bool first_bvec = true;
 
 	ASSERT(!bio_flagged(bio, BIO_CLONED));
 	bio_for_each_segment_all(bvec, bio, iter_all) {
@@ -2769,6 +2770,11 @@ static void end_bio_extent_writepage(struct bio *bio)
 		start = page_offset(page);
 		end = start + bvec->bv_offset + bvec->bv_len - 1;
 
+		if (first_bvec) {
+			btrfs_record_physical_zoned(inode, start, bio);
+			first_bvec = false;
+		}
+
 		end_extent_writepage(page, error, start, end);
 		end_page_writeback(page);
 	}
@@ -3531,6 +3537,7 @@ static noinline_for_stack int __extent_writepage_io(struct btrfs_inode *inode,
 	size_t blocksize;
 	int ret = 0;
 	int nr = 0;
+	int opf = REQ_OP_WRITE;
 	const unsigned int write_flags = wbc_to_write_flags(wbc);
 	bool compressed;
 
@@ -3543,6 +3550,9 @@ static noinline_for_stack int __extent_writepage_io(struct btrfs_inode *inode,
 		return 1;
 	}
 
+	if (btrfs_is_zoned(inode->root->fs_info))
+		opf = REQ_OP_ZONE_APPEND;
+
 	/*
 	 * we don't want to touch the inode after unlocking the page,
 	 * so we update the mapping writeback index now
@@ -3603,7 +3613,7 @@ static noinline_for_stack int __extent_writepage_io(struct btrfs_inode *inode,
 			       page->index, cur, end);
 		}
 
-		ret = submit_extent_page(REQ_OP_WRITE | write_flags, wbc,
+		ret = submit_extent_page(opf | write_flags, wbc,
 					 page, offset, iosize, pg_offset,
 					 &epd->bio,
 					 end_bio_extent_writepage,
fs/btrfs/file.c +1 -1
index 68938a43081e..f41cdcbf44f5 100644
@@ -2226,7 +2226,7 @@ int btrfs_sync_file(struct file *file, loff_t start, loff_t end, int datasync)
 	 * the current transaction commits before the ordered extents complete
 	 * and a power failure happens right after that.
 	 */
-	if (full_sync) {
+	if (full_sync || fs_info->zoned) {
 		ret = btrfs_wait_ordered_range(inode, start, len);
 	} else {
 		/*
fs/btrfs/inode.c +4 -0
index 6b2569dfc3bd..6d40affa833f 100644
@@ -51,6 +51,7 @@
 #include "delalloc-space.h"
 #include "block-group.h"
 #include "space-info.h"
+#include "zoned.h"
 
 struct btrfs_iget_args {
 	u64 ino;
@@ -2676,6 +2677,9 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)
 	bool clear_reserved_extent = true;
 	unsigned int clear_bits;
 
+	if (ordered_extent->disk)
+		btrfs_rewrite_logical_zoned(ordered_extent);
+
 	start = ordered_extent->file_offset;
 	end = start + ordered_extent->num_bytes - 1;
 
fs/btrfs/ordered-data.c +3 -0
index 28fb9d5f48d3..b2d185d4479e 100644
@@ -199,6 +199,9 @@ static int __btrfs_add_ordered_extent(struct btrfs_inode *inode, u64 file_offset
 	entry->compress_type = compress_type;
 	entry->truncated_len = (u64)-1;
 	entry->qgroup_rsv = ret;
+	entry->physical = (u64)-1;
+	entry->disk = NULL;
+	entry->partno = (u8)-1;
 	if (type != BTRFS_ORDERED_IO_DONE && type != BTRFS_ORDERED_COMPLETE)
 		set_bit(type, &entry->flags);
 
fs/btrfs/ordered-data.h +8 -0
index e346b03bd66a..084c609afd83 100644
@@ -127,6 +127,14 @@ struct btrfs_ordered_extent {
 	struct completion completion;
 	struct btrfs_work flush_work;
 	struct list_head work_list;
+
+	/*
+	 * used to reverse-map physical address returned from ZONE_APPEND
+	 * write command in a workqueue context.
+	 */
+	u64 physical;
+	struct gendisk *disk;
+	u8 partno;
 };
 
 /*
fs/btrfs/volumes.c +9 -0
index a0056b205964..26669007d367 100644
@@ -6498,6 +6498,15 @@ static void submit_stripe_bio(struct btrfs_bio *bbio, struct bio *bio,
 	btrfs_io_bio(bio)->device = dev;
 	bio->bi_end_io = btrfs_end_bio;
 	bio->bi_iter.bi_sector = physical >> 9;
+	/*
+	 * For zone append writing, bi_sector must point the beginning of the
+	 * zone
+	 */
+	if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+		u64 zone_start = round_down(physical, fs_info->zone_size);
+
+		bio->bi_iter.bi_sector = zone_start >> SECTOR_SHIFT;
+	}
 	btrfs_debug_in_rcu(fs_info,
 	"btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u",
 		bio_op(bio), bio->bi_opf, (u64)bio->bi_iter.bi_sector,
fs/btrfs/zoned.c +68 -0

71 lines not on this page · raw message

fs/btrfs/zoned.h +9 -0
index 845623932fa5..d3ed4d7dae2b 100644
@@ -44,6 +44,9 @@ void btrfs_calc_zone_unusable(struct btrfs_block_group *cache);
 void btrfs_redirty_list_add(struct btrfs_transaction *trans,
 			    struct extent_buffer *eb);
 void btrfs_free_redirty_list(struct btrfs_transaction *trans);
+void btrfs_record_physical_zoned(struct inode *inode, u64 file_offset,
+				 struct bio *bio);
+void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -113,6 +116,12 @@ static inline void btrfs_calc_zone_unusable(struct btrfs_block_group *cache) { }
 static inline void btrfs_redirty_list_add(struct btrfs_transaction *trans,
 					  struct extent_buffer *eb) { }
 static inline void btrfs_free_redirty_list(struct btrfs_transaction *trans) { }
+static inline void btrfs_record_physical_zoned(struct inode *inode,
+					       u64 file_offset, struct bio *bio)
+{
+}
+static inline void
+btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered) { }
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
Reviewed-by on v9 25/41
#81 lore raw

Re: [PATCH v9 25/41] btrfs: use ZONE_APPEND write for ZONED btrfs

Quoted 16 lines · Naohiro Aota

This commit enables zone append writing for zoned btrfs. When using zone append, a bio is issued to the start of a target zone and the device decides to place it inside the zone. Upon completion the device reports the actual written position back to the host.

Three parts are necessary to enable zone append in btrfs. First, modify the bio to use REQ_OP_ZONE_APPEND in btrfs_submit_bio_hook() and adjust the bi_sector to point the beginning of the zone.

Secondly, records the returned physical address (and disk/partno) to the ordered extent in end_bio_extent_writepage() after the bio has been completed. We cannot resolve the physical address to the logical address because we can neither take locks nor allocate a buffer in this end_bio context. So, we need to record the physical address to resolve it later in btrfs_finish_ordered_io().

Quoted 7 lines

And finally, rewrites the logical addresses of the extent mapping and checksum data according to the physical address (using __btrfs_rmap_block). If the returned address matches the originally allocated address, we can skip this rewriting process.

I hate this, but I suppose its better than forcing us to wait for ordered extents in order of submission, and I can't really think of anything better,

Thanks,

Josef

#82 lore raw

PATCH v9 26/41 btrfs: enable zone append writing for direct IO

In reply to Naohiro Aota

Likewise to buffered IO, enable zone append writing for direct IO when its used on a zoned block device.

---
 fs/btrfs/inode.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
fs/btrfs/inode.c +17 -0
index 6d40affa833f..bc853a4f22cc 100644
@@ -7542,6 +7542,9 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start,
 	iomap->bdev = fs_info->fs_devices->latest_bdev;
 	iomap->length = len;
 
+	if (write && btrfs_is_zoned(fs_info) && fs_info->max_zone_append_size)
+		iomap->flags |= IOMAP_F_ZONE_APPEND;
+
 	free_extent_map(em);
 
 	return 0;
@@ -7779,6 +7782,8 @@ static void btrfs_end_dio_bio(struct bio *bio)
 	if (err)
 		dip->dio_bio->bi_status = err;
 
+	btrfs_record_physical_zoned(dip->inode, dip->logical_offset, bio);
+
 	bio_put(bio);
 	btrfs_dio_private_put(dip);
 }
@@ -7933,6 +7938,18 @@ static blk_qc_t btrfs_submit_direct(struct inode *inode, struct iomap *iomap,
 		bio->bi_end_io = btrfs_end_dio_bio;
 		btrfs_io_bio(bio)->logical = file_offset;
 
+		WARN_ON_ONCE(write && btrfs_is_zoned(fs_info) &&
+			     fs_info->max_zone_append_size &&
+			     bio_op(bio) != REQ_OP_ZONE_APPEND);
+
+		if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
+			ret = extract_ordered_extent(inode, bio, file_offset);
+			if (ret) {
+				bio_put(bio);
+				goto out_err;
+			}
+		}
+
 		ASSERT(submit_len >= clone_len);
 		submit_len -= clone_len;
 
Signature · 1 line
2.27.0
#84 lore raw

PATCH v9 27/41 btrfs: introduce dedicated data write path for ZONED mode

In reply to Naohiro Aota

If more than one IO is issued for one file extent, these IO can be written to separate regions on a device. Since we cannot map one file extent to such a separate area, we need to follow the "one IO == one ordered extent" rule.

The Normal buffered, uncompressed, not pre-allocated write path (used by cow_file_range()) sometimes does not follow this rule. It can write a part of an ordered extent when specified a region to write e.g., when its called from fdatasync().

Introduces a dedicated (uncompressed buffered) data write path for ZONED mode. This write path will CoW the region and write it at once.

---
 fs/btrfs/inode.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)
fs/btrfs/inode.c +32 -2
index bc853a4f22cc..fdc367a39194 100644
@@ -1350,6 +1350,29 @@ static int cow_file_range_async(struct btrfs_inode *inode,
 	return 0;
 }
 
+static noinline int run_delalloc_zoned(struct btrfs_inode *inode,
+				       struct page *locked_page, u64 start,
+				       u64 end, int *page_started,
+				       unsigned long *nr_written)
+{
+	int ret;
+
+	ret = cow_file_range(inode, locked_page, start, end,
+			     page_started, nr_written, 0);
+	if (ret)
+		return ret;
+
+	if (*page_started)
+		return 0;
+
+	__set_page_dirty_nobuffers(locked_page);
+	account_page_redirty(locked_page);
+	extent_write_locked_range(&inode->vfs_inode, start, end, WB_SYNC_ALL);
+	*page_started = 1;
+
+	return 0;
+}
+
 static noinline int csum_exist_in_range(struct btrfs_fs_info *fs_info,
 					u64 bytenr, u64 num_bytes)
 {
@@ -1820,17 +1843,24 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct page *locked_page
 {
 	int ret;
 	int force_cow = need_force_cow(inode, start, end);
+	int do_compress = inode_can_compress(inode) &&
+		inode_need_compress(inode, start, end);
+	bool zoned = btrfs_is_zoned(inode->root->fs_info);
 
 	if (inode->flags & BTRFS_INODE_NODATACOW && !force_cow) {
+		ASSERT(!zoned);
 		ret = run_delalloc_nocow(inode, locked_page, start, end,
 					 page_started, 1, nr_written);
 	} else if (inode->flags & BTRFS_INODE_PREALLOC && !force_cow) {
+		ASSERT(!zoned);
 		ret = run_delalloc_nocow(inode, locked_page, start, end,
 					 page_started, 0, nr_written);
-	} else if (!inode_can_compress(inode) ||
-		   !inode_need_compress(inode, start, end)) {
+	} else if (!do_compress && !zoned) {
 		ret = cow_file_range(inode, locked_page, start, end,
 				     page_started, nr_written, 1);
+	} else if (!do_compress && zoned) {
+		ret = run_delalloc_zoned(inode, locked_page, start, end,
+					 page_started, nr_written);
 	} else {
 		set_bit(BTRFS_INODE_HAS_ASYNC_EXTENT, &inode->runtime_flags);
 		ret = cow_file_range_async(inode, wbc, locked_page, start, end,
Signature · 1 line
2.27.0
#85 lore raw

Re: [PATCH v9 27/41] btrfs: introduce dedicated data write path for ZONED mode

Quoted 10 lines · Naohiro Aota

If more than one IO is issued for one file extent, these IO can be written to separate regions on a device. Since we cannot map one file extent to such a separate area, we need to follow the "one IO == one ordered extent" rule.

The Normal buffered, uncompressed, not pre-allocated write path (used by cow_file_range()) sometimes does not follow this rule. It can write a part of an ordered extent when specified a region to write e.g., when its called from fdatasync().

Quoted 4 lines

Introduces a dedicated (uncompressed buffered) data write path for ZONED mode. This write path will CoW the region and write it at once.

This can be modified to work in the way I described in the 'split ordered extent' patch, just flesh this out to do all the work together and that way you bypass all the ugly splitting stuff. Thanks,

Josef

#86 lore raw

PATCH v9 28/41 btrfs: serialize meta IOs on ZONED mode

In reply to Naohiro Aota

We cannot use zone append for writing metadata, because the B-tree nodes have references to each other using the logical address. Without knowing the address in advance, we cannot construct the tree in the first place. So we need to serialize write IOs for metadata.

We cannot add a mutex around allocation and submission because metadata blocks are allocated in an earlier stage to build up B-trees.

Add a zoned_meta_io_lock and hold it during metadata IO submission in btree_write_cache_pages() to serialize IOs. Furthermore, this add a per-block group metadata IO submission pointer "meta_write_pointer" to ensure sequential writing, which can be caused when writing back blocks in an unfinished transaction.

---
 fs/btrfs/block-group.h |  1 +
 fs/btrfs/ctree.h       |  1 +
 fs/btrfs/disk-io.c     |  1 +
 fs/btrfs/extent_io.c   | 27 ++++++++++++++++++++++-
 fs/btrfs/zoned.c       | 50 ++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h       | 31 ++++++++++++++++++++++++++
 6 files changed, 110 insertions(+), 1 deletion(-)
fs/btrfs/block-group.h +1 -0
index 401e9bcefaec..b2a8a3beceac 100644
@@ -190,6 +190,7 @@ struct btrfs_block_group {
 	 */
 	u64 alloc_offset;
 	u64 zone_unusable;
+	u64 meta_write_pointer;
 };
 
 static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/ctree.h +1 -0
index 383c83a1f5b5..736f679f1310 100644
@@ -955,6 +955,7 @@ struct btrfs_fs_info {
 	};
 	/* max size to emit ZONE_APPEND write command */
 	u64 max_zone_append_size;
+	struct mutex zoned_meta_io_lock;
 
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 	spinlock_t ref_verify_lock;
fs/btrfs/disk-io.c +1 -0
index 778716e223ff..f02b121d8213 100644
@@ -2652,6 +2652,7 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info)
 	mutex_init(&fs_info->delete_unused_bgs_mutex);
 	mutex_init(&fs_info->reloc_mutex);
 	mutex_init(&fs_info->delalloc_root_mutex);
+	mutex_init(&fs_info->zoned_meta_io_lock);
 	seqlock_init(&fs_info->profiles_lock);
 
 	INIT_LIST_HEAD(&fs_info->dirty_cowonly_roots);
fs/btrfs/extent_io.c +26 -1
index 3f49febafc69..3cce444d5dbb 100644
@@ -25,6 +25,7 @@
 #include "backref.h"
 #include "disk-io.h"
 #include "zoned.h"
+#include "block-group.h"
 
 static struct kmem_cache *extent_state_cache;
 static struct kmem_cache *extent_buffer_cache;
@@ -4001,6 +4002,7 @@ int btree_write_cache_pages(struct address_space *mapping,
 				   struct writeback_control *wbc)
 {
 	struct extent_buffer *eb, *prev_eb = NULL;
+	struct btrfs_block_group *cache = NULL;
 	struct extent_page_data epd = {
 		.bio = NULL,
 		.extent_locked = 0,
@@ -4035,6 +4037,7 @@ int btree_write_cache_pages(struct address_space *mapping,
 		tag = PAGECACHE_TAG_TOWRITE;
 	else
 		tag = PAGECACHE_TAG_DIRTY;
+	btrfs_zoned_meta_io_lock(fs_info);
 retry:
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		tag_pages_for_writeback(mapping, index, end);
@@ -4077,12 +4080,30 @@ int btree_write_cache_pages(struct address_space *mapping,
 			if (!ret)
 				continue;
 
+			if (!btrfs_check_meta_write_pointer(fs_info, eb,
+							    &cache)) {
+				/*
+				 * If for_sync, this hole will be filled with
+				 * trasnsaction commit.
+				 */
+				if (wbc->sync_mode == WB_SYNC_ALL &&
+				    !wbc->for_sync)
+					ret = -EAGAIN;
+				else
+					ret = 0;
+				done = 1;
+				free_extent_buffer(eb);
+				break;
+			}
+
 			prev_eb = eb;
 			ret = lock_extent_buffer_for_io(eb, &epd);
 			if (!ret) {
+				btrfs_revert_meta_write_pointer(cache, eb);
 				free_extent_buffer(eb);
 				continue;
 			} else if (ret < 0) {
+				btrfs_revert_meta_write_pointer(cache, eb);
 				done = 1;
 				free_extent_buffer(eb);
 				break;
@@ -4115,10 +4136,12 @@ int btree_write_cache_pages(struct address_space *mapping,
 		index = 0;
 		goto retry;
 	}
+	if (cache)
+		btrfs_put_block_group(cache);
 	ASSERT(ret <= 0);
 	if (ret < 0) {
 		end_write_bio(&epd, ret);
-		return ret;
+		goto out;
 	}
 	/*
 	 * If something went wrong, don't allow any metadata write bio to be
@@ -4153,6 +4176,8 @@ int btree_write_cache_pages(struct address_space *mapping,
 		ret = -EROFS;
 		end_write_bio(&epd, ret);
 	}
+out:
+	btrfs_zoned_meta_io_unlock(fs_info);
 	return ret;
 }
 
fs/btrfs/zoned.c +50 -0

59 lines not on this page · raw message

fs/btrfs/zoned.h +31 -0
index d3ed4d7dae2b..262b248a9085 100644
@@ -47,6 +47,11 @@ void btrfs_free_redirty_list(struct btrfs_transaction *trans);
 void btrfs_record_physical_zoned(struct inode *inode, u64 file_offset,
 				 struct bio *bio);
 void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered);
+bool btrfs_check_meta_write_pointer(struct btrfs_fs_info *fs_info,
+				    struct extent_buffer *eb,
+				    struct btrfs_block_group **cache_ret);
+void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
+				     struct extent_buffer *eb);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -122,6 +127,18 @@ static inline void btrfs_record_physical_zoned(struct inode *inode,
 }
 static inline void
 btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered) { }
+static inline bool
+btrfs_check_meta_write_pointer(struct btrfs_fs_info *fs_info,
+			       struct extent_buffer *eb,
+			       struct btrfs_block_group **cache_ret)
+{
+	return true;
+}
+static inline void
+btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
+				struct extent_buffer *eb)
+{
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
@@ -224,4 +241,18 @@ static inline bool btrfs_can_zone_reset(struct btrfs_device *device,
 	return true;
 }
 
+static inline void btrfs_zoned_meta_io_lock(struct btrfs_fs_info *fs_info)
+{
+	if (!btrfs_is_zoned(fs_info))
+		return;
+	mutex_lock(&fs_info->zoned_meta_io_lock);
+}
+
+static inline void btrfs_zoned_meta_io_unlock(struct btrfs_fs_info *fs_info)
+{
+	if (!btrfs_is_zoned(fs_info))
+		return;
+	mutex_unlock(&fs_info->zoned_meta_io_lock);
+}
+
 #endif
Signature · 1 line
2.27.0
Reviewed-by on v9 28/41
#87 lore raw

Re: [PATCH v9 28/41] btrfs: serialize meta IOs on ZONED mode

Quoted 174 lines · Naohiro Aota

We cannot use zone append for writing metadata, because the B-tree nodes have references to each other using the logical address. Without knowing the address in advance, we cannot construct the tree in the first place. So we need to serialize write IOs for metadata.

We cannot add a mutex around allocation and submission because metadata blocks are allocated in an earlier stage to build up B-trees.

Add a zoned_meta_io_lock and hold it during metadata IO submission in btree_write_cache_pages() to serialize IOs. Furthermore, this add a per-block group metadata IO submission pointer "meta_write_pointer" to ensure sequential writing, which can be caused when writing back blocks in an unfinished transaction.

---
  fs/btrfs/block-group.h |  1 +
  fs/btrfs/ctree.h       |  1 +
  fs/btrfs/disk-io.c     |  1 +
  fs/btrfs/extent_io.c   | 27 ++++++++++++++++++++++-
  fs/btrfs/zoned.c       | 50 ++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/zoned.h       | 31 ++++++++++++++++++++++++++
  6 files changed, 110 insertions(+), 1 deletion(-)
fs/btrfs/block-group.h +1 -0
index 401e9bcefaec..b2a8a3beceac 100644
@@ -190,6 +190,7 @@ struct btrfs_block_group {
  	 */
  	u64 alloc_offset;
  	u64 zone_unusable;
+	u64 meta_write_pointer;
  };
 
  static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
fs/btrfs/ctree.h +1 -0
index 383c83a1f5b5..736f679f1310 100644
@@ -955,6 +955,7 @@ struct btrfs_fs_info {
  	};
  	/* max size to emit ZONE_APPEND write command */
  	u64 max_zone_append_size;
+	struct mutex zoned_meta_io_lock;
 
  #ifdef CONFIG_BTRFS_FS_REF_VERIFY
  	spinlock_t ref_verify_lock;
fs/btrfs/disk-io.c +1 -0
index 778716e223ff..f02b121d8213 100644
@@ -2652,6 +2652,7 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info)
  	mutex_init(&fs_info->delete_unused_bgs_mutex);
  	mutex_init(&fs_info->reloc_mutex);
  	mutex_init(&fs_info->delalloc_root_mutex);
+	mutex_init(&fs_info->zoned_meta_io_lock);
  	seqlock_init(&fs_info->profiles_lock);
 
  	INIT_LIST_HEAD(&fs_info->dirty_cowonly_roots);
fs/btrfs/extent_io.c +26 -1
index 3f49febafc69..3cce444d5dbb 100644
@@ -25,6 +25,7 @@
  #include "backref.h"
  #include "disk-io.h"
  #include "zoned.h"
+#include "block-group.h"
 
  static struct kmem_cache *extent_state_cache;
  static struct kmem_cache *extent_buffer_cache;
@@ -4001,6 +4002,7 @@ int btree_write_cache_pages(struct address_space *mapping,
  				   struct writeback_control *wbc)
  {
  	struct extent_buffer *eb, *prev_eb = NULL;
+	struct btrfs_block_group *cache = NULL;
  	struct extent_page_data epd = {
  		.bio = NULL,
  		.extent_locked = 0,
@@ -4035,6 +4037,7 @@ int btree_write_cache_pages(struct address_space *mapping,
  		tag = PAGECACHE_TAG_TOWRITE;
  	else
  		tag = PAGECACHE_TAG_DIRTY;
+	btrfs_zoned_meta_io_lock(fs_info);
  retry:
  	if (wbc->sync_mode == WB_SYNC_ALL)
  		tag_pages_for_writeback(mapping, index, end);
@@ -4077,12 +4080,30 @@ int btree_write_cache_pages(struct address_space *mapping,
  			if (!ret)
  				continue;
 
+			if (!btrfs_check_meta_write_pointer(fs_info, eb,
+							    &cache)) {
+				/*
+				 * If for_sync, this hole will be filled with
+				 * trasnsaction commit.
+				 */
+				if (wbc->sync_mode == WB_SYNC_ALL &&
+				    !wbc->for_sync)
+					ret = -EAGAIN;
+				else
+					ret = 0;
+				done = 1;
+				free_extent_buffer(eb);
+				break;
+			}
+
  			prev_eb = eb;
  			ret = lock_extent_buffer_for_io(eb, &epd);
  			if (!ret) {
+				btrfs_revert_meta_write_pointer(cache, eb);
  				free_extent_buffer(eb);
  				continue;
  			} else if (ret < 0) {
+				btrfs_revert_meta_write_pointer(cache, eb);
  				done = 1;
  				free_extent_buffer(eb);
  				break;
@@ -4115,10 +4136,12 @@ int btree_write_cache_pages(struct address_space *mapping,
  		index = 0;
  		goto retry;
  	}
+	if (cache)
+		btrfs_put_block_group(cache);
  	ASSERT(ret <= 0);
  	if (ret < 0) {
  		end_write_bio(&epd, ret);
-		return ret;
+		goto out;
  	}
  	/*
  	 * If something went wrong, don't allow any metadata write bio to be
@@ -4153,6 +4176,8 @@ int btree_write_cache_pages(struct address_space *mapping,
  		ret = -EROFS;
  		end_write_bio(&epd, ret);
  	}
+out:
+	btrfs_zoned_meta_io_unlock(fs_info);
  	return ret;
  }
 
fs/btrfs/zoned.c +17 -0
index 50393d560c9a..15bc7d451348 100644
@@ -989,6 +989,9 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
  		ret = -EIO;
  	}
 
+	if (!ret)
+		cache->meta_write_pointer = cache->alloc_offset + cache->start;
+
  	kfree(alloc_offsets);
  	free_extent_map(em);
 
@@ -1120,3 +1123,50 @@ void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered)
  	kfree(logical);
  	bdput(bdev);
  }
+
+bool btrfs_check_meta_write_pointer(struct btrfs_fs_info *fs_info,
+				    struct extent_buffer *eb,
+				    struct btrfs_block_group **cache_ret)
+{
+	struct btrfs_block_group *cache;
+
+	if (!btrfs_is_zoned(fs_info))
+		return true;
+
+	cache = *cache_ret;
+
+	if (cache && (eb->start < cache->start ||
+		      cache->start + cache->length <= eb->start)) {

fs/btrfs/zoned.c void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered) · line 1149

Quoted 10 lines
fs/btrfs/zoned.c +10 -0
+		btrfs_put_block_group(cache);
+		cache = NULL;
+		*cache_ret = NULL;
+	}
+
+	if (!cache)
+		cache = btrfs_lookup_block_group(fs_info, eb->start);
+
+	if (cache) {
+		*cache_ret = cache;

Don't set this here, set it after the if statement.

fs/btrfs/zoned.c void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered) · line 1154

Quoted 5 lines
fs/btrfs/zoned.c +5 -0
+
+		if (cache->meta_write_pointer != eb->start) {
+			btrfs_put_block_group(cache);
+			cache = NULL;
+			*cache_ret = NULL;

And delete these two lines. Then you can add

Thanks,

Josef

#88 lore raw

PATCH v9 29/41 btrfs: wait existing extents before truncating

In reply to Naohiro Aota

When truncating a file, file buffers which have already been allocated but not yet written may be truncated. Truncating these buffers could cause breakage of a sequential write pattern in a block group if the truncated blocks are for example followed by blocks allocated to another file. To avoid this problem, always wait for write out of all unwritten buffers before proceeding with the truncate execution.

---
 fs/btrfs/inode.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
fs/btrfs/inode.c +10 -0
index fdc367a39194..de8b58abed1d 100644
@@ -4955,6 +4955,16 @@ static int btrfs_setsize(struct inode *inode, struct iattr *attr)
 		btrfs_drew_write_unlock(&root->snapshot_lock);
 		btrfs_end_transaction(trans);
 	} else {
+		struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
+
+		if (btrfs_is_zoned(fs_info)) {
+			ret = btrfs_wait_ordered_range(
+				inode,
+				ALIGN(newsize, fs_info->sectorsize),
+				(u64)-1);
+			if (ret)
+				return ret;
+		}
 
 		/*
 		 * We're truncating a file that used to have good data down to
Signature · 1 line
2.27.0
#89 lore raw

PATCH v9 30/41 btrfs: avoid async metadata checksum on ZONED mode

In reply to Naohiro Aota

In ZONED, btrfs uses per-FS zoned_meta_io_lock to serialize the metadata write IOs.

Even with these serialization, write bios sent from btree_write_cache_pages can be reordered by async checksum workers as these workers are per CPU and not per zone.

To preserve write BIO ordering, we can disable async metadata checksum on ZONED. This does not result in lower performance with HDDs as a single CPU core is fast enough to do checksum for a single zone write stream with the maximum possible bandwidth of the device. If multiple zones are being written simultaneously, HDD seek overhead lowers the achievable maximum bandwidth, resulting again in a per zone checksum serialization not affecting performance.

---
 fs/btrfs/disk-io.c | 2 ++
 1 file changed, 2 insertions(+)
fs/btrfs/disk-io.c +2 -0
index f02b121d8213..2b30ef8a7034 100644
@@ -813,6 +813,8 @@ static blk_status_t btree_submit_bio_start(void *private_data, struct bio *bio,
 static int check_async_write(struct btrfs_fs_info *fs_info,
 			     struct btrfs_inode *bi)
 {
+	if (btrfs_is_zoned(fs_info))
+		return 0;
 	if (atomic_read(&bi->sync_writers))
 		return 0;
 	if (test_bit(BTRFS_FS_CSUM_IMPL_FAST, &fs_info->flags))
Signature · 1 line
2.27.0
Reviewed-by on v9 30/41
#90 lore raw

Re: [PATCH v9 30/41] btrfs: avoid async metadata checksum on ZONED mode

Quoted 15 lines · Naohiro Aota

In ZONED, btrfs uses per-FS zoned_meta_io_lock to serialize the metadata write IOs.

Even with these serialization, write bios sent from btree_write_cache_pages can be reordered by async checksum workers as these workers are per CPU and not per zone.

To preserve write BIO ordering, we can disable async metadata checksum on ZONED. This does not result in lower performance with HDDs as a single CPU core is fast enough to do checksum for a single zone write stream with the maximum possible bandwidth of the device. If multiple zones are being written simultaneously, HDD seek overhead lowers the achievable maximum bandwidth, resulting again in a per zone checksum serialization not affecting performance.

Quoted 1 line

Thanks,

Josef

#91 lore raw

PATCH v9 31/41 btrfs: mark block groups to copy for device-replace

In reply to Naohiro Aota

This is the 1/4 patch to support device-replace in ZONED mode.

We have two types of I/Os during the device-replace process. One is an I/O to "copy" (by the scrub functions) all the device extents on the source device to the destination device. The other one is an I/O to "clone" (by handle_ops_on_dev_replace()) new incoming write I/Os from users to the source device into the target device.

Cloning incoming I/Os can break the sequential write rule in the target device. When writing is mapped in the middle of a block group, the I/O is directed in the middle of a target device zone, which breaks the sequential write rule.

However, the cloning function cannot be merely disabled since incoming I/Os targeting already copied device extents must be cloned so that the I/O is executed on the target device.

We cannot use dev_replace->cursor_{left,right} to determine whether bio is going to not yet copied region. Since we have a time gap between finishing btrfs_scrub_dev() and rewriting the mapping tree in btrfs_dev_replace_finishing(), we can have a newly allocated device extent which is never cloned nor copied.

So the point is to copy only already existing device extents. This patch introduces mark_block_group_to_copy() to mark existing block groups as a target of copying. Then, handle_ops_on_dev_replace() and dev-replace can check the flag to do their job.

---
 fs/btrfs/block-group.h |   1 +
 fs/btrfs/dev-replace.c | 175 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/dev-replace.h |   3 +
 fs/btrfs/scrub.c       |  17 ++++
 4 files changed, 196 insertions(+)
fs/btrfs/block-group.h +1 -0
index b2a8a3beceac..e91123495d68 100644
@@ -95,6 +95,7 @@ struct btrfs_block_group {
 	unsigned int iref:1;
 	unsigned int has_caching_ctl:1;
 	unsigned int removed:1;
+	unsigned int to_copy:1;
 
 	int disk_cache_state;
 
fs/btrfs/dev-replace.c +175 -0

193 lines not on this page · raw message

fs/btrfs/dev-replace.h +3 -0
index 60b70dacc299..3911049a5f23 100644
@@ -18,5 +18,8 @@ int btrfs_dev_replace_cancel(struct btrfs_fs_info *fs_info);
 void btrfs_dev_replace_suspend_for_unmount(struct btrfs_fs_info *fs_info);
 int btrfs_resume_dev_replace_async(struct btrfs_fs_info *fs_info);
 int __pure btrfs_dev_replace_is_ongoing(struct btrfs_dev_replace *dev_replace);
+bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev,
+				      struct btrfs_block_group *cache,
+				      u64 physical);
 
 #endif
fs/btrfs/scrub.c +17 -0
index aa1b36cf5c88..d0d7db3c8b0b 100644
@@ -3500,6 +3500,17 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 		if (!cache)
 			goto skip;
 
+
+		if (sctx->is_dev_replace && btrfs_fs_incompat(fs_info, ZONED)) {
+			spin_lock(&cache->lock);
+			if (!cache->to_copy) {
+				spin_unlock(&cache->lock);
+				ro_set = 0;
+				goto done;
+			}
+			spin_unlock(&cache->lock);
+		}
+
 		/*
 		 * Make sure that while we are scrubbing the corresponding block
 		 * group doesn't get its logical address and its device extents
@@ -3631,6 +3642,12 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 
 		scrub_pause_off(fs_info);
 
+		if (sctx->is_dev_replace &&
+		    !btrfs_finish_block_group_to_copy(dev_replace->srcdev,
+						      cache, found_key.offset))
+			ro_set = 0;
+
+done:
 		down_write(&dev_replace->rwsem);
 		dev_replace->cursor_left = dev_replace->cursor_right;
 		dev_replace->item_needs_writeback = 1;
Signature · 1 line
2.27.0
#92 lore raw

Re: [PATCH v9 31/41] btrfs: mark block groups to copy for device-replace

Quoted 72 lines · Naohiro Aota

This is the 1/4 patch to support device-replace in ZONED mode.

We have two types of I/Os during the device-replace process. One is an I/O to "copy" (by the scrub functions) all the device extents on the source device to the destination device. The other one is an I/O to "clone" (by handle_ops_on_dev_replace()) new incoming write I/Os from users to the source device into the target device.

Cloning incoming I/Os can break the sequential write rule in the target device. When writing is mapped in the middle of a block group, the I/O is directed in the middle of a target device zone, which breaks the sequential write rule.

However, the cloning function cannot be merely disabled since incoming I/Os targeting already copied device extents must be cloned so that the I/O is executed on the target device.

We cannot use dev_replace->cursor_{left,right} to determine whether bio is going to not yet copied region. Since we have a time gap between finishing btrfs_scrub_dev() and rewriting the mapping tree in btrfs_dev_replace_finishing(), we can have a newly allocated device extent which is never cloned nor copied.

So the point is to copy only already existing device extents. This patch introduces mark_block_group_to_copy() to mark existing block groups as a target of copying. Then, handle_ops_on_dev_replace() and dev-replace can check the flag to do their job.

---
  fs/btrfs/block-group.h |   1 +
  fs/btrfs/dev-replace.c | 175 +++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/dev-replace.h |   3 +
  fs/btrfs/scrub.c       |  17 ++++
  4 files changed, 196 insertions(+)
fs/btrfs/block-group.h +1 -0
index b2a8a3beceac..e91123495d68 100644
@@ -95,6 +95,7 @@ struct btrfs_block_group {
  	unsigned int iref:1;
  	unsigned int has_caching_ctl:1;
  	unsigned int removed:1;
+	unsigned int to_copy:1;
 
  	int disk_cache_state;
 
fs/btrfs/dev-replace.c +9 -0
index 5e3554482af1..e86aff38aea4 100644
@@ -22,6 +22,7 @@
  #include "dev-replace.h"
  #include "sysfs.h"
  #include "zoned.h"
+#include "block-group.h"
 
  /*
   * Device replace overview
@@ -437,6 +438,176 @@ static char* btrfs_dev_name(struct btrfs_device *device)
  		return rcu_str_deref(device->name);
  }
 
+static int mark_block_group_to_copy(struct btrfs_fs_info *fs_info,
+				    struct btrfs_device *src_dev)
+{
+	struct btrfs_path *path;
+	struct btrfs_key key;
+	struct btrfs_key found_key;
+	struct btrfs_root *root = fs_info->dev_root;
+	struct btrfs_dev_extent *dev_extent = NULL;

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 458

Quoted 10 lines
fs/btrfs/dev-replace.c +10 -0
+	struct btrfs_block_group *cache;
+	struct extent_buffer *l;
+	struct btrfs_trans_handle *trans;
+	int slot;
+	int ret = 0;
+	u64 chunk_offset, length;
+
+	/* Do not use "to_copy" on non-ZONED for now */
+	if (!btrfs_fs_incompat(fs_info, ZONED))
+		return 0;

if (!btrfs_is_zoned())

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 462

Quoted 4 lines
fs/btrfs/dev-replace.c +4 -0
+
+	mutex_lock(&fs_info->chunk_mutex);
+
+	/* ensulre we don't have pending new block group */

ensure.

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 464

Quoted 2 lines
fs/btrfs/dev-replace.c +2 -0
+	while (fs_info->running_transaction &&
+	       !list_empty(&fs_info->running_transaction->dev_update_list)) {

This is not safe to check, running_transaction can change without the trans_lock held.

Quoted 19 lines
fs/btrfs/dev-replace.c +19 -0
+		mutex_unlock(&fs_info->chunk_mutex);
+		trans = btrfs_attach_transaction(root);
+		if (IS_ERR(trans)) {
+			ret = PTR_ERR(trans);
+			mutex_lock(&fs_info->chunk_mutex);
+			if (ret == -ENOENT)
+				continue;
+			else
+				goto out;
+		}
+
+		ret = btrfs_commit_transaction(trans);
+		mutex_lock(&fs_info->chunk_mutex);
+		if (ret)
+			goto out;
+	}
+
+	path = btrfs_alloc_path();
+	if (!path) {

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 493

Quoted 10 lines
fs/btrfs/dev-replace.c +10 -0
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	path->reada = READA_FORWARD;
+	path->search_commit_root = 1;
+	path->skip_locking = 1;
+
+	key.objectid = src_dev->devid;
+	key.offset = 0ull;

0 is fine here right?

Quoted 43 lines
fs/btrfs/dev-replace.c +43 -0
+	key.type = BTRFS_DEV_EXTENT_KEY;
+
+	while (1) {
+		ret = btrfs_search_slot(NULL, root, &key, path, 0, 0);
+		if (ret < 0)
+			break;
+		if (ret > 0) {
+			if (path->slots[0] >=
+			    btrfs_header_nritems(path->nodes[0])) {
+				ret = btrfs_next_leaf(root, path);
+				if (ret < 0)
+					break;
+				if (ret > 0) {
+					ret = 0;
+					break;
+				}
+			} else {
+				ret = 0;
+			}
+		}
+
+		l = path->nodes[0];
+		slot = path->slots[0];
+
+		btrfs_item_key_to_cpu(l, &found_key, slot);
+
+		if (found_key.objectid != src_dev->devid)
+			break;
+
+		if (found_key.type != BTRFS_DEV_EXTENT_KEY)
+			break;
+
+		if (found_key.offset < key.offset)
+			break;
+
+		dev_extent = btrfs_item_ptr(l, slot, struct btrfs_dev_extent);
+		length = btrfs_dev_extent_length(l, dev_extent);
+
+		chunk_offset = btrfs_dev_extent_chunk_offset(l, dev_extent);
+
+		cache = btrfs_lookup_block_group(fs_info, chunk_offset);
+		if (!cache)
+			goto skip;

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 546

Quoted 10 lines
fs/btrfs/dev-replace.c +10 -0
+
+		spin_lock(&cache->lock);
+		cache->to_copy = 1;
+		spin_unlock(&cache->lock);
+
+		btrfs_put_block_group(cache);
+
+skip:
+		key.offset = found_key.offset + length;
+		btrfs_release_path(path);

Why are you releasing the path here?

This can be re-arranged into a loop that just uses btrfs_next_item().

Quoted 13 lines
fs/btrfs/dev-replace.c +13 -0
+	}
+
+	btrfs_free_path(path);
+out:
+	mutex_unlock(&fs_info->chunk_mutex);
+
+	return ret;
+}
+
+bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev,
+				      struct btrfs_block_group *cache,
+				      u64 physical)
+{

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 569

Quoted 10 lines
fs/btrfs/dev-replace.c +10 -0
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct extent_map *em;
+	struct map_lookup *map;
+	u64 chunk_offset = cache->start;
+	int num_extents, cur_extent;
+	int i;
+
+	/* Do not use "to_copy" on non-ZONED for now */
+	if (!btrfs_fs_incompat(fs_info, ZONED))
+		return true;

btrfs_is_zoned();

fs/btrfs/dev-replace.c static char* btrfs_dev_name(struct btrfs_device *device) · line 579

Quoted 10 lines
fs/btrfs/dev-replace.c +10 -0
+
+	spin_lock(&cache->lock);
+	if (cache->removed) {
+		spin_unlock(&cache->lock);
+		return true;
+	}
+	spin_unlock(&cache->lock);
+
+	em = btrfs_get_chunk_map(fs_info, chunk_offset, 1);
+	BUG_ON(IS_ERR(em));

ASSERT(!IS_ERR(em));

Quoted 63 lines
fs/btrfs/dev-replace.c +35 -0
+	map = em->map_lookup;
+
+	num_extents = cur_extent = 0;
+	for (i = 0; i < map->num_stripes; i++) {
+		/* we have more device extent to copy */
+		if (srcdev != map->stripes[i].dev)
+			continue;
+
+		num_extents++;
+		if (physical == map->stripes[i].physical)
+			cur_extent = i;
+	}
+
+	free_extent_map(em);
+
+	if (num_extents > 1 && cur_extent < num_extents - 1) {
+		/*
+		 * Has more stripes on this device. Keep this BG
+		 * readonly until we finish all the stripes.
+		 */
+		return false;
+	}
+
+	/* last stripe on this device */
+	spin_lock(&cache->lock);
+	cache->to_copy = 0;
+	spin_unlock(&cache->lock);
+
+	return true;
+}
+
  static int btrfs_dev_replace_start(struct btrfs_fs_info *fs_info,
  		const char *tgtdev_name, u64 srcdevid, const char *srcdev_name,
  		int read_src)
@@ -478,6 +649,10 @@ static int btrfs_dev_replace_start(struct btrfs_fs_info *fs_info,
  	if (ret)
  		return ret;
 
+	ret = mark_block_group_to_copy(fs_info, src_device);
+	if (ret)
+		return ret;
+
  	down_write(&dev_replace->rwsem);
  	switch (dev_replace->replace_state) {
  	case BTRFS_IOCTL_DEV_REPLACE_STATE_NEVER_STARTED:
fs/btrfs/dev-replace.h +3 -0
index 60b70dacc299..3911049a5f23 100644
@@ -18,5 +18,8 @@ int btrfs_dev_replace_cancel(struct btrfs_fs_info *fs_info);
  void btrfs_dev_replace_suspend_for_unmount(struct btrfs_fs_info *fs_info);
  int btrfs_resume_dev_replace_async(struct btrfs_fs_info *fs_info);
  int __pure btrfs_dev_replace_is_ongoing(struct btrfs_dev_replace *dev_replace);
+bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev,
+				      struct btrfs_block_group *cache,
+				      u64 physical);
 
  #endif
fs/btrfs/scrub.c +0 -0
index aa1b36cf5c88..d0d7db3c8b0b 100644

fs/btrfs/scrub.c int scrub_enumerate_chunks(struct scrub_ctx *sctx, · line 3504

Quoted 5 lines
fs/btrfs/scrub.c +2 -0
@@ -3500,6 +3500,17 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
  		if (!cache)
  			goto skip;
 
+
+		if (sctx->is_dev_replace && btrfs_fs_incompat(fs_info, ZONED)) {

btrfs_is_zoned(). Thanks,

Josef

#93 lore raw

PATCH v9 32/41 btrfs: implement cloning for ZONED device-replace

In reply to Naohiro Aota

This is 2/4 patch to implement device-replace for ZONED mode.

On zoned mode, a block group must be either copied (from the source device to the destination device) or cloned (to the both device).

This commit implements the cloning part. If a block group targeted by an IO is marked to copy, we should not clone the IO to the destination device, because the block group is eventually copied by the replace process.

This commit also handles cloning of device reset.

---
 fs/btrfs/dev-replace.c |  4 ++--
 fs/btrfs/extent-tree.c | 20 ++++++++++++++++++--
 fs/btrfs/scrub.c       |  2 +-
 fs/btrfs/volumes.c     | 33 +++++++++++++++++++++++++++++++--
 fs/btrfs/zoned.c       | 11 +++++++++++
 5 files changed, 63 insertions(+), 7 deletions(-)
fs/btrfs/dev-replace.c +2 -2
index e86aff38aea4..848f8063dc1c 100644
@@ -454,7 +454,7 @@ static int mark_block_group_to_copy(struct btrfs_fs_info *fs_info,
 	u64 chunk_offset, length;
 
 	/* Do not use "to_copy" on non-ZONED for now */
-	if (!btrfs_fs_incompat(fs_info, ZONED))
+	if (!btrfs_is_zoned(fs_info))
 		return 0;
 
 	mutex_lock(&fs_info->chunk_mutex);
@@ -565,7 +565,7 @@ bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev,
 	int i;
 
 	/* Do not use "to_copy" on non-ZONED for now */
-	if (!btrfs_fs_incompat(fs_info, ZONED))
+	if (!btrfs_is_zoned(fs_info))
 		return true;
 
 	spin_lock(&cache->lock);
fs/btrfs/extent-tree.c +18 -2
index 57454ef4c91e..1bb95d5aaed2 100644
@@ -35,6 +35,7 @@
 #include "discard.h"
 #include "rcu-string.h"
 #include "zoned.h"
+#include "dev-replace.h"
 
 #undef SCRAMBLE_DELAYED_REFS
 
@@ -1336,6 +1337,8 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 			u64 length = stripe->length;
 			u64 bytes;
 			struct request_queue *req_q;
+			struct btrfs_dev_replace *dev_replace =
+				&fs_info->dev_replace;
 
 			if (!stripe->dev->bdev) {
 				ASSERT(btrfs_test_opt(fs_info, DEGRADED));
@@ -1344,15 +1347,28 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 
 			req_q = bdev_get_queue(stripe->dev->bdev);
 			/* zone reset in ZONED mode */
-			if (btrfs_can_zone_reset(dev, physical, length))
+			if (btrfs_can_zone_reset(dev, physical, length)) {
 				ret = btrfs_reset_device_zone(dev, physical,
 							      length, &bytes);
-			else if (blk_queue_discard(req_q))
+				if (ret)
+					goto next;
+				if (!btrfs_dev_replace_is_ongoing(
+					    dev_replace) ||
+				    dev != dev_replace->srcdev)
+					goto next;
+
+				discarded_bytes += bytes;
+				/* send to replace target as well */
+				ret = btrfs_reset_device_zone(
+					dev_replace->tgtdev,
+					physical, length, &bytes);
+			} else if (blk_queue_discard(req_q))
 				ret = btrfs_issue_discard(dev->bdev, physical,
 							  length, &bytes);
 			else
 				continue;
 
+next:
 			if (!ret) {
 				discarded_bytes += bytes;
 			} else if (ret != -EOPNOTSUPP) {
fs/btrfs/scrub.c +1 -1
index d0d7db3c8b0b..371bb6437cab 100644
@@ -3501,7 +3501,7 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 			goto skip;
 
 
-		if (sctx->is_dev_replace && btrfs_fs_incompat(fs_info, ZONED)) {
+		if (sctx->is_dev_replace && btrfs_is_zoned(fs_info)) {
 			spin_lock(&cache->lock);
 			if (!cache->to_copy) {
 				spin_unlock(&cache->lock);
fs/btrfs/volumes.c +31 -2
index 26669007d367..920292d0fca7 100644
@@ -5964,9 +5964,29 @@ static int get_extra_mirror_from_replace(struct btrfs_fs_info *fs_info,
 	return ret;
 }
 
+static bool is_block_group_to_copy(struct btrfs_fs_info *fs_info, u64 logical)
+{
+	struct btrfs_block_group *cache;
+	bool ret;
+
+	/* non-ZONED mode does not use "to_copy" flag */
+	if (!btrfs_is_zoned(fs_info))
+		return false;
+
+	cache = btrfs_lookup_block_group(fs_info, logical);
+
+	spin_lock(&cache->lock);
+	ret = cache->to_copy;
+	spin_unlock(&cache->lock);
+
+	btrfs_put_block_group(cache);
+	return ret;
+}
+
 static void handle_ops_on_dev_replace(enum btrfs_map_op op,
 				      struct btrfs_bio **bbio_ret,
 				      struct btrfs_dev_replace *dev_replace,
+				      u64 logical,
 				      int *num_stripes_ret, int *max_errors_ret)
 {
 	struct btrfs_bio *bbio = *bbio_ret;
@@ -5979,6 +5999,15 @@ static void handle_ops_on_dev_replace(enum btrfs_map_op op,
 	if (op == BTRFS_MAP_WRITE) {
 		int index_where_to_add;
 
+		/*
+		 * a block group which have "to_copy" set will
+		 * eventually copied by dev-replace process. We can
+		 * avoid cloning IO here.
+		 */
+		if (is_block_group_to_copy(dev_replace->srcdev->fs_info,
+					   logical))
+			return;
+
 		/*
 		 * duplicate the write operations while the dev replace
 		 * procedure is running. Since the copying of the old disk to
@@ -6374,8 +6403,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info,
 
 	if (dev_replace_is_ongoing && dev_replace->tgtdev != NULL &&
 	    need_full_stripe(op)) {
-		handle_ops_on_dev_replace(op, &bbio, dev_replace, &num_stripes,
-					  &max_errors);
+		handle_ops_on_dev_replace(op, &bbio, dev_replace, logical,
+					  &num_stripes, &max_errors);
 	}
 
 	*bbio_ret = bbio;
fs/btrfs/zoned.c +11 -0
index 15bc7d451348..f672465d1bb1 100644
@@ -11,6 +11,7 @@
 #include "disk-io.h"
 #include "block-group.h"
 #include "transaction.h"
+#include "dev-replace.h"
 
 /* Maximum number of zones to report per blkdev_report_zones() call */
 #define BTRFS_REPORT_NR_ZONES   4096
@@ -890,6 +891,8 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 	for (i = 0; i < map->num_stripes; i++) {
 		bool is_sequential;
 		struct blk_zone zone;
+		struct btrfs_dev_replace *dev_replace = &fs_info->dev_replace;
+		int dev_replace_is_ongoing = 0;
 
 		device = map->stripes[i].dev;
 		physical = map->stripes[i].physical;
@@ -916,6 +919,14 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache)
 		 */
 		btrfs_dev_clear_zone_empty(device, physical);
 
+		down_read(&dev_replace->rwsem);
+		dev_replace_is_ongoing =
+			btrfs_dev_replace_is_ongoing(dev_replace);
+		if (dev_replace_is_ongoing && dev_replace->tgtdev != NULL)
+			btrfs_dev_clear_zone_empty(dev_replace->tgtdev,
+						   physical);
+		up_read(&dev_replace->rwsem);
+
 		/*
 		 * The group is mapped to a sequential zone. Get the zone write
 		 * pointer to determine the allocation offset within the zone.
Signature · 1 line
2.27.0
#94 lore raw

Re: [PATCH v9 32/41] btrfs: implement cloning for ZONED device-replace

Quoted 34 lines · Naohiro Aota

This is 2/4 patch to implement device-replace for ZONED mode.

On zoned mode, a block group must be either copied (from the source device to the destination device) or cloned (to the both device).

This commit implements the cloning part. If a block group targeted by an IO is marked to copy, we should not clone the IO to the destination device, because the block group is eventually copied by the replace process.

This commit also handles cloning of device reset.

---
  fs/btrfs/dev-replace.c |  4 ++--
  fs/btrfs/extent-tree.c | 20 ++++++++++++++++++--
  fs/btrfs/scrub.c       |  2 +-
  fs/btrfs/volumes.c     | 33 +++++++++++++++++++++++++++++++--
  fs/btrfs/zoned.c       | 11 +++++++++++
  5 files changed, 63 insertions(+), 7 deletions(-)
fs/btrfs/dev-replace.c +1 -1
index e86aff38aea4..848f8063dc1c 100644
@@ -454,7 +454,7 @@ static int mark_block_group_to_copy(struct btrfs_fs_info *fs_info,
  	u64 chunk_offset, length;
 
  	/* Do not use "to_copy" on non-ZONED for now */
-	if (!btrfs_fs_incompat(fs_info, ZONED))
+	if (!btrfs_is_zoned(fs_info))
  		return 0;
 
  	mutex_lock(&fs_info->chunk_mutex);

fs/btrfs/dev-replace.c bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev, · line 570

Quoted 7 lines
fs/btrfs/dev-replace.c +1 -1
@@ -565,7 +565,7 @@ bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev,
  	int i;
 
  	/* Do not use "to_copy" on non-ZONED for now */
-	if (!btrfs_fs_incompat(fs_info, ZONED))
+	if (!btrfs_is_zoned(fs_info))
  		return true;
 

Heh these

Quoted 34 lines
fs/btrfs/dev-replace.c +0 -0
  	spin_lock(&cache->lock);
fs/btrfs/extent-tree.c +7 -2
index 57454ef4c91e..1bb95d5aaed2 100644
@@ -35,6 +35,7 @@
  #include "discard.h"
  #include "rcu-string.h"
  #include "zoned.h"
+#include "dev-replace.h"
 
  #undef SCRAMBLE_DELAYED_REFS
 
@@ -1336,6 +1337,8 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
  			u64 length = stripe->length;
  			u64 bytes;
  			struct request_queue *req_q;
+			struct btrfs_dev_replace *dev_replace =
+				&fs_info->dev_replace;
 
  			if (!stripe->dev->bdev) {
  				ASSERT(btrfs_test_opt(fs_info, DEGRADED));
@@ -1344,15 +1347,28 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,
 
  			req_q = bdev_get_queue(stripe->dev->bdev);
  			/* zone reset in ZONED mode */
-			if (btrfs_can_zone_reset(dev, physical, length))
+			if (btrfs_can_zone_reset(dev, physical, length)) {
  				ret = btrfs_reset_device_zone(dev, physical,
  							      length, &bytes);
-			else if (blk_queue_discard(req_q))
+				if (ret)
+					goto next;
+				if (!btrfs_dev_replace_is_ongoing(

fs/btrfs/extent-tree.c int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, · line 1365

Quoted 10 lines
fs/btrfs/extent-tree.c +10 -0
+					    dev_replace) ||
+				    dev != dev_replace->srcdev)
+					goto next;
+
+				discarded_bytes += bytes;
+				/* send to replace target as well */
+				ret = btrfs_reset_device_zone(
+					dev_replace->tgtdev,
+					physical, length, &bytes);
+			} else if (blk_queue_discard(req_q))

And this needs to go into a helper, because lord it's not pretty. Thanks,

Josef

#95 lore raw

PATCH v9 33/41 btrfs: implement copying for ZONED device-replace

In reply to Naohiro Aota

This is 3/4 patch to implement device-replace on ZONED mode.

This commit implement copying. So, it track the write pointer during device replace process. Device-replace's copying is smart to copy only used extents on source device, we have to fill the gap to honor the sequential write rule in the target device.

Device-replace process in ZONED mode must copy or clone all the extents in the source device exactly once. So, we need to use to ensure allocations started just before the dev-replace process to have their corresponding extent information in the B-trees. finish_extent_writes_for_zoned() implements that functionality, which basically is the removed code in the commit 042528f8d840 ("Btrfs: fix block group remaining RO forever after error during device replace").

---
 fs/btrfs/scrub.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.c | 12 +++++++
 fs/btrfs/zoned.h |  7 ++++
 3 files changed, 105 insertions(+)
fs/btrfs/scrub.c +86 -0
index 371bb6437cab..aaf7882dee06 100644
@@ -169,6 +169,7 @@ struct scrub_ctx {
 	int			pages_per_rd_bio;
 
 	int			is_dev_replace;
+	u64			write_pointer;
 
 	struct scrub_bio        *wr_curr_bio;
 	struct mutex            wr_lock;
@@ -1623,6 +1624,25 @@ static int scrub_write_page_to_dev_replace(struct scrub_block *sblock,
 	return scrub_add_page_to_wr_bio(sblock->sctx, spage);
 }
 
+static int fill_writer_pointer_gap(struct scrub_ctx *sctx, u64 physical)
+{
+	int ret = 0;
+	u64 length;
+
+	if (!btrfs_is_zoned(sctx->fs_info))
+		return 0;
+
+	if (sctx->write_pointer < physical) {
+		length = physical - sctx->write_pointer;
+
+		ret = btrfs_zoned_issue_zeroout(sctx->wr_tgtdev,
+						sctx->write_pointer, length);
+		if (!ret)
+			sctx->write_pointer = physical;
+	}
+	return ret;
+}
+
 static int scrub_add_page_to_wr_bio(struct scrub_ctx *sctx,
 				    struct scrub_page *spage)
 {
@@ -1645,6 +1665,13 @@ static int scrub_add_page_to_wr_bio(struct scrub_ctx *sctx,
 	if (sbio->page_count == 0) {
 		struct bio *bio;
 
+		ret = fill_writer_pointer_gap(sctx,
+					      spage->physical_for_dev_replace);
+		if (ret) {
+			mutex_unlock(&sctx->wr_lock);
+			return ret;
+		}
+
 		sbio->physical = spage->physical_for_dev_replace;
 		sbio->logical = spage->logical;
 		sbio->dev = sctx->wr_tgtdev;
@@ -1706,6 +1733,10 @@ static void scrub_wr_submit(struct scrub_ctx *sctx)
 	 * doubled the write performance on spinning disks when measured
 	 * with Linux 3.5 */
 	btrfsic_submit_bio(sbio->bio);
+
+	if (btrfs_is_zoned(sctx->fs_info))
+		sctx->write_pointer = sbio->physical +
+			sbio->page_count * PAGE_SIZE;
 }
 
 static void scrub_wr_bio_end_io(struct bio *bio)
@@ -2973,6 +3004,21 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 	return ret < 0 ? ret : 0;
 }
 
+static void sync_replace_for_zoned(struct scrub_ctx *sctx)
+{
+	if (!btrfs_is_zoned(sctx->fs_info))
+		return;
+
+	sctx->flush_all_writes = true;
+	scrub_submit(sctx);
+	mutex_lock(&sctx->wr_lock);
+	scrub_wr_submit(sctx);
+	mutex_unlock(&sctx->wr_lock);
+
+	wait_event(sctx->list_wait,
+		   atomic_read(&sctx->bios_in_flight) == 0);
+}
+
 static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 					   struct map_lookup *map,
 					   struct btrfs_device *scrub_dev,
@@ -3105,6 +3151,14 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	 */
 	blk_start_plug(&plug);
 
+	if (sctx->is_dev_replace &&
+	    btrfs_dev_is_sequential(sctx->wr_tgtdev, physical)) {
+		mutex_lock(&sctx->wr_lock);
+		sctx->write_pointer = physical;
+		mutex_unlock(&sctx->wr_lock);
+		sctx->flush_all_writes = true;
+	}
+
 	/*
 	 * now find all extents for each stripe and scrub them
 	 */
@@ -3292,6 +3346,9 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 			if (ret)
 				goto out;
 
+			if (sctx->is_dev_replace)
+				sync_replace_for_zoned(sctx);
+
 			if (extent_logical + extent_len <
 			    key.objectid + bytes) {
 				if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
@@ -3414,6 +3471,25 @@ static noinline_for_stack int scrub_chunk(struct scrub_ctx *sctx,
 	return ret;
 }
 
+static int finish_extent_writes_for_zoned(struct btrfs_root *root,
+					  struct btrfs_block_group *cache)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct btrfs_trans_handle *trans;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	btrfs_wait_block_group_reservations(cache);
+	btrfs_wait_nocow_writers(cache);
+	btrfs_wait_ordered_roots(fs_info, U64_MAX, cache->start, cache->length);
+
+	trans = btrfs_join_transaction(root);
+	if (IS_ERR(trans))
+		return PTR_ERR(trans);
+	return btrfs_commit_transaction(trans);
+}
+
 static noinline_for_stack
 int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 			   struct btrfs_device *scrub_dev, u64 start, u64 end)
@@ -3569,6 +3645,16 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 		 * group is not RO.
 		 */
 		ret = btrfs_inc_block_group_ro(cache, sctx->is_dev_replace);
+		if (!ret && sctx->is_dev_replace) {
+			ret = finish_extent_writes_for_zoned(root, cache);
+			if (ret) {
+				btrfs_dec_block_group_ro(cache);
+				scrub_pause_off(fs_info);
+				btrfs_put_block_group(cache);
+				break;
+			}
+		}
+
 		if (ret == 0) {
 			ro_set = 1;
 		} else if (ret == -ENOSPC && !sctx->is_dev_replace) {
fs/btrfs/zoned.c +12 -0
index f672465d1bb1..1b080184440d 100644
@@ -1181,3 +1181,15 @@ void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
 	ASSERT(cache->meta_write_pointer == eb->start + eb->len);
 	cache->meta_write_pointer = eb->start;
 }
+
+int btrfs_zoned_issue_zeroout(struct btrfs_device *device, u64 physical,
+			      u64 length)
+{
+	if (!btrfs_dev_is_sequential(device, physical))
+		return -EOPNOTSUPP;
+
+	return blkdev_issue_zeroout(device->bdev,
+				    physical >> SECTOR_SHIFT,
+				    length >> SECTOR_SHIFT,
+				    GFP_NOFS, 0);
+}
fs/btrfs/zoned.h +7 -0
index 262b248a9085..fb4f1cfce1e5 100644
@@ -52,6 +52,8 @@ bool btrfs_check_meta_write_pointer(struct btrfs_fs_info *fs_info,
 				    struct btrfs_block_group **cache_ret);
 void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
 				     struct extent_buffer *eb);
+int btrfs_zoned_issue_zeroout(struct btrfs_device *device, u64 physical,
+			      u64 length);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -139,6 +141,11 @@ btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
 				struct extent_buffer *eb)
 {
 }
+static inline int btrfs_zoned_issue_zeroout(struct btrfs_device *device,
+					    u64 physical, u64 length)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
#96 lore raw

Re: [PATCH v9 33/41] btrfs: implement copying for ZONED device-replace

Quoted 180 lines · Naohiro Aota

This is 3/4 patch to implement device-replace on ZONED mode.

This commit implement copying. So, it track the write pointer during device replace process. Device-replace's copying is smart to copy only used extents on source device, we have to fill the gap to honor the sequential write rule in the target device.

Device-replace process in ZONED mode must copy or clone all the extents in the source device exactly once. So, we need to use to ensure allocations started just before the dev-replace process to have their corresponding extent information in the B-trees. finish_extent_writes_for_zoned() implements that functionality, which basically is the removed code in the commit 042528f8d840 ("Btrfs: fix block group remaining RO forever after error during device replace").

---
  fs/btrfs/scrub.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/zoned.c | 12 +++++++
  fs/btrfs/zoned.h |  7 ++++
  3 files changed, 105 insertions(+)
fs/btrfs/scrub.c +86 -0
index 371bb6437cab..aaf7882dee06 100644
@@ -169,6 +169,7 @@ struct scrub_ctx {
  	int			pages_per_rd_bio;
 
  	int			is_dev_replace;
+	u64			write_pointer;
 
  	struct scrub_bio        *wr_curr_bio;
  	struct mutex            wr_lock;
@@ -1623,6 +1624,25 @@ static int scrub_write_page_to_dev_replace(struct scrub_block *sblock,
  	return scrub_add_page_to_wr_bio(sblock->sctx, spage);
  }
 
+static int fill_writer_pointer_gap(struct scrub_ctx *sctx, u64 physical)
+{
+	int ret = 0;
+	u64 length;
+
+	if (!btrfs_is_zoned(sctx->fs_info))
+		return 0;
+
+	if (sctx->write_pointer < physical) {
+		length = physical - sctx->write_pointer;
+
+		ret = btrfs_zoned_issue_zeroout(sctx->wr_tgtdev,
+						sctx->write_pointer, length);
+		if (!ret)
+			sctx->write_pointer = physical;
+	}
+	return ret;
+}
+
  static int scrub_add_page_to_wr_bio(struct scrub_ctx *sctx,
  				    struct scrub_page *spage)
  {
@@ -1645,6 +1665,13 @@ static int scrub_add_page_to_wr_bio(struct scrub_ctx *sctx,
  	if (sbio->page_count == 0) {
  		struct bio *bio;
 
+		ret = fill_writer_pointer_gap(sctx,
+					      spage->physical_for_dev_replace);
+		if (ret) {
+			mutex_unlock(&sctx->wr_lock);
+			return ret;
+		}
+
  		sbio->physical = spage->physical_for_dev_replace;
  		sbio->logical = spage->logical;
  		sbio->dev = sctx->wr_tgtdev;
@@ -1706,6 +1733,10 @@ static void scrub_wr_submit(struct scrub_ctx *sctx)
  	 * doubled the write performance on spinning disks when measured
  	 * with Linux 3.5 */
  	btrfsic_submit_bio(sbio->bio);
+
+	if (btrfs_is_zoned(sctx->fs_info))
+		sctx->write_pointer = sbio->physical +
+			sbio->page_count * PAGE_SIZE;
  }
 
  static void scrub_wr_bio_end_io(struct bio *bio)
@@ -2973,6 +3004,21 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
  	return ret < 0 ? ret : 0;
  }
 
+static void sync_replace_for_zoned(struct scrub_ctx *sctx)
+{
+	if (!btrfs_is_zoned(sctx->fs_info))
+		return;
+
+	sctx->flush_all_writes = true;
+	scrub_submit(sctx);
+	mutex_lock(&sctx->wr_lock);
+	scrub_wr_submit(sctx);
+	mutex_unlock(&sctx->wr_lock);
+
+	wait_event(sctx->list_wait,
+		   atomic_read(&sctx->bios_in_flight) == 0);
+}
+
  static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
  					   struct map_lookup *map,
  					   struct btrfs_device *scrub_dev,
@@ -3105,6 +3151,14 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
  	 */
  	blk_start_plug(&plug);
 
+	if (sctx->is_dev_replace &&
+	    btrfs_dev_is_sequential(sctx->wr_tgtdev, physical)) {
+		mutex_lock(&sctx->wr_lock);
+		sctx->write_pointer = physical;
+		mutex_unlock(&sctx->wr_lock);
+		sctx->flush_all_writes = true;
+	}
+
  	/*
  	 * now find all extents for each stripe and scrub them
  	 */
@@ -3292,6 +3346,9 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
  			if (ret)
  				goto out;
 
+			if (sctx->is_dev_replace)
+				sync_replace_for_zoned(sctx);
+
  			if (extent_logical + extent_len <
  			    key.objectid + bytes) {
  				if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
@@ -3414,6 +3471,25 @@ static noinline_for_stack int scrub_chunk(struct scrub_ctx *sctx,
  	return ret;
  }
 
+static int finish_extent_writes_for_zoned(struct btrfs_root *root,
+					  struct btrfs_block_group *cache)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct btrfs_trans_handle *trans;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	btrfs_wait_block_group_reservations(cache);
+	btrfs_wait_nocow_writers(cache);
+	btrfs_wait_ordered_roots(fs_info, U64_MAX, cache->start, cache->length);
+
+	trans = btrfs_join_transaction(root);
+	if (IS_ERR(trans))
+		return PTR_ERR(trans);
+	return btrfs_commit_transaction(trans);
+}
+
  static noinline_for_stack
  int scrub_enumerate_chunks(struct scrub_ctx *sctx,
  			   struct btrfs_device *scrub_dev, u64 start, u64 end)
@@ -3569,6 +3645,16 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
  		 * group is not RO.
  		 */
  		ret = btrfs_inc_block_group_ro(cache, sctx->is_dev_replace);
+		if (!ret && sctx->is_dev_replace) {
+			ret = finish_extent_writes_for_zoned(root, cache);
+			if (ret) {
+				btrfs_dec_block_group_ro(cache);
+				scrub_pause_off(fs_info);
+				btrfs_put_block_group(cache);
+				break;
+			}
+		}
+
  		if (ret == 0) {
  			ro_set = 1;
  		} else if (ret == -ENOSPC && !sctx->is_dev_replace) {
fs/btrfs/zoned.c +0 -0
index f672465d1bb1..1b080184440d 100644

fs/btrfs/zoned.c void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache, · line 1190

Quoted 10 lines
fs/btrfs/zoned.c +7 -0
@@ -1181,3 +1181,15 @@ void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
  	ASSERT(cache->meta_write_pointer == eb->start + eb->len);
  	cache->meta_write_pointer = eb->start;
  }
+
+int btrfs_zoned_issue_zeroout(struct btrfs_device *device, u64 physical,
+			      u64 length)
+{
+	if (!btrfs_dev_is_sequential(device, physical))
+		return -EOPNOTSUPP;
+

This is going to result in a bunch of scrub errors. Is this unlikely to happen?
And if it isn't, should we do something different? If it's not sequential
then we can probably just write to whatever offset we want right? So do we need an error here at all? Thanks,

Josef

#97 lore raw

Re: [PATCH v9 33/41] btrfs: implement copying for ZONED device-replace

Quoted 185 lines · Josef Bacik
Quoted 171 lines · Naohiro Aota

This is 3/4 patch to implement device-replace on ZONED mode.

This commit implement copying. So, it track the write pointer during device replace process. Device-replace's copying is smart to copy only used extents on source device, we have to fill the gap to honor the sequential write rule in the target device.

Device-replace process in ZONED mode must copy or clone all the extents in the source device exactly once. So, we need to use to ensure allocations started just before the dev-replace process to have their corresponding extent information in the B-trees. finish_extent_writes_for_zoned() implements that functionality, which basically is the removed code in the commit 042528f8d840 ("Btrfs: fix block group remaining RO forever after error during device replace").

---
 fs/btrfs/scrub.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.c | 12 +++++++
 fs/btrfs/zoned.h |  7 ++++
 3 files changed, 105 insertions(+)
fs/btrfs/scrub.c +86 -0
index 371bb6437cab..aaf7882dee06 100644
@@ -169,6 +169,7 @@ struct scrub_ctx {
 	int			pages_per_rd_bio;
 	int			is_dev_replace;
+	u64			write_pointer;
 	struct scrub_bio        *wr_curr_bio;
 	struct mutex            wr_lock;
@@ -1623,6 +1624,25 @@ static int scrub_write_page_to_dev_replace(struct scrub_block *sblock,
 	return scrub_add_page_to_wr_bio(sblock->sctx, spage);
 }
+static int fill_writer_pointer_gap(struct scrub_ctx *sctx, u64 physical)
+{
+	int ret = 0;
+	u64 length;
+
+	if (!btrfs_is_zoned(sctx->fs_info))
+		return 0;
+
+	if (sctx->write_pointer < physical) {
+		length = physical - sctx->write_pointer;
+
+		ret = btrfs_zoned_issue_zeroout(sctx->wr_tgtdev,
+						sctx->write_pointer, length);
+		if (!ret)
+			sctx->write_pointer = physical;
+	}
+	return ret;
+}
+
 static int scrub_add_page_to_wr_bio(struct scrub_ctx *sctx,
 				    struct scrub_page *spage)
 {
@@ -1645,6 +1665,13 @@ static int scrub_add_page_to_wr_bio(struct scrub_ctx *sctx,
 	if (sbio->page_count == 0) {
 		struct bio *bio;
+		ret = fill_writer_pointer_gap(sctx,
+					      spage->physical_for_dev_replace);
+		if (ret) {
+			mutex_unlock(&sctx->wr_lock);
+			return ret;
+		}
+
 		sbio->physical = spage->physical_for_dev_replace;
 		sbio->logical = spage->logical;
 		sbio->dev = sctx->wr_tgtdev;
@@ -1706,6 +1733,10 @@ static void scrub_wr_submit(struct scrub_ctx *sctx)
 	 * doubled the write performance on spinning disks when measured
 	 * with Linux 3.5 */
 	btrfsic_submit_bio(sbio->bio);
+
+	if (btrfs_is_zoned(sctx->fs_info))
+		sctx->write_pointer = sbio->physical +
+			sbio->page_count * PAGE_SIZE;
 }
 static void scrub_wr_bio_end_io(struct bio *bio)
@@ -2973,6 +3004,21 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 	return ret < 0 ? ret : 0;
 }
+static void sync_replace_for_zoned(struct scrub_ctx *sctx)
+{
+	if (!btrfs_is_zoned(sctx->fs_info))
+		return;
+
+	sctx->flush_all_writes = true;
+	scrub_submit(sctx);
+	mutex_lock(&sctx->wr_lock);
+	scrub_wr_submit(sctx);
+	mutex_unlock(&sctx->wr_lock);
+
+	wait_event(sctx->list_wait,
+		   atomic_read(&sctx->bios_in_flight) == 0);
+}
+
 static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 					   struct map_lookup *map,
 					   struct btrfs_device *scrub_dev,
@@ -3105,6 +3151,14 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	 */
 	blk_start_plug(&plug);
+	if (sctx->is_dev_replace &&
+	    btrfs_dev_is_sequential(sctx->wr_tgtdev, physical)) {
+		mutex_lock(&sctx->wr_lock);
+		sctx->write_pointer = physical;
+		mutex_unlock(&sctx->wr_lock);
+		sctx->flush_all_writes = true;
+	}
+
 	/*
 	 * now find all extents for each stripe and scrub them
 	 */
@@ -3292,6 +3346,9 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 			if (ret)
 				goto out;
+			if (sctx->is_dev_replace)
+				sync_replace_for_zoned(sctx);
+
 			if (extent_logical + extent_len <
 			    key.objectid + bytes) {
 				if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
@@ -3414,6 +3471,25 @@ static noinline_for_stack int scrub_chunk(struct scrub_ctx *sctx,
 	return ret;
 }
+static int finish_extent_writes_for_zoned(struct btrfs_root *root,
+					  struct btrfs_block_group *cache)
+{
+	struct btrfs_fs_info *fs_info = cache->fs_info;
+	struct btrfs_trans_handle *trans;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	btrfs_wait_block_group_reservations(cache);
+	btrfs_wait_nocow_writers(cache);
+	btrfs_wait_ordered_roots(fs_info, U64_MAX, cache->start, cache->length);
+
+	trans = btrfs_join_transaction(root);
+	if (IS_ERR(trans))
+		return PTR_ERR(trans);
+	return btrfs_commit_transaction(trans);
+}
+
 static noinline_for_stack
 int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 			   struct btrfs_device *scrub_dev, u64 start, u64 end)
@@ -3569,6 +3645,16 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
 		 * group is not RO.
 		 */
 		ret = btrfs_inc_block_group_ro(cache, sctx->is_dev_replace);
+		if (!ret && sctx->is_dev_replace) {
+			ret = finish_extent_writes_for_zoned(root, cache);
+			if (ret) {
+				btrfs_dec_block_group_ro(cache);
+				scrub_pause_off(fs_info);
+				btrfs_put_block_group(cache);
+				break;
+			}
+		}
+
 		if (ret == 0) {
 			ro_set = 1;
 		} else if (ret == -ENOSPC && !sctx->is_dev_replace) {
fs/btrfs/zoned.c +0 -0
index f672465d1bb1..1b080184440d 100644

fs/btrfs/zoned.c void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache, · line 1190

Quoted 10 lines
fs/btrfs/zoned.c +7 -0
@@ -1181,3 +1181,15 @@ void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
 	ASSERT(cache->meta_write_pointer == eb->start + eb->len);
 	cache->meta_write_pointer = eb->start;
 }
+
+int btrfs_zoned_issue_zeroout(struct btrfs_device *device, u64 physical,
+			      u64 length)
+{
+	if (!btrfs_dev_is_sequential(device, physical))
+		return -EOPNOTSUPP;
+

This is going to result in a bunch of scrub errors. Is this unlikely to happen? And if it isn't, should we do something different? If it's not sequential then we can probably just write to whatever offset we want right? So do we need an error here at all? Thanks,

Quoted 1 line

Josef

I'm intended to use this to synchronize the write pointer. For conventional zones, we don't need to synchronize it anyway. We can just skip the unwritten area in the source device and continue the writing at the different location.

#98 lore raw

PATCH v9 34/41 btrfs: support dev-replace in ZONED mode

In reply to Naohiro Aota

This is 4/4 patch to implement device-replace on ZONED mode.

Even after the copying is done, the write pointers of the source device and the destination device may not be synchronized. For example, when the last allocated extent is freed before device-replace process, the extent is not copied, leaving a hole there.

This patch synchronize the write pointers by writing zeros to the destination device.

---
 fs/btrfs/scrub.c | 36 +++++++++++++++++++++++++
 fs/btrfs/zoned.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/zoned.h |  8 ++++++
 3 files changed, 113 insertions(+)
fs/btrfs/scrub.c +36 -0
index aaf7882dee06..0e2211b9c810 100644
@@ -3019,6 +3019,31 @@ static void sync_replace_for_zoned(struct scrub_ctx *sctx)
 		   atomic_read(&sctx->bios_in_flight) == 0);
 }
 
+static int sync_write_pointer_for_zoned(struct scrub_ctx *sctx, u64 logical,
+					u64 physical, u64 physical_end)
+{
+	struct btrfs_fs_info *fs_info = sctx->fs_info;
+	int ret = 0;
+
+	if (!btrfs_is_zoned(fs_info))
+		return 0;
+
+	wait_event(sctx->list_wait, atomic_read(&sctx->bios_in_flight) == 0);
+
+	mutex_lock(&sctx->wr_lock);
+	if (sctx->write_pointer < physical_end) {
+		ret = btrfs_sync_zone_write_pointer(sctx->wr_tgtdev, logical,
+						    physical,
+						    sctx->write_pointer);
+		if (ret)
+			btrfs_err(fs_info, "failed to recover write pointer");
+	}
+	mutex_unlock(&sctx->wr_lock);
+	btrfs_dev_clear_zone_empty(sctx->wr_tgtdev, physical);
+
+	return ret;
+}
+
 static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 					   struct map_lookup *map,
 					   struct btrfs_device *scrub_dev,
@@ -3416,6 +3441,17 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	blk_finish_plug(&plug);
 	btrfs_free_path(path);
 	btrfs_free_path(ppath);
+
+	if (sctx->is_dev_replace && ret >= 0) {
+		int ret2;
+
+		ret2 = sync_write_pointer_for_zoned(sctx, base + offset,
+						    map->stripes[num].physical,
+						    physical_end);
+		if (ret2)
+			ret = ret2;
+	}
+
 	return ret < 0 ? ret : 0;
 }
 
fs/btrfs/zoned.c +69 -0

78 lines not on this page · raw message

fs/btrfs/zoned.h +8 -0
index fb4f1cfce1e5..bd73f3c48c0c 100644
@@ -54,6 +54,8 @@ void btrfs_revert_meta_write_pointer(struct btrfs_block_group *cache,
 				     struct extent_buffer *eb);
 int btrfs_zoned_issue_zeroout(struct btrfs_device *device, u64 physical,
 			      u64 length);
+int btrfs_sync_zone_write_pointer(struct btrfs_device *tgt_dev, u64 logical,
+				  u64 physical_start, u64 physical_pos);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
 				     struct blk_zone *zone)
@@ -146,6 +148,12 @@ static inline int btrfs_zoned_issue_zeroout(struct btrfs_device *device,
 {
 	return -EOPNOTSUPP;
 }
+static inline int btrfs_sync_zone_write_pointer(struct btrfs_device *tgt_dev,
+						u64 logical, u64 physical_start,
+						u64 physical_pos)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
Signature · 1 line
2.27.0
Reviewed-by on v9 34/41
#99 lore raw

Re: [PATCH v9 34/41] btrfs: support dev-replace in ZONED mode

Quoted 11 lines · Naohiro Aota

This is 4/4 patch to implement device-replace on ZONED mode.

Even after the copying is done, the write pointers of the source device and the destination device may not be synchronized. For example, when the last allocated extent is freed before device-replace process, the extent is not copied, leaving a hole there.

This patch synchronize the write pointers by writing zeros to the destination device.

Thanks,

Josef

#100 lore raw

PATCH v9 35/41 btrfs: enable relocation in ZONED mode

In reply to Naohiro Aota

To serialize allocation and submit_bio, we introduced mutex around them. As a result, preallocation must be completely disabled to avoid a deadlock.

Since current relocation process relies on preallocation to move file data extents, it must be handled in another way. In ZONED mode, we just truncate the inode to the size that we wanted to pre-allocate. Then, we flush dirty pages on the file before finishing relocation process. run_delalloc_zoned() will handle all the allocation and submit IOs to the underlying layers.

---
 fs/btrfs/relocation.c | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)
fs/btrfs/relocation.c +33 -2
index 3602806d71bd..44b697b881b6 100644
@@ -2603,6 +2603,32 @@ static noinline_for_stack int prealloc_file_extent_cluster(
 	if (ret)
 		return ret;
 
+	/*
+	 * In ZONED mode, we cannot preallocate the file region. Instead, we
+	 * dirty and fiemap_write the region.
+	 */
+
+	if (btrfs_is_zoned(inode->root->fs_info)) {
+		struct btrfs_root *root = inode->root;
+		struct btrfs_trans_handle *trans;
+
+		end = cluster->end - offset + 1;
+		trans = btrfs_start_transaction(root, 1);
+		if (IS_ERR(trans))
+			return PTR_ERR(trans);
+
+		inode->vfs_inode.i_ctime = current_time(&inode->vfs_inode);
+		i_size_write(&inode->vfs_inode, end);
+		ret = btrfs_update_inode(trans, root, &inode->vfs_inode);
+		if (ret) {
+			btrfs_abort_transaction(trans, ret);
+			btrfs_end_transaction(trans);
+			return ret;
+		}
+
+		return btrfs_end_transaction(trans);
+	}
+
 	inode_lock(&inode->vfs_inode);
 	for (nr = 0; nr < cluster->nr; nr++) {
 		start = cluster->boundary[nr] - offset;
@@ -2799,6 +2825,8 @@ static int relocate_file_extent_cluster(struct inode *inode,
 		}
 	}
 	WARN_ON(nr != cluster->nr);
+	if (btrfs_is_zoned(fs_info) && !ret)
+		ret = btrfs_wait_ordered_range(inode, 0, (u64)-1);
 out:
 	kfree(ra);
 	return ret;
@@ -3434,8 +3462,12 @@ static int __insert_orphan_inode(struct btrfs_trans_handle *trans,
 	struct btrfs_path *path;
 	struct btrfs_inode_item *item;
 	struct extent_buffer *leaf;
+	u64 flags = BTRFS_INODE_NOCOMPRESS | BTRFS_INODE_PREALLOC;
 	int ret;
 
+	if (btrfs_is_zoned(trans->fs_info))
+		flags &= ~BTRFS_INODE_PREALLOC;
+
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
@@ -3450,8 +3482,7 @@ static int __insert_orphan_inode(struct btrfs_trans_handle *trans,
 	btrfs_set_inode_generation(leaf, item, 1);
 	btrfs_set_inode_size(leaf, item, 0);
 	btrfs_set_inode_mode(leaf, item, S_IFREG | 0600);
-	btrfs_set_inode_flags(leaf, item, BTRFS_INODE_NOCOMPRESS |
-					  BTRFS_INODE_PREALLOC);
+	btrfs_set_inode_flags(leaf, item, flags);
 	btrfs_mark_buffer_dirty(leaf);
 out:
 	btrfs_free_path(path);
Signature · 1 line
2.27.0
#101 lore raw

Re: [PATCH v9 35/41] btrfs: enable relocation in ZONED mode

Quoted 38 lines · Naohiro Aota

To serialize allocation and submit_bio, we introduced mutex around them. As a result, preallocation must be completely disabled to avoid a deadlock.

Since current relocation process relies on preallocation to move file data extents, it must be handled in another way. In ZONED mode, we just truncate the inode to the size that we wanted to pre-allocate. Then, we flush dirty pages on the file before finishing relocation process. run_delalloc_zoned() will handle all the allocation and submit IOs to the underlying layers.

---
  fs/btrfs/relocation.c | 35 +++++++++++++++++++++++++++++++++--
  1 file changed, 33 insertions(+), 2 deletions(-)
fs/btrfs/relocation.c +15 -0
index 3602806d71bd..44b697b881b6 100644
@@ -2603,6 +2603,32 @@ static noinline_for_stack int prealloc_file_extent_cluster(
  	if (ret)
  		return ret;
 
+	/*
+	 * In ZONED mode, we cannot preallocate the file region. Instead, we
+	 * dirty and fiemap_write the region.
+	 */
+
+	if (btrfs_is_zoned(inode->root->fs_info)) {
+		struct btrfs_root *root = inode->root;
+		struct btrfs_trans_handle *trans;
+
+		end = cluster->end - offset + 1;
+		trans = btrfs_start_transaction(root, 1);
+		if (IS_ERR(trans))
+			return PTR_ERR(trans);
+
+		inode->vfs_inode.i_ctime = current_time(&inode->vfs_inode);

fs/btrfs/relocation.c static noinline_for_stack int prealloc_file_extent_cluster( · line 2630

Quoted 10 lines
fs/btrfs/relocation.c +10 -0
+		i_size_write(&inode->vfs_inode, end);
+		ret = btrfs_update_inode(trans, root, &inode->vfs_inode);
+		if (ret) {
+			btrfs_abort_transaction(trans, ret);
+			btrfs_end_transaction(trans);
+			return ret;
+		}
+
+		return btrfs_end_transaction(trans);
+	}

The reason we pre-allocate is because we need the extents to be exactly how long they are on disk. If we have a 123mib extent, we need the extent it goes into to be exactly 123mib, otherwise we cannot simply relocate out its location. This doesn't do anything for us. I'm not even sure we can do relocation for zoned since we have to maintain the original extent size, but if you can then it's going to be more complicated than just this. You'll have to either forgo WRITE_APPEND for this case, or make sure you can actually write the whole ordered extent in one go. Thanks,

Josef

#102 lore raw

PATCH v9 36/41 btrfs: relocate block group to repair IO failure in ZONED

In reply to Naohiro Aota

When btrfs find a checksum error and if the file system has a mirror of the damaged data, btrfs read the correct data from the mirror and write the data to damaged blocks. This repairing, however, is against the sequential write required rule.

We can consider three methods to repair an IO failure in ZONED mode:
(1) Reset and rewrite the damaged zone
(2) Allocate new device extent and replace the damaged device extent to the new extent
(3) Relocate the corresponding block group

Method (1) is most similar to a behavior done with regular devices. However, it also wipes non-damaged data in the same device extent, and so it unnecessary degrades non-damaged data.

Method (2) is much like device replacing but done in the same device. It is safe because it keeps the device extent until the replacing finish. However, extending device replacing is non-trivial. It assumes "src_dev->physical == dst_dev->physical". Also, the extent mapping replacing function should be extended to support replacing device extent position in one device.

Method (3) invokes relocation of the damaged block group, so it is straightforward to implement. It relocates all the mirrored device extents, so it is, potentially, a more costly operation than method (1) or (2). But it relocates only using extents which reduce the total IO size.

Let's apply method (3) for now. In the future, we can extend device-replace and apply method (2).

For protecting a block group gets relocated multiple time with multiple IO errors, this commit introduces "relocating_repair" bit to show it's now relocating to repair IO failures. Also it uses a new kthread "btrfs-relocating-repair", not to block IO path with relocating process.

This commit also supports repairing in the scrub process.

---
 fs/btrfs/block-group.h |  1 +
 fs/btrfs/extent_io.c   |  3 ++
 fs/btrfs/scrub.c       |  3 ++
 fs/btrfs/volumes.c     | 71 ++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h     |  1 +
 5 files changed, 79 insertions(+)
fs/btrfs/block-group.h +1 -0
index e91123495d68..50e5ddb0a19b 100644
@@ -96,6 +96,7 @@ struct btrfs_block_group {
 	unsigned int has_caching_ctl:1;
 	unsigned int removed:1;
 	unsigned int to_copy:1;
+	unsigned int relocating_repair:1;
 
 	int disk_cache_state;
 
fs/btrfs/extent_io.c +3 -0
index 3cce444d5dbb..8ab5161a68b4 100644
@@ -2268,6 +2268,9 @@ int repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start,
 	ASSERT(!(fs_info->sb->s_flags & SB_RDONLY));
 	BUG_ON(!mirror_num);
 
+	if (btrfs_is_zoned(fs_info))
+		return btrfs_repair_one_zone(fs_info, logical);
+
 	bio = btrfs_io_bio_alloc(1);
 	bio->bi_iter.bi_size = 0;
 	map_length = length;
fs/btrfs/scrub.c +3 -0
index 0e2211b9c810..e6a8df8a8f4f 100644
@@ -861,6 +861,9 @@ static int scrub_handle_errored_block(struct scrub_block *sblock_to_check)
 	have_csum = sblock_to_check->pagev[0]->have_csum;
 	dev = sblock_to_check->pagev[0]->dev;
 
+	if (btrfs_is_zoned(fs_info) && !sctx->is_dev_replace)
+		return btrfs_repair_one_zone(fs_info, logical);
+
 	/*
 	 * We must use GFP_NOFS because the scrub task might be waiting for a
 	 * worker task executing this function and in turn a transaction commit
fs/btrfs/volumes.c +71 -0

74 lines not on this page · raw message

fs/btrfs/volumes.h +1 -0
index cff1f7689eac..7c1ad6901791 100644
@@ -584,5 +584,6 @@ void btrfs_scratch_superblocks(struct btrfs_fs_info *fs_info,
 int btrfs_bg_type_to_factor(u64 flags);
 const char *btrfs_bg_type_to_raid_name(u64 flags);
 int btrfs_verify_dev_extents(struct btrfs_fs_info *fs_info);
+int btrfs_repair_one_zone(struct btrfs_fs_info *fs_info, u64 logical);
 
 #endif
Signature · 1 line
2.27.0
#103 lore raw

PATCH v9 37/41 btrfs: split alloc_log_tree()

In reply to Naohiro Aota

This is a preparation for the next patch. This commit split alloc_log_tree() to allocating tree structure part (remains in alloc_log_tree()) and allocating tree node part (moved in btrfs_alloc_log_tree_node()). The latter part is also exported to be used in the next patch.

---
 fs/btrfs/disk-io.c | 31 +++++++++++++++++++++++++------
 fs/btrfs/disk-io.h |  2 ++
 2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/disk-io.c +25 -6
index 2b30ef8a7034..70885f3d3321 100644
@@ -1211,7 +1211,6 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
 					 struct btrfs_fs_info *fs_info)
 {
 	struct btrfs_root *root;
-	struct extent_buffer *leaf;
 
 	root = btrfs_alloc_root(fs_info, BTRFS_TREE_LOG_OBJECTID, GFP_NOFS);
 	if (!root)
@@ -1221,6 +1220,14 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
 	root->root_key.type = BTRFS_ROOT_ITEM_KEY;
 	root->root_key.offset = BTRFS_TREE_LOG_OBJECTID;
 
+	return root;
+}
+
+int btrfs_alloc_log_tree_node(struct btrfs_trans_handle *trans,
+			      struct btrfs_root *root)
+{
+	struct extent_buffer *leaf;
+
 	/*
 	 * DON'T set SHAREABLE bit for log trees.
 	 *
@@ -1233,26 +1240,31 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
 
 	leaf = btrfs_alloc_tree_block(trans, root, 0, BTRFS_TREE_LOG_OBJECTID,
 			NULL, 0, 0, 0, BTRFS_NESTING_NORMAL);
-	if (IS_ERR(leaf)) {
-		btrfs_put_root(root);
-		return ERR_CAST(leaf);
-	}
+	if (IS_ERR(leaf))
+		return PTR_ERR(leaf);
 
 	root->node = leaf;
 
 	btrfs_mark_buffer_dirty(root->node);
 	btrfs_tree_unlock(root->node);
-	return root;
+
+	return 0;
 }
 
 int btrfs_init_log_root_tree(struct btrfs_trans_handle *trans,
 			     struct btrfs_fs_info *fs_info)
 {
 	struct btrfs_root *log_root;
+	int ret;
 
 	log_root = alloc_log_tree(trans, fs_info);
 	if (IS_ERR(log_root))
 		return PTR_ERR(log_root);
+	ret = btrfs_alloc_log_tree_node(trans, log_root);
+	if (ret) {
+		kfree(log_root);
+		return ret;
+	}
 	WARN_ON(fs_info->log_root_tree);
 	fs_info->log_root_tree = log_root;
 	return 0;
@@ -1264,11 +1276,18 @@ int btrfs_add_log_tree(struct btrfs_trans_handle *trans,
 	struct btrfs_fs_info *fs_info = root->fs_info;
 	struct btrfs_root *log_root;
 	struct btrfs_inode_item *inode_item;
+	int ret;
 
 	log_root = alloc_log_tree(trans, fs_info);
 	if (IS_ERR(log_root))
 		return PTR_ERR(log_root);
 
+	ret = btrfs_alloc_log_tree_node(trans, log_root);
+	if (ret) {
+		btrfs_put_root(log_root);
+		return ret;
+	}
+
 	log_root->last_trans = trans->transid;
 	log_root->root_key.offset = root->root_key.objectid;
 
fs/btrfs/disk-io.h +2 -0
index fee69ced58b4..b82ae3711c42 100644
@@ -115,6 +115,8 @@ blk_status_t btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct bio *bio,
 			extent_submit_bio_start_t *submit_bio_start);
 blk_status_t btrfs_submit_bio_done(void *private_data, struct bio *bio,
 			  int mirror_num);
+int btrfs_alloc_log_tree_node(struct btrfs_trans_handle *trans,
+			      struct btrfs_root *root);
 int btrfs_init_log_root_tree(struct btrfs_trans_handle *trans,
 			     struct btrfs_fs_info *fs_info);
 int btrfs_add_log_tree(struct btrfs_trans_handle *trans,
Signature · 1 line
2.27.0
#104 lore raw

Re: [PATCH v9 37/41] btrfs: split alloc_log_tree()

Quoted 58 lines · Naohiro Aota

This is a preparation for the next patch. This commit split alloc_log_tree() to allocating tree structure part (remains in alloc_log_tree()) and allocating tree node part (moved in btrfs_alloc_log_tree_node()). The latter part is also exported to be used in the next patch.

---
  fs/btrfs/disk-io.c | 31 +++++++++++++++++++++++++------
  fs/btrfs/disk-io.h |  2 ++
  2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/disk-io.c +12 -6
index 2b30ef8a7034..70885f3d3321 100644
@@ -1211,7 +1211,6 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
  					 struct btrfs_fs_info *fs_info)
  {
  	struct btrfs_root *root;
-	struct extent_buffer *leaf;
 
  	root = btrfs_alloc_root(fs_info, BTRFS_TREE_LOG_OBJECTID, GFP_NOFS);
  	if (!root)
@@ -1221,6 +1220,14 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
  	root->root_key.type = BTRFS_ROOT_ITEM_KEY;
  	root->root_key.offset = BTRFS_TREE_LOG_OBJECTID;
 
+	return root;
+}
+
+int btrfs_alloc_log_tree_node(struct btrfs_trans_handle *trans,
+			      struct btrfs_root *root)
+{
+	struct extent_buffer *leaf;
+
  	/*
  	 * DON'T set SHAREABLE bit for log trees.
  	 *
@@ -1233,26 +1240,31 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
 
  	leaf = btrfs_alloc_tree_block(trans, root, 0, BTRFS_TREE_LOG_OBJECTID,
  			NULL, 0, 0, 0, BTRFS_NESTING_NORMAL);
-	if (IS_ERR(leaf)) {
-		btrfs_put_root(root);
-		return ERR_CAST(leaf);
-	}
+	if (IS_ERR(leaf))
+		return PTR_ERR(leaf);
 
  	root->node = leaf;
 
  	btrfs_mark_buffer_dirty(root->node);
  	btrfs_tree_unlock(root->node);
-	return root;
+
+	return 0;

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1261

Quoted 10 lines
fs/btrfs/disk-io.c +1 -0
  }
 
  int btrfs_init_log_root_tree(struct btrfs_trans_handle *trans,
  			     struct btrfs_fs_info *fs_info)
  {
  	struct btrfs_root *log_root;
+	int ret;
 
  	log_root = alloc_log_tree(trans, fs_info);
  	if (IS_ERR(log_root))

newline.

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1265

Quoted 4 lines
fs/btrfs/disk-io.c +3 -0
  		return PTR_ERR(log_root);
+	ret = btrfs_alloc_log_tree_node(trans, log_root);
+	if (ret) {
+		kfree(log_root);

btrfs_put_root(log_root);

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1267

Quoted 2 lines
fs/btrfs/disk-io.c +2 -0
+		return ret;
+	}

newline. Thanks,

Josef

#105 lore raw

Re: [PATCH v9 37/41] btrfs: split alloc_log_tree()

Quoted 78 lines · Josef Bacik
Quoted 58 lines · Naohiro Aota

This is a preparation for the next patch. This commit split alloc_log_tree() to allocating tree structure part (remains in alloc_log_tree()) and allocating tree node part (moved in btrfs_alloc_log_tree_node()). The latter part is also exported to be used in the next patch.

---
  fs/btrfs/disk-io.c | 31 +++++++++++++++++++++++++------
  fs/btrfs/disk-io.h |  2 ++
  2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/disk-io.c +12 -6
index 2b30ef8a7034..70885f3d3321 100644
@@ -1211,7 +1211,6 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
                                       struct btrfs_fs_info *fs_info)
  {
      struct btrfs_root *root;
-     struct extent_buffer *leaf;
 
      root = btrfs_alloc_root(fs_info, BTRFS_TREE_LOG_OBJECTID, GFP_NOFS);
      if (!root)
@@ -1221,6 +1220,14 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
      root->root_key.type = BTRFS_ROOT_ITEM_KEY;
      root->root_key.offset = BTRFS_TREE_LOG_OBJECTID;
 
+     return root;
+}
+
+int btrfs_alloc_log_tree_node(struct btrfs_trans_handle *trans,
+                           struct btrfs_root *root)
+{
+     struct extent_buffer *leaf;
+
      /*
       * DON'T set SHAREABLE bit for log trees.
       *
@@ -1233,26 +1240,31 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
 
      leaf = btrfs_alloc_tree_block(trans, root, 0, BTRFS_TREE_LOG_OBJECTID,
                      NULL, 0, 0, 0, BTRFS_NESTING_NORMAL);
-     if (IS_ERR(leaf)) {
-             btrfs_put_root(root);
-             return ERR_CAST(leaf);
-     }
+     if (IS_ERR(leaf))
+             return PTR_ERR(leaf);
 
      root->node = leaf;
 
      btrfs_mark_buffer_dirty(root->node);
      btrfs_tree_unlock(root->node);
-     return root;
+
+     return 0;

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1261

Quoted 10 lines
fs/btrfs/disk-io.c +1 -0
  }
 
  int btrfs_init_log_root_tree(struct btrfs_trans_handle *trans,
                           struct btrfs_fs_info *fs_info)
  {
      struct btrfs_root *log_root;
+     int ret;
 
      log_root = alloc_log_tree(trans, fs_info);
      if (IS_ERR(log_root))

newline.

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1265

Quoted 4 lines
fs/btrfs/disk-io.c +3 -0
              return PTR_ERR(log_root);
+     ret = btrfs_alloc_log_tree_node(trans, log_root);
+     if (ret) {
+             kfree(log_root);
Quoted 8 lines

btrfs_put_root(log_root);

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1267

Quoted 2 lines
fs/btrfs/disk-io.c +2 -0
+             return ret;
+     }

newline. Thanks,

Josef

These should've shown up on the patch formatter... was it simply not run, or did the patch formatter not catch it?

Best regards,
Amy Parker
(they/them)

#106 lore raw

Re: [PATCH v9 37/41] btrfs: split alloc_log_tree()

Quoted 84 lines · Amy Parker
Quoted 83 lines · Josef Bacik
Quoted 58 lines · Naohiro Aota

This is a preparation for the next patch. This commit split alloc_log_tree() to allocating tree structure part (remains in alloc_log_tree()) and allocating tree node part (moved in btrfs_alloc_log_tree_node()). The latter part is also exported to be used in the next patch.

---
  fs/btrfs/disk-io.c | 31 +++++++++++++++++++++++++------
  fs/btrfs/disk-io.h |  2 ++
  2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/disk-io.c +12 -6
index 2b30ef8a7034..70885f3d3321 100644
@@ -1211,7 +1211,6 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
                                       struct btrfs_fs_info *fs_info)
  {
      struct btrfs_root *root;
-     struct extent_buffer *leaf;
 
      root = btrfs_alloc_root(fs_info, BTRFS_TREE_LOG_OBJECTID, GFP_NOFS);
      if (!root)
@@ -1221,6 +1220,14 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
      root->root_key.type = BTRFS_ROOT_ITEM_KEY;
      root->root_key.offset = BTRFS_TREE_LOG_OBJECTID;
 
+     return root;
+}
+
+int btrfs_alloc_log_tree_node(struct btrfs_trans_handle *trans,
+                           struct btrfs_root *root)
+{
+     struct extent_buffer *leaf;
+
      /*
       * DON'T set SHAREABLE bit for log trees.
       *
@@ -1233,26 +1240,31 @@ static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans,
 
      leaf = btrfs_alloc_tree_block(trans, root, 0, BTRFS_TREE_LOG_OBJECTID,
                      NULL, 0, 0, 0, BTRFS_NESTING_NORMAL);
-     if (IS_ERR(leaf)) {
-             btrfs_put_root(root);
-             return ERR_CAST(leaf);
-     }
+     if (IS_ERR(leaf))
+             return PTR_ERR(leaf);
 
      root->node = leaf;
 
      btrfs_mark_buffer_dirty(root->node);
      btrfs_tree_unlock(root->node);
-     return root;
+
+     return 0;

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1261

Quoted 10 lines
fs/btrfs/disk-io.c +1 -0
  }
 
  int btrfs_init_log_root_tree(struct btrfs_trans_handle *trans,
                           struct btrfs_fs_info *fs_info)
  {
      struct btrfs_root *log_root;
+     int ret;
 
      log_root = alloc_log_tree(trans, fs_info);
      if (IS_ERR(log_root))

newline.

fs/btrfs/disk-io.c static struct btrfs_root *alloc_log_tree(struct btrfs_trans_handle *trans, · line 1265

Quoted 4 lines
fs/btrfs/disk-io.c +3 -0
              return PTR_ERR(log_root);
+     ret = btrfs_alloc_log_tree_node(trans, log_root);
+     if (ret) {
+             kfree(log_root);

btrfs_put_root(log_root);

Quoted 2 lines
fs/btrfs/disk-io.c +2 -0
+             return ret;
+     }
Quoted 6 lines
Quoted 3 lines

newline. Thanks,

Josef

These should've shown up on the patch formatter... was it simply not run, or did the patch formatter not catch it?

Unfotunately, it was not caught by checkpatch.pl...

Quoted 4 lines

Best regards,
Amy Parker
(they/them)