btrfs: delay compression to bbio submission time
cover btrfs: delay compression to bbio submission time
[PROOF OF CONCEPT ONLY]
!!! There are a lot of known bugs !!!
It's only for dicussion of design choices!
[DESIGN CHOICES]
The following design choices need to be discussed:
- Is delayed ordered extent needed
In theory we can skip the delayed ordered extent.
But later I found it harder to handle cases like > i_size writes. As beyond i_size writes needs to grab the OE and manually mark them finished.
Furthermore it will need extra handling for cow fixup.
Although the current delayed OE (and its child OEs) implementation is complex, it requires the least amount of changes for existing code.
- Async thread vs regular workqueue
The existing compression is done in async thread, which gives a special gift, the on-disk extent is mostly sequential.
This is ensured by the ordered function of async thread, where the extent is allocated sequentially.
The new PoC is using regular workqueue, can result out-of-order on-disk extents for compressed writes.
The main quirk is from the fact that ordered functions require re-execution of the same workload with @do_free set to true.
However in the current code we can not do the re-execution as the bbio can already be gone after the compressed bio is submitted.
[BACKGROUND]
Btrfs currently goes with async submission for compressed write, I'll go the following example to explain the async submission:
The page and fs block sizes are all 4K, no large folio involved. The dirty range is [0, 4K), [8K, 128K).
0 4K 8K 128K
|//| |/////////////////////////////////////////|- Write back folio 0
* Delalloc
writepage_delalloc() will find the delalloc range [0, 4K), and since it can not be inlined and too small for compression, it will be go through COW path, thus a new data extent is allocated, with corresponding EM/OE created.
* Submission
That folio 0 will be added into a bbio, and since we reached the OE end, the bbio will be submitted.
- Write back folio 8K
* Delalloc
writepage_delalloc() find the delalloc range [8K, 128K) and go compression.
Instead of allocating an extent immediately, it queues the work into delalloc_workers.
Please note that the range [8K, 128K) is completely locked during compression.
* Skip submission
As the whole folio 8K went through async submission, we skip bbio submission.
- Write back folio 12K
We wait for the folio to be unlocked (after compression is done and compressed bio is submitted).
When the folio is unlocked, the folio will have writeback flag set and its dirty flag cleared. Thus we either wait for the writeback or skip the folio completely.
This step repeats for the range [8K, 128K).
AFAIK the async submission is required as we can not submit two different bbios for a single compressed range.
Which is different from the uncompressed write path, where we can have several different bbios for a single ordered extent.
[PROBLEMS]
The async submission has the following problems:
- Non-sequential writeback
Especially when large folios are involved, we can have some blocks submitted immediately (uncompressed), and some submitted later (compressed).
That breaks the assumption of iomap and DONTCACHE writes.
- Not really async
As the example given above, we keep the whole range locked during compression, making later read and even writeback itself to wait.
[DELAYED COMPRESSION]
The new idea is to delay the compression at bbio submission time. Now the workflow will be:
- Write back folio 0
The same, submitting it immediately
- Write back folio 8K
* Delalloc
writepage_delalloc() find the delalloc range [8K, 128K) and go compression, but this time we allocated delayed EM and OE for the range [8K, 128K).
* Submission
That folio 8K will be added into a bbio, with its dirty flag removed and writeback flag set.
- Writeback folio 12K ~ 124K
* Delalloc
No new delalloc range.
* Submission
Those folios will be added to the same bbio above.
And after the last folio 124K is queued, we reached the OE end, and will submit the delayed bbio.
- Delayed bbio submission
As the bbio has a special is_delayed flag set, it will not be submitted directly, but queued into a workqueue for compression.
* Compression in the workqueue
* Real delalloc
Now an on-disk extent is reserved. The real EM will replace the delayed one.
And the real OE will be added as a child of the original delayed one.
* Compressed data submission
* Delayed bbio finish
When all child compressed/uncompressed writes finished, the delayed bbio will finish.
The full delayed OE is also finished, which will insert all of its child OEs into the subvolume tree.
This solves both the problems mentioned above, but is definitely way more complex than the current async submission:
- Layered OEs
And we need to manage the child/parent OEs properly
- Possible extra split
Since the delayed OE is allocated first, we can still submit two different delayed bbio for the same OE.
This means we can have two smaller compressed extents compared to one, which may reduce the compression ratio.
- More complex error handling
- Unsolved bugs
* Em leak
* Bytes mayuse leak (METADATA)
* Busy inode after umount
* More hidden
Qu Wenruo (6):
btrfs: add skeleton for delayed btrfs bio
btrfs: add delayed ordered extent support
btrfs: introduce the skeleton of delayed bbio endio function
btrfs: introduce compression for delayed bbio
btrfs: implement uncompressed fallback for delayed bbio
btrfs: enable experimental delayed compression support
fs/btrfs/bio.c | 1 + fs/btrfs/bio.h | 3 + fs/btrfs/btrfs_inode.h | 3 + fs/btrfs/extent_io.c | 29 ++- fs/btrfs/extent_map.h | 9 +- fs/btrfs/inode.c | 475 +++++++++++++++++++++++++++++++++++++++- fs/btrfs/ordered-data.c | 73 +++++- fs/btrfs/ordered-data.h | 14 ++ 8 files changed, 595 insertions(+), 12 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/6 btrfs: add skeleton for delayed btrfs bio
The objective of such new delayed btrfs bio infrastructure is to allow compressed write to go the regular extent_writepage_io() path, without going through the async submission path.
This will make it easier to align our write path to iomap.
The core ideas of delayed btrfs bio are:
- A place holder ordered extent created at delalloc time No space is reserved at that time.
- A delayed extent map created at delalloc time
It will have a special disk_bytenr (-4) to indicate the range is delayed.
- Delayed btrfs bio will be limited to BTRFS_MAX_COMPRESSED size
- Btrfs bio assembly mostly follows the regular path There are several small exceptions:
* btrfs_bio_is_contig() needs to handle delayed disk_bytenr/bbio
* New bbio needs to have its is_delayed flag set if disk_bytenr is EXTENT_MAP_DELAYED
- Real ordered extent will be created at bbio submission time This part is not implemented in this patch.
--- fs/btrfs/bio.c | 1 + fs/btrfs/bio.h | 3 +++ fs/btrfs/btrfs_inode.h | 3 +++ fs/btrfs/extent_io.c | 29 +++++++++++++++++++++++++---- fs/btrfs/extent_map.h | 9 ++++++++- fs/btrfs/inode.c | 41 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 81 insertions(+), 5 deletions(-)
fs/btrfs/bio.c +1 -0
@@ -900,6 +900,7 @@ void btrfs_submit_bbio(struct btrfs_bio *bbio, int mirror_num) {/* If bbio->inode is not populated, its file_offset must be 0. */ ASSERT (bbio->inode || bbio->file_offset ==0 ); +ASSERT (!bbio->is_delayed);assert_bbio_alignment (bbio);
fs/btrfs/bio.h +3 -0
@@ -99,6 +99,9 @@ struct btrfs_bio {/* Whether the bio is written using zone append. */ bool can_use_append:1; +/* If the bio is delayed (aka, no backing OE). */ + bool is_delayed:1; +/* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last.
fs/btrfs/btrfs_inode.h +3 -0
@@ -669,5 +669,8 @@ u64 btrfs_get_extent_allocation_hint(struct btrfs_inode *inode, u64 start,struct extent_map *btrfs_create_io_em (struct btrfs_inode *inode,u64 start,const struct btrfs_file_extent *file_extent,int type); +struct extent_map *btrfs_create_delayed_em (struct btrfs_inode *inode, +u64 start,u32 length); +void btrfs_submit_delayed_write (struct btrfs_bio *bbio);#endif
fs/btrfs/extent_io.c +25 -4
@@ -180,12 +180,16 @@ static void submit_one_bio(struct btrfs_bio_ctrl *bio_ctrl)/* Caller should ensure the bio has at least some range added */ ASSERT (bbio->bio.bi_iter.bi_size); - +/* Delayed bbio is only for write. */ +if (bbio->is_delayed) +ASSERT (btrfs_op (&bbio->bio) ==BTRFS_MAP_WRITE );bio_set_csum_search_commit_root (bio_ctrl);if (btrfs_op (&bbio->bio) ==BTRFS_MAP_READ && bio_ctrl->compress_type !=BTRFS_COMPRESS_NONE )btrfs_submit_compressed_read (bbio); +else if (bbio->is_delayed) +btrfs_submit_delayed_write (bbio);else btrfs_submit_bbio (bbio,0 );
@@ -723,6 +727,14 @@ static bool btrfs_bio_is_contig(struct btrfs_bio_ctrl *bio_ctrl,struct bio *bio = &bio_ctrl->bbio->bio;const sector_t sector = disk_bytenr >>SECTOR_SHIFT ; +/* One is delayed bbio and one is not, definitely not contig. */ +if (bio_ctrl->bbio->is_delayed != (disk_bytenr ==EXTENT_MAP_DELAYED )) +return false; + +/* For delayed bbio, only need to check if the file range is contig. */ +if (bio_ctrl->bbio->is_delayed) +return bio_ctrl->next_file_offset == file_offset; +if (bio_ctrl->compress_type !=BTRFS_COMPRESS_NONE ) {/* * For compression, all IO should have its logical bytenr set
@@ -748,7 +760,13 @@ static void alloc_new_bio(struct btrfs_inode *inode, bbio =btrfs_bio_alloc (BIO_MAX_VECS , bio_ctrl->opf, inode, file_offset, bio_ctrl->end_io_func,NULL ); - bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; +if (disk_bytenr ==EXTENT_MAP_DELAYED ) { + bbio->is_delayed = true; + bbio->bio.bi_iter.bi_sector =0 ; + }else { + bbio->is_delayed = false; + bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; + } bbio->bio.bi_write_hint = inode->vfs_inode.i_write_hint; bio_ctrl->bbio = bbio; bio_ctrl->len_to_oe_boundary =U32_MAX ;
@@ -762,7 +780,7 @@ static void alloc_new_bio(struct btrfs_inode *inode,if (ordered) { bio_ctrl->len_to_oe_boundary =min_t (u32,U32_MAX , ordered->file_offset + - ordered->disk_num_bytes - file_offset); + ordered->num_bytes - file_offset); bbio->ordered = ordered; }
@@ -1688,7 +1706,10 @@ static int submit_one_sector(struct btrfs_inode *inode,ASSERT (IS_ALIGNED (em->len, sectorsize)); block_start =btrfs_extent_map_block_start (em); - disk_bytenr =btrfs_extent_map_block_start (em) + extent_offset; +if (block_start ==EXTENT_MAP_DELAYED ) + disk_bytenr = block_start; +else + disk_bytenr = block_start + extent_offset;ASSERT (!btrfs_extent_map_is_compressed (em));ASSERT (block_start !=EXTENT_MAP_HOLE );
fs/btrfs/extent_map.h +8 -1
@@ -13,7 +13,8 @@struct btrfs_inode ;struct btrfs_fs_info ; -#define EXTENT_MAP_LAST_BYTE ((u64)-4) +#define EXTENT_MAP_LAST_BYTE ((u64)-5) +#define EXTENT_MAP_DELAYED ((u64)-4)#define EXTENT_MAP_HOLE ((u64)-3)#define EXTENT_MAP_INLINE ((u64)-2)
@@ -30,6 +31,12 @@ enum {ENUM_BIT (EXTENT_FLAG_LOGGING ),/* This em is merged from two or more physically adjacent ems */ ENUM_BIT (EXTENT_FLAG_MERGED ), +/* +* This real on-disk extent allocation is delayed until bio submission. +* For now it's only a place holder with EXTENT_MAP_DELAYED as +* its disk_bytenr. +*/ + ENUM_BIT(EXTENT_FLAG_DELAYED), };/*
fs/btrfs/inode.c +41 -0
47 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH 2/6 btrfs: add delayed ordered extent support
In reply to Qu WenruoA delayed ordered extent acts as a place holder ordered extent, thus its disk_bytenr/disk_num_bytes will all be zero.
A real OE inside the delayed OE will be added to the child list of the delayed OE.
When the parent delayed OE finished, child OEs will be finished and have their valid file extents inserted.
And if some range has no child OE, e.g. beyond i_size, the range without child OE will be manually released.
--- fs/btrfs/inode.c | 66 +++++++++++++++++++++++++++++++++++++ fs/btrfs/ordered-data.c | 73 ++++++++++++++++++++++++++++++++++++++--- fs/btrfs/ordered-data.h | 14 ++++++++ 3 files changed, 149 insertions(+), 4 deletions(-)
fs/btrfs/inode.c +66 -0
@@ -3162,6 +3162,69 @@ static int insert_ordered_extent_file_extent(struct btrfs_trans_handle *trans, update_inode_bytes, oe->qgroup_rsv); } +static int finish_delayed_ordered (struct btrfs_ordered_extent *oe) +{ +struct btrfs_inode *inode = oe->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct btrfs_ordered_extent *child; +struct btrfs_ordered_extent *tmp; +const u32 nr_bits = oe->num_bytes >> fs_info->sectorsize_bits; +bool io_error =test_bit (BTRFS_ORDERED_IOERR , &oe->flags); +u64 cur = oe->file_offset; +int ret =0 ; +int saved_ret =0 ; + +/* Finish each child OE. */ +list_for_each_entry_safe (child, tmp, &oe->child_list, child_list) { +list_del_init (&child->child_list); +refcount_inc (&child->refs); + +/* The range should have been marked in the bitmap. */ +ASSERT (bitmap_test_range_all_set (oe->child_bitmap, + (child->file_offset - oe->file_offset) >> fs_info->sectorsize_bits, + child->num_bytes >> fs_info->sectorsize_bits)); + +if (io_error) +set_bit (BTRFS_ORDERED_IOERR , &child->flags); + + ret =btrfs_finish_one_ordered (child); +if (ret && !saved_ret) + saved_ret = ret; + } + +/* For ranges that doesn't have a child OE, manually clean them up. */ +while (cur < oe->file_offset + oe->num_bytes) { +const u32 cur_bit = (cur - oe->file_offset) >> fs_info->sectorsize_bits; +u32 first_zero; +u32 next_set; +u64 range_start; +u64 range_end; +u32 range_len; + + first_zero =find_next_zero_bit (oe->child_bitmap, nr_bits, cur_bit); +if (first_zero >= nr_bits) +break ; + next_set =find_next_bit (oe->child_bitmap, nr_bits, first_zero); +ASSERT (next_set > first_zero); + + range_start = oe->file_offset + (first_zero << fs_info->sectorsize_bits); + range_len = (next_set - first_zero) << fs_info->sectorsize_bits; + range_end = range_start + range_len -1 ; + +/* +* The range has no real OE created, thus its reserved data/meta space +* needs to be manually released by EXTENT_DO_ACCOUNTING. +*/ +btrfs_clear_extent_bit (&inode->io_tree, range_start, range_end, +EXTENT_DELALLOC_NEW |EXTENT_DEFRAG |EXTENT_DO_ACCOUNTING ,NULL ); + cur = range_end +1 ; + } +btrfs_remove_ordered_extent (oe); +btrfs_put_ordered_extent (oe); +btrfs_put_ordered_extent (oe); +return saved_ret; +} +/* * As ordered data IO finishes, this gets called so we can finish * an ordered extent if the range of bytes in the file it covers are
@@ -3184,6 +3247,9 @@ int btrfs_finish_one_ordered(struct btrfs_ordered_extent *ordered_extent)bool clear_reserved_extent = true;unsigned int clear_bits =EXTENT_DEFRAG ; +if (test_bit (BTRFS_ORDERED_DELAYED , &ordered_extent->flags)) +return finish_delayed_ordered (ordered_extent); + start = ordered_extent->file_offset; end = start + ordered_extent->num_bytes -1 ;
fs/btrfs/ordered-data.c +69 -4
116 lines not on this page · raw message
fs/btrfs/ordered-data.h +14 -0
@@ -13,6 +13,7 @@#include <linux/rbtree.h> #include <linux/wait.h> #include "async-thread.h" +#include "compression.h" struct inode ;struct page ;
@@ -87,6 +88,12 @@ enum {*/ BTRFS_ORDERED_DIRECT , +/* +* Extra bit for delayed OE, can only be set for REGULAR. +* Can not be set with COMPRESSED/ENCODED/DIRECT. +*/ +BTRFS_ORDERED_DELAYED , +BTRFS_ORDERED_NR_FLAGS , };static_assert (BTRFS_ORDERED_NR_FLAGS <=BITS_PER_LONG );
@@ -155,6 +162,11 @@ struct btrfs_ordered_extent {/* a per root list of all the pending ordered extents */ struct list_head root_extent_list; +/* Child ordered extent list for delayed OE. */ +struct list_head child_list; + +unsigned long child_bitmap[BITS_TO_LONGS (BTRFS_MAX_COMPRESSED /BTRFS_MIN_BLOCKSIZE )]; +struct btrfs_work work;struct completion completion;
@@ -192,6 +204,8 @@ struct btrfs_file_extent {struct btrfs_ordered_extent *btrfs_alloc_ordered_extent (struct btrfs_inode *inode,u64 file_offset,const struct btrfs_file_extent *file_extent,unsigned long flags); +struct btrfs_ordered_extent *btrfs_alloc_delayed_ordered_extent ( +struct btrfs_inode *inode,u64 file_offset,u32 length);void btrfs_add_ordered_sum (struct btrfs_ordered_extent *entry,struct btrfs_ordered_sum *sum); struct btrfs_ordered_extent *btrfs_lookup_ordered_extent(struct btrfs_inode *inode,
Signature · 1 line
2.53.0
PATCH 3/6 btrfs: introduce the skeleton of delayed bbio endio function
In reply to Qu WenruoA delayed bbio will not be directly submitted, but queued into a workqueue, doing the heavy lifting compression there.
The compression and uncompressed fallback are not implemented in this patch.
Only the main endio function and helper to queue workload into a workqueue is implemented.
The endio function is mostly the same as end_bbio_data_write(), except the extra memory allocation/freeing for the bbio->private.
--- fs/btrfs/inode.c | 67 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 63 insertions(+), 4 deletions(-)
fs/btrfs/inode.c +63 -4
@@ -97,6 +97,12 @@ struct data_reloc_warn {int mirror_num; }; +struct delayed_bio_private { +struct work_struct work; +struct btrfs_bio *delayed_bbio; +atomic_t pending_ios; +}; +/* * For the file_extent_tree, we want to hold the inode lock when we lookup and * update the disk_i_size, but lockdep will complain because our io_tree we hold
@@ -7647,18 +7653,71 @@ struct extent_map *btrfs_create_delayed_em(struct btrfs_inode *inode,return em; } -void btrfs_submit_delayed_write (struct btrfs_bio *bbio) +static void run_delayed_bbio (struct work_struct *work) { -ASSERT (bbio->is_delayed); + struct delayed_bio_private *dbp = container_of(work, struct delayed_bio_private, work); +struct btrfs_bio *parent = dbp->delayed_bbio;/* -* Not yet implemented, and should not hit this path as we have no -* caller to create delayed extent map. +* Increase the pending_ios so that parent bbio won't end +* until all child ones are submitted. */ +atomic_inc (&dbp->pending_ios); +/* Compressed and uncompressed fallback is not yet implemented. */ ASSERT (0 ); +if (atomic_dec_and_test (&dbp->pending_ios)) +btrfs_bio_end_io (parent, parent->status); +} + +static void end_bbio_delayed (struct btrfs_bio *bbio) +{ +struct delayed_bio_private *dbp = bbio->private; +struct btrfs_inode *inode = bbio->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct folio_iter fi; +const u32 bio_size =bio_get_size (&bbio->bio); +const bool uptodate = bbio->status ==BLK_STS_OK ; + +ASSERT (bbio->is_delayed); + +bio_for_each_folio_all (fi, &bbio->bio) { +u64 start =folio_pos (fi.folio) + fi.offset; +u32 len = fi.length; + +btrfs_folio_clear_writeback (fs_info, fi.folio, start, len); + } +btrfs_mark_ordered_io_finished (inode, bbio->file_offset, bio_size, uptodate); +kfree (dbp);bio_put (&bbio->bio); } +void btrfs_submit_delayed_write (struct btrfs_bio *bbio) +{ +struct delayed_bio_private *dbp; + +ASSERT (bbio->is_delayed); + + bbio->end_io = end_bbio_delayed; + dbp =kzalloc (sizeof (struct delayed_bio_private ),GFP_NOFS ); +if (!dbp) { +btrfs_bio_end_io (bbio,errno_to_blk_status (-ENOMEM )); +return ; + } +atomic_set (&dbp->pending_ios,0 ); + dbp->delayed_bbio = bbio; + bbio->private = dbp; +/* +* TODO: find a way to properly allow sequential extent allocation. +* +* The existing btrfs async workqueue will execute the sequential workload +* twice, the second one to free the structure. +* But our current submission path can only be called once, after that +* the bbio will be gone thus can not afford to use btrfs async workqueue. +*/ +INIT_WORK (&dbp->work, run_delayed_bbio); +schedule_work (&dbp->work); +} +/* * For release_folio() and invalidate_folio() we have a race window where * folio_end_writeback() is called but the subpage spinlock is not yet released.
Signature · 1 line
2.53.0
PATCH 4/6 btrfs: introduce compression for delayed bbio
In reply to Qu WenruoThe compressed write path inside a delayed bbio is mostly the same as regular compression, but with some differences:
- The error handling should not touch folio flags
It will be handled by the parent delayed bbio.
- A successful compression will lead to a child compressed bio That compressed bio will be properly submitted, and if there is no more pending ios of the delayed bbio, end the delayed bbio.
- No sequential execution of data extent reservation The existing async thread is some quirk related to the ordered function execution, which is not suitable for this call site.
After the compressed bio is submitted, we can no longer touch the child compressed bio (it can finished immediately and also finish the parent delayed bbio).
Meanwhile the async ordered function needs different entries to handle the workload and free involved structures.
This will be the major changes compared to the existing compressed write.
--- fs/btrfs/inode.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 111 insertions(+), 1 deletion(-)
fs/btrfs/inode.c +111 -1
@@ -7653,6 +7653,111 @@ struct extent_map *btrfs_create_delayed_em(struct btrfs_inode *inode,return em; } +static void end_bbio_delayed_compressed (struct btrfs_bio *bbio) +{ +struct delayed_bio_private *dbp = bbio->private; +struct btrfs_bio *parent = dbp->delayed_bbio; +struct folio_iter fi; + +bio_for_each_folio_all (fi, &bbio->bio) +btrfs_free_compr_folio (fi.folio); +bio_put (&bbio->bio); + +cmpxchg (&parent->status,BLK_STS_OK , bbio->status); +if (atomic_dec_and_test (&dbp->pending_ios)) +btrfs_bio_end_io (parent, parent->status); +} + +static bool try_submit_compressed (struct btrfs_bio *parent) +{ +struct delayed_bio_private *dbp = parent->private; +struct btrfs_bio *bbio = dbp->delayed_bbio; +struct btrfs_inode *inode = bbio->inode; +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct btrfs_key ins; +struct compressed_bio *cb; +struct extent_state *cached =NULL ; +struct extent_map *em; +struct btrfs_ordered_extent *ordered; +struct btrfs_file_extent file_extent; +u64 alloc_hint; +const u32 len =bio_get_size (&bbio->bio); +const u64 fileoff = bbio->file_offset; +const u64 end = fileoff + len -1 ; +u32 compressed_size; +int compress_type = fs_info->compress_type; +int compress_level = fs_info->compress_level; +int ret; + +if (!btrfs_inode_can_compress (inode) || + !inode_need_compress (inode, fileoff, len, false)) +return false; + +if (inode->defrag_compress >0 && + inode->defrag_compress <BTRFS_NR_COMPRESS_TYPES ) { + compress_type = inode->defrag_compress; + compress_level = inode->defrag_compress_level; + }else if (inode->prop_compress) { + compress_type = inode->prop_compress; + } + cb =btrfs_compress_bio (inode, fileoff, len, compress_type, + compress_level,0 ); +if (IS_ERR (cb)) +return false; + +round_up_last_block (cb, fs_info->sectorsize); + compressed_size = cb->bbio.bio.bi_iter.bi_size; + + alloc_hint =btrfs_get_extent_allocation_hint (inode, fileoff, len); + ret =btrfs_reserve_extent (inode->root, len, + compressed_size, compressed_size, +0 , alloc_hint, &ins, true, true); +if (ret <0 ) { +cleanup_compressed_bio (cb); +return false; + } +btrfs_lock_extent (&inode->io_tree, fileoff, end, &cached); + file_extent.disk_bytenr = ins.objectid; + file_extent.disk_num_bytes = ins.offset; + file_extent.ram_bytes = len; + file_extent.num_bytes = len; + file_extent.offset =0 ; + file_extent.compression = cb->compress_type; + + cb->bbio.bio.bi_iter.bi_sector = ins.objectid >>SECTOR_SHIFT ; + em =btrfs_create_io_em (inode, fileoff, &file_extent,BTRFS_ORDERED_COMPRESSED ); +if (IS_ERR (em)) { + ret =PTR_ERR (em); + goto out_free_reserve; + } +btrfs_free_extent_map (em); + + ordered =btrfs_alloc_ordered_extent (inode, fileoff, &file_extent, +1U <<BTRFS_ORDERED_COMPRESSED ); +if (IS_ERR (ordered)) { +btrfs_drop_extent_map_range (inode, fileoff, end, false); + ret =PTR_ERR (ordered); + goto out_free_reserve; + } + cb->bbio.ordered = ordered; +btrfs_dec_block_group_reservations (fs_info, ins.objectid); +btrfs_unlock_extent (&inode->io_tree, fileoff, end, &cached); + + cb->bbio.end_io = end_bbio_delayed_compressed; + cb->bbio.private = dbp; +atomic_inc (&dbp->pending_ios); +btrfs_submit_bbio (&cb->bbio,0 ); +return true; + +out_free_reserve: +btrfs_dec_block_group_reservations (fs_info, ins.objectid); +btrfs_free_reserved_extent (fs_info, ins.objectid, ins.offset, true); +mapping_set_error (inode->vfs_inode.i_mapping, -EIO ); +btrfs_unlock_extent (&inode->io_tree, fileoff, end, &cached); +cleanup_compressed_bio (cb); +return false; +} +static void run_delayed_bbio (struct work_struct *work) { struct delayed_bio_private *dbp = container_of(work, struct delayed_bio_private, work);
@@ -7663,8 +7768,13 @@ static void run_delayed_bbio(struct work_struct *work)* until all child ones are submitted. */ atomic_inc (&dbp->pending_ios); -/* Compressed and uncompressed fallback is not yet implemented. */ +if (try_submit_compressed (parent)) + goto finish; + +/* Uncompressed fallback is not yet implemented. */ ASSERT (0 ); + +finish:if (atomic_dec_and_test (&dbp->pending_ios))btrfs_bio_end_io (parent, parent->status); }
Signature · 1 line
2.53.0
PATCH 5/6 btrfs: implement uncompressed fallback for delayed bbio
In reply to Qu WenruoWhen the compression failed (either bad ratio or fragmented free space), we have to fall back to uncompressed write.
And in that case we may need to split the existing range into multiple child bbios.
The child bbios will use folios from the page cache, but do not update their flags, which is done by the parent bbio.
--- fs/btrfs/inode.c | 144 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 142 insertions(+), 2 deletions(-)
fs/btrfs/inode.c +142 -2
@@ -7758,10 +7758,139 @@ static bool try_submit_compressed(struct btrfs_bio *parent)return false; } +static void end_bbio_delayed_uncompressed (struct btrfs_bio *bbio) +{ +struct delayed_bio_private *dbp = bbio->private; +struct btrfs_bio *parent = dbp->delayed_bbio; +struct folio_iter fi; + +bio_for_each_folio_all (fi, &bbio->bio) +folio_put (fi.folio); +bio_put (&bbio->bio); + +cmpxchg (&parent->status,BLK_STS_OK , bbio->status); +if (atomic_dec_and_test (&dbp->pending_ios)) +btrfs_bio_end_io (parent, parent->status); +} + +static struct btrfs_bio *child_bbio_from_page_cache (struct btrfs_bio *parent, +u64 fileoff,u32 len) +{ +struct btrfs_inode *inode = parent->inode; +struct address_space *mapping = inode->vfs_inode.i_mapping; +struct btrfs_bio *bbio; +struct folio_iter fi; +u64 cur = fileoff; +int ret; + + bbio =btrfs_bio_alloc (round_up (len,PAGE_SIZE ) >>PAGE_SHIFT ,REQ_OP_WRITE , + inode, fileoff, end_bbio_delayed_uncompressed, + parent->private); + +while (cur < fileoff + len) { +struct folio *folio; +u32 cur_len = fileoff + len - cur; + + folio =filemap_get_folio (mapping, cur >>PAGE_SHIFT ); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); + goto error; + } + ret =bio_add_folio (&bbio->bio, folio, cur_len, +offset_in_folio (folio, cur)); +ASSERT (ret); + } + +return bbio; +error: +bio_for_each_folio_all (fi, &bbio->bio) +folio_put (fi.folio); +bio_put (&bbio->bio); +return ERR_PTR (ret); +} + +static int submit_one_uncompressed_range (struct btrfs_bio *parent,struct btrfs_key *ins, +struct extent_state **cached,u64 file_offset, +u32 num_bytes,u64 alloc_hint,u32 *ret_alloc_size) +{ +struct btrfs_inode *inode = parent->inode; +struct delayed_bio_private *dbp = parent->private; +struct btrfs_root *root = inode->root; +struct btrfs_fs_info *fs_info = root->fs_info; +struct btrfs_ordered_extent *ordered; +struct btrfs_file_extent file_extent; +struct btrfs_bio *child; +struct extent_map *em; +u64 cur_end; +u32 cur_len =0 ; +int ret; + + ret =btrfs_reserve_extent (root, num_bytes, num_bytes, fs_info->sectorsize, +0 , alloc_hint, ins, true, true); +if (ret <0 ) +return ret; + + cur_len = ins->offset; + cur_end = file_offset + cur_len -1 ; + + file_extent.disk_bytenr = ins->objectid; + file_extent.disk_num_bytes = ins->offset; + file_extent.num_bytes = ins->offset; + file_extent.ram_bytes = ins->offset; + file_extent.offset =0 ; + file_extent.compression =BTRFS_COMPRESS_NONE ; + +btrfs_lock_extent (&inode->io_tree, file_offset, cur_end, cached); + em =btrfs_create_io_em (inode, file_offset, &file_extent,BTRFS_ORDERED_REGULAR ); +if (IS_ERR (em)) { + ret =PTR_ERR (em); + goto free_reserved; + } +btrfs_free_extent_map (em); + ordered =btrfs_alloc_ordered_extent (inode, file_offset, &file_extent, +1U <<BTRFS_ORDERED_REGULAR ); +if (IS_ERR (ordered)) { +btrfs_drop_extent_map_range (inode, file_offset, cur_end, false); + ret =PTR_ERR (ordered); + goto free_reserved; + } + child =child_bbio_from_page_cache (parent, file_offset, cur_len); +if (IS_ERR (child)) { +btrfs_put_ordered_extent (ordered); +btrfs_drop_extent_map_range (inode, file_offset, cur_end, false); + ret =PTR_ERR (ordered); + goto free_reserved; + } + child->ordered = ordered; + child->private = parent->private; + child->end_io = end_bbio_delayed_uncompressed; + child->bio.bi_iter.bi_sector = ins->objectid >>SECTOR_SHIFT ; +atomic_inc (&dbp->pending_ios); +btrfs_submit_bbio (child,0 ); + *ret_alloc_size = cur_len; +return 0 ; + +free_reserved: +btrfs_unlock_extent (&inode->io_tree, file_offset, cur_end, cached); +btrfs_qgroup_free_data (inode,NULL , file_offset, cur_len,NULL ); +btrfs_dec_block_group_reservations (fs_info, ins->objectid); +btrfs_free_reserved_extent (fs_info, ins->objectid, ins->offset, true); +ASSERT (ret != -EAGAIN ); +return ret; +} +static void run_delayed_bbio (struct work_struct *work) { struct delayed_bio_private *dbp = container_of(work, struct delayed_bio_private, work);struct btrfs_bio *parent = dbp->delayed_bbio; +struct btrfs_key ins; +struct extent_state *cached =NULL ; +const u32 uncompressed_size =bio_get_size (&parent->bio); +const u64 start = parent->file_offset; +const u64 end = start + uncompressed_size -1 ; +u64 cur = start; +u64 alloc_hint; +int ret =0 ;/* * Increase the pending_ios so that parent bbio won't end
@@ -7771,8 +7900,19 @@ static void run_delayed_bbio(struct work_struct *work)if (try_submit_compressed (parent)) goto finish; -/* Uncompressed fallback is not yet implemented. */ -ASSERT (0 ); + alloc_hint =btrfs_get_extent_allocation_hint (parent->inode, start, + uncompressed_size); +while (cur < end) { +u32 cur_len; + + ret =submit_one_uncompressed_range (parent, &ins, &cached, + cur, end +1 - cur, + alloc_hint, &cur_len); +if (ret <0 ) + goto finish; + cur += cur_len; + alloc_hint += cur_len; + } finish:if (atomic_dec_and_test (&dbp->pending_ios))
Signature · 1 line
2.53.0
PATCH 6/6 btrfs: enable experimental delayed compression support
In reply to Qu Wenruo!!! WIP !!!
There are still a lot of bugs that need to be resolved, this is just a proof-of-concept patch for now.
Instead of the existing async submission path, the new delayed bbio will handle compressed write by:
- Allocating delayed em/oe at run_delalloc_*() time Thus no data extent is reserved at that time.
- Delayed bbio will be assembled at extent_writepage_io() time
- Delayed bbio will be intercepted
Which will run compression (or fallback to uncompressed write). Data extents will only be reserved at that time, and the delayed em will be replaced by real ones.
Meanwhile the real OE will be added as a child of the parent delayed OE, and when the parent OE finishes, the child OE will be finished with their file extents inserted.
This has some benefits:
- Higher concurrency
Previously async submission will hold the folio and io tree range locked, this means we can not even read the uptodate folio.
Furthermore although the compressed write is queued into a workqueue for submission and extent_writepage_io() will skip the compressed range, when we need to write the next folio of the compressed range, we will need to wait for the folio to be unlocked.
This makes async submission less async.
- Future DONTCACHE writes support
We do not support DONTCACHE because that flag requires write back path to clear the folio dirty and submit them sequentially.
Meanwhile async submission makes the writeback async, breaking the sequential submission requirement.
This is also why we need a complex per-block tracking for writeback flags, meanwhile iomap only requires a counter tracking.
Unfortunately there are still quite some known bugs that I'm still debugging:
- Em leaks
- Busy inode after umount
- Metadata bytes_mayuse leak
--- fs/btrfs/inode.c | 59 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-)
fs/btrfs/inode.c +56 -3
@@ -1673,6 +1673,56 @@ static bool run_delalloc_compressed(struct btrfs_inode *inode,return true; } +static int run_delalloc_delayed (struct btrfs_inode *inode,struct folio *locked_folio, +u64 start,u64 end) +{ +struct btrfs_root *root = inode->root; +struct btrfs_fs_info *fs_info = root->fs_info; +struct extent_state *cached =NULL ; +u64 cur = start; +int ret; + +if (btrfs_is_shutdown (fs_info)) { + ret = -EIO ; + goto error; + } +while (cur < end) { +struct extent_map *em; +struct btrfs_ordered_extent *oe; +u32 cur_len =min_t (u64, end +1 - cur,BTRFS_MAX_COMPRESSED ); + +btrfs_lock_extent (&inode->io_tree, cur, cur + cur_len -1 , &cached); + em =btrfs_create_delayed_em (inode, cur, cur_len); +if (IS_ERR (em)) { + ret =PTR_ERR (em); + goto error; + } +btrfs_free_extent_map (em); + oe =btrfs_alloc_delayed_ordered_extent (inode, cur, cur_len); +if (IS_ERR (oe)) { +btrfs_drop_extent_map_range (inode, cur, cur + cur_len -1 , false); + ret =PTR_ERR (em); + goto error; + } + + cur += cur_len; + } +extent_clear_unlock_delalloc (inode, start, end, locked_folio, &cached, +EXTENT_LOCKED |EXTENT_DELALLOC , +PAGE_UNLOCK |PAGE_SET_ORDERED ); +return 0 ; +error: +/* We have to drop any created delayed extent maps. */ +if (start < cur) +btrfs_drop_extent_map_range (inode, start, cur -1 , false); +/* No range has any extent reserved, just clear them all. */ +extent_clear_unlock_delalloc (inode, start, end, locked_folio, &cached, +EXTENT_LOCKED |EXTENT_DELALLOC |EXTENT_DELALLOC_NEW | +EXTENT_DEFRAG |EXTENT_DO_ACCOUNTING , +PAGE_UNLOCK |PAGE_START_WRITEBACK |PAGE_END_WRITEBACK ); +return ret; +} +/* * Run the delalloc range from start to end, and write back any dirty pages * covered by the range.
@@ -2436,9 +2486,12 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct folio *locked_folreturn run_delalloc_nocow (inode, locked_folio, start, end);if (btrfs_inode_can_compress (inode) && -inode_need_compress (inode, start, end, false) && -run_delalloc_compressed (inode, locked_folio, start, end, wbc)) -return 1 ; +inode_need_compress (inode, start, end, false)) { +if (IS_ENABLED (CONFIG_BTRFS_EXPERIMENTAL )) +return run_delalloc_delayed (inode, locked_folio, start, end); +else if (run_delalloc_compressed (inode, locked_folio, start, end, wbc)) +return 1 ; + }if (zoned)return run_delalloc_cow (inode, locked_folio, start, end, wbc, true);
Signature · 1 line
2.53.0