md/llbitmap: md/md-llbitmap: introduce a new lockless bitmap
Subsystems (3)
v4 cover md/llbitmap: md/md-llbitmap: introduce a new lockless bitmap
From: Yu Kuai <[email protected]>
Changes from v3:
- fix redundant setting mddev->bitmap_id in patch 6;
- add explanation of bitmap attributes in Documentation;
- add llbitmap/barrier_idle in patch 11;
- add some comments in patch 11;
Changes from v2:
- add comments about KOBJECT_CHANGE uevent in patch 6;
- convert llbitmap_suspend() to llbitmap_suspend_timeout() in patch 11;
- add some comments in patch 11;
- add review tag:
- patch 3,4,5,9 from Hannes
Changes from v1:
- explain md_bitmap_fn in commit message, patch 3;
- handle the case CONFIG_MD_BITMAP is disabled, patch 4;
- split patch 7 from v1 into patch 5 + 6;
- rewrite bitmap_type_store, patch 5;
- fix dm-raid regerssion that md-bitmap sysfs entries should not be created under mddev kobject, patch 6
- merge llbitmap patches into one patch, with lots of cleanups;
- add review tag:
- patch 1,2,7,8,9,10 from Christoph
- patch 1,2,7,8,10 from Hannes
- patch 1,2,3,7 from Xiao
v3: https://lore.kernel.org/linux-raid/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
RFC: https://lore.kernel.org/all/[email protected]/
Branch for review:
https://git.kernel.org/pub/scm/linux/kernel/git/yukuai/linux.git/log/?h=yukuai/md-llbitmap-v4
#### Background
Redundant data is used to enhance data fault tolerance, and the storage method for redundant data vary depending on the RAID levels. And it's important to maintain the consistency of redundant data.
Bitmap is used to record which data blocks have been synchronized and which ones need to be resynchronized or recovered. Each bit in the bitmap represents a segment of data in the array. When a bit is set, it indicates that the multiple redundant copies of that data segment may not be consistent. Data synchronization can be performed based on the bitmap after power failure or readding a disk. If there is no bitmap, a full disk synchronization is required.
#### Key Features
- IO fastpath is lockless, if user issues lots of write IO to the same bitmap bit in a short time, only the first write have additional overhead to update bitmap bit, no additional overhead for the following writes;
- support only resync or recover written data, means in the case creating new array or replacing with a new disk, there is no need to do a full disk resync/recovery;
#### Key Concept
##### State Machine
Each bit is one byte, contain 6 difference state, see llbitmap_state. And there are total 8 differenct actions, see llbitmap_action, can change state:
llbitmap state machine: transitions between states
| | Startwrite | Startsync | Endsync | Abortsync| | --------- | ---------- | --------- | ------- | ------- | | Unwritten | Dirty | x | x | x | | Clean | Dirty | x | x | x | | Dirty | x | x | x | x | | NeedSync | x | Syncing | x | x | | Syncing | x | Syncing | Dirty | NeedSync |
| | Reload | Daemon | Discard | Stale | | --------- | -------- | ------ | --------- | --------- | | Unwritten | x | x | x | x | | Clean | x | x | Unwritten | NeedSync | | Dirty | NeedSync | Clean | Unwritten | NeedSync | | NeedSync | x | x | Unwritten | x | | Syncing | NeedSync | x | Unwritten | NeedSync |
Typical scenarios:
1) Create new array
All bits will be set to Unwritten by default, if --assume-clean is set, all bits will be set to Clean instead.
2) write data, raid1/raid10 have full copy of data, while raid456 doesn't and rely on xor data
2.1) write new data to raid1/raid10:
Unwritten --StartWrite--> Dirty
2.2) write new data to raid456:
Unwritten --StartWrite--> NeedSync
Because the initial recover for raid456 is skipped, the xor data is not build yet, the bit must set to NeedSync first and after lazy initial recover is finished, the bit will finially set to Dirty(see 5.1 and 5.4);
2.3) cover write
Clean --StartWrite--> Dirty
3) daemon, if the array is not degraded:
Dirty --Daemon--> Clean
For degraded array, the Dirty bit will never be cleared, prevent full disk recovery while readding a removed disk.
4) discard
{Clean, Dirty, NeedSync, Syncing} --Discard--> Unwritten
5) resync and recover
5.1) common process
NeedSync --Startsync--> Syncing --Endsync--> Dirty --Daemon--> Clean
5.2) resync after power failure
Dirty --Reload--> NeedSync
5.3) recover while replacing with a new disk
By default, the old bitmap framework will recover all data, and llbitmap implement this by a new helper, see llbitmap_skip_sync_blocks:
skip recover for bits other than dirty or clean;
5.4) lazy initial recover for raid5:
By default, the old bitmap framework will only allow new recover when there are spares(new disk), a new recovery flag MD_RECOVERY_LAZY_RECOVER is add to perform raid456 lazy recover for set bits(from 2.2).
##### Bitmap IO
##### Chunksize
The default bitmap size is 128k, incluing 1k bitmap super block, and the default size of segment of data in the array each bit(chunksize) is 64k, and chunksize will adjust to twice the old size each time if the total number bits is not less than 127k.(see llbitmap_init)
##### READ
While creating bitmap, all pages will be allocated and read for llbitmap, there won't be read afterwards
##### WRITE
WRITE IO is divided into logical_block_size of the array, the dirty state of each block is tracked independently, for example:
each page is 4k, contain 8 blocks; each block is 512 bytes contain 512 bit;
| page0 | page1 | ... | page 31 | | | | \-----------------------\ | | | block0 | block1 | ... | block 8| | | | \-----------------\ | | | bit0 | bit1 | ... | bit511 |
Quoted 1 line
From IO path, if one bit is changed to Dirty or NeedSync, the corresponding
subpage will be marked dirty, such block must write first before the IO is issued. This behaviour will affect IO performance, to reduce the impact, if multiple bits are changed in the same block in a short time, all bits in this block will be changed to Dirty/NeedSync, so that there won't be any overhead until daemon clears dirty bits.
##### Dirty Bits syncronization
IO fast path will set bits to dirty, and those dirty bits will be cleared by daemon after IO is done. llbitmap_page_ctl is used to synchronize between IO path and daemon;
IO path:
1) try to grab a reference, if succeed, set expire time after 5s and return;
2) if failed to grab a reference, wait for daemon to finish clearing dirty bits;
Daemon(Daemon will be waken up every daemon_sleep seconds):
For each page:
1) check if page expired, if not skip this page; for expired page:
2) suspend the page and wait for inflight write IO to be done;
3) change dirty page to clean;
4) resume the page;
Performance Test:
Simple fio randwrite test to build array with 20GB ramdisk in my VM:
| | none | bitmap | llbitmap | | -------------------- | --------- | --------- | --------- | | raid1 | 13.7MiB/s | 9696KiB/s | 19.5MiB/s | | raid1(assume clean) | 19.5MiB/s | 11.9MiB/s | 19.5MiB/s | | raid10 | 21.9MiB/s | 11.6MiB/s | 27.8MiB/s | | raid10(assume clean) | 27.8MiB/s | 15.4MiB/s | 27.8MiB/s | | raid5 | 14.0MiB/s | 11.6MiB/s | 12.9MiB/s | | raid5(assume clean) | 17.8MiB/s | 13.4MiB/s | 13.9MiB/s |
For raid1/raid10 llbitmap can be better than none bitmap with background initial resync, and it's the same as none bitmap without it.
Noted that llbitmap performance improvement for raid5 is not obvious, this is due to raid5 has many other performance bottleneck, perf results still shows that bitmap overhead will be much less.
Yu Kuai (11):
md: add a new parameter 'offset' to md_super_write()
md: factor out a helper raid_is_456()
md/md-bitmap: support discard for bitmap ops
md: add a new mddev field 'bitmap_id'
md/md-bitmap: add a new sysfs api bitmap_type md/md-bitmap: delay registration of bitmap_ops until creating bitmap md/md-bitmap: add a new method skip_sync_blocks() in bitmap_operations md/md-bitmap: add a new method blocks_synced() in bitmap_operations
md: add a new recovery_flag MD_RECOVERY_LAZY_RECOVER md/md-bitmap: make method bitmap_ops->daemon_work optional md/md-llbitmap: introduce new lockless bitmap
Documentation/admin-guide/md.rst | 86 +- drivers/md/Kconfig | 11 + drivers/md/Makefile | 1 + drivers/md/md-bitmap.c | 15 +- drivers/md/md-bitmap.h | 45 +- drivers/md/md-llbitmap.c | 1598 ++++++++++++++++++++++++++++++ drivers/md/md.c | 288 ++++-- drivers/md/md.h | 20 +- drivers/md/raid5.c | 6 + 9 files changed, 1963 insertions(+), 107 deletions(-) create mode 100644 drivers/md/md-llbitmap.c
Signature · 1 line
2.43.0
PATCH v4 1/11 md: add a new parameter 'offset' to md_super_write()
From: Yu Kuai <[email protected]>
The parameter is always set to 0 for now, following patches will use this helper to write llbitmap to underlying disks, allow writing dirty sectors instead of the whole page.
Also rename md_super_write to md_write_metadata since there is nothing super-block specific.
--- drivers/md/md-bitmap.c | 3 ++- drivers/md/md.c | 52 +++++++++++++++++++++++++----------------- drivers/md/md.h | 5 ++-- 3 files changed, 36 insertions(+), 24 deletions(-)
drivers/md/md-bitmap.c +2 -1
@@ -485,7 +485,8 @@ static int __write_sb_page(struct md_rdev *rdev, struct bitmap *bitmap,return -EINVAL ; } -md_super_write (mddev, rdev, sboff + ps, (int )min (size, bitmap_limit), page); +md_write_metadata (mddev, rdev, sboff + ps, (int )min (size, bitmap_limit), + page,0 );return 0 ; }
drivers/md/md.c +31 -21
87 lines not on this page · raw message
drivers/md/md.h +3 -2
@@ -908,8 +908,9 @@ void md_account_bio(struct mddev *mddev, struct bio **bio);void md_free_cloned_bio (struct bio *bio);extern bool __must_check md_flush_request (struct mddev *mddev,struct bio *bio); -extern void md_super_write (struct mddev *mddev,struct md_rdev *rdev, -sector_t sector,int size,struct page *page); +void md_write_metadata (struct mddev *mddev,struct md_rdev *rdev, +sector_t sector,int size,struct page *page, +unsigned int offset);extern int md_super_wait (struct mddev *mddev);extern int sync_page_io (struct md_rdev *rdev,sector_t sector,int size,struct page *page,blk_opf_t opf,bool metadata_op);
Signature · 1 line
2.43.0
PATCH v4 2/11 md: factor out a helper raid_is_456()
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
There are no functional changes, the helper will be used by llbitmap in following patches.
--- drivers/md/md.c | 9 +-------- drivers/md/md.h | 6 ++++++ 2 files changed, 7 insertions(+), 8 deletions(-)
drivers/md/md.c +1 -8
@@ -9050,19 +9050,12 @@ static sector_t md_sync_position(struct mddev *mddev, enum sync_action action)static bool sync_io_within_limit (struct mddev *mddev) { -int io_sectors; -/* * For raid456, sync IO is stripe(4k) per IO, for other levels, it's * RESYNC_PAGES(64k) per IO. */ -if (mddev->level ==4 || mddev->level ==5 || mddev->level ==6 ) - io_sectors =8 ; -else - io_sectors =128 ; -return atomic_read (&mddev->recovery_active) < - io_sectors *sync_io_depth (mddev); + (raid_is_456 (mddev) ?8 :128 ) *sync_io_depth (mddev); }#define SYNC_MARKS 10
drivers/md/md.h +6 -0
@@ -1033,6 +1033,12 @@ static inline bool mddev_is_dm(struct mddev *mddev)return !mddev->gendisk; } +static inline bool raid_is_456 (struct mddev *mddev) +{ +return mddev->level ==ID_RAID4 || mddev->level ==ID_RAID5 || + mddev->level ==ID_RAID6 ; +} +static inline void mddev_trace_remap (struct mddev *mddev,struct bio *bio,sector_t sector) {
Signature · 1 line
2.43.0
PATCH v4 3/11 md/md-bitmap: support discard for bitmap ops
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Use two new methods {start, end}_discard in bitmap_ops and a new field 'rw' in struct md_io_clone to handle discard IO, prepare to support new md bitmap.
Since all bitmap functions to hanlde write IO are the same, also add typedef to make code cleaner.
--- drivers/md/md-bitmap.c | 3 +++ drivers/md/md-bitmap.h | 12 ++++++++---- drivers/md/md.c | 15 +++++++++++---- drivers/md/md.h | 1 + 4 files changed, 23 insertions(+), 8 deletions(-)
drivers/md/md-bitmap.c +3 -0
@@ -3006,6 +3006,9 @@ static struct bitmap_operations bitmap_ops = { .start_write = bitmap_start_write, .end_write = bitmap_end_write, + .start_discard = bitmap_start_write, + .end_discard = bitmap_end_write, + .start_sync = bitmap_start_sync, .end_sync = bitmap_end_sync, .cond_end_sync = bitmap_cond_end_sync,
drivers/md/md-bitmap.h +8 -4
@@ -61,6 +61,9 @@ struct md_bitmap_stats {struct file *file; }; +typedef void (md_bitmap_fn )(struct mddev *mddev,sector_t offset, +unsigned long sectors); +struct bitmap_operations {struct md_submodule_head head;
@@ -81,10 +84,11 @@ struct bitmap_operations {void (*end_behind_write)(struct mddev *mddev);void (*wait_behind_writes)(struct mddev *mddev); -void (*start_write)(struct mddev *mddev,sector_t offset, -unsigned long sectors); -void (*end_write)(struct mddev *mddev,sector_t offset, -unsigned long sectors); +md_bitmap_fn *start_write; +md_bitmap_fn *end_write; +md_bitmap_fn *start_discard; +md_bitmap_fn *end_discard; +bool (*start_sync)(struct mddev *mddev,sector_t offset,sector_t *blocks,bool degraded);void (*end_sync)(struct mddev *mddev,sector_t offset,sector_t *blocks);
drivers/md/md.c +11 -4
@@ -8858,18 +8858,24 @@ EXPORT_SYMBOL_GPL(md_submit_discard_bio);static void md_bitmap_start (struct mddev *mddev,struct md_io_clone *md_io_clone) { +md_bitmap_fn *fn =unlikely (md_io_clone->rw ==STAT_DISCARD ) ? + mddev->bitmap_ops->start_discard : + mddev->bitmap_ops->start_write; +if (mddev->pers->bitmap_sector) mddev->pers->bitmap_sector (mddev, &md_io_clone->offset, &md_io_clone->sectors); - mddev->bitmap_ops->start_write (mddev, md_io_clone->offset, - md_io_clone->sectors); +fn (mddev, md_io_clone->offset, md_io_clone->sectors); }static void md_bitmap_end (struct mddev *mddev,struct md_io_clone *md_io_clone) { - mddev->bitmap_ops->end_write (mddev, md_io_clone->offset, - md_io_clone->sectors); +md_bitmap_fn *fn =unlikely (md_io_clone->rw ==STAT_DISCARD ) ? + mddev->bitmap_ops->end_discard : + mddev->bitmap_ops->end_write; + +fn (mddev, md_io_clone->offset, md_io_clone->sectors); }static void md_end_clone_io (struct bio *bio)
@@ -8908,6 +8914,7 @@ static void md_clone_bio(struct mddev *mddev, struct bio **bio)if (bio_data_dir (*bio) ==WRITE &&md_bitmap_enabled (mddev, false)) { md_io_clone->offset = (*bio)->bi_iter.bi_sector; md_io_clone->sectors =bio_sectors (*bio); + md_io_clone->rw =op_stat_group (bio_op (*bio));md_bitmap_start (mddev, md_io_clone); }
drivers/md/md.h +1 -0
@@ -872,6 +872,7 @@ struct md_io_clone {unsigned long start_time;sector_t offset;unsigned long sectors; +enum stat_group rw;struct bio bio_clone; };
Signature · 1 line
2.43.0
PATCH v4 4/11 md: add a new mddev field 'bitmap_id'
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Prepare to store the bitmap id selected by user, also refactor mddev_set_bitmap_ops a bit in case the value is invalid.
--- drivers/md/md.c | 37 +++++++++++++++++++++++++++++++------ drivers/md/md.h | 2 ++ 2 files changed, 33 insertions(+), 6 deletions(-)
drivers/md/md.c +31 -6
@@ -669,13 +669,33 @@ static void active_io_release(struct percpu_ref *ref)static void no_op (struct percpu_ref *r) {} -static void mddev_set_bitmap_ops (struct mddev *mddev,enum md_submodule_id id) +static bool mddev_set_bitmap_ops (struct mddev *mddev) { +struct md_submodule_head *head; + +if (mddev->bitmap_id ==ID_BITMAP_NONE ) +return true; +xa_lock (&md_submodule); - mddev->bitmap_ops =xa_load (&md_submodule, id); + head =xa_load (&md_submodule, mddev->bitmap_id); + +if (!head) { +pr_warn ("md: can't find bitmap id %d\n" , mddev->bitmap_id); + goto err; + } + +if (head->type !=MD_BITMAP ) { +pr_warn ("md: invalid bitmap id %d\n" , mddev->bitmap_id); + goto err; + } + + mddev->bitmap_ops = (void *)head;xa_unlock (&md_submodule); -if (!mddev->bitmap_ops) -pr_warn_once ("md: can't find bitmap id %d\n" , id); +return true; + +err: +xa_unlock (&md_submodule); +return false; }static void mddev_clear_bitmap_ops (struct mddev *mddev)
@@ -685,8 +705,13 @@ static void mddev_clear_bitmap_ops(struct mddev *mddev)int mddev_init (struct mddev *mddev) { -/* TODO: support more versions */ -mddev_set_bitmap_ops (mddev,ID_BITMAP ); +if (!IS_ENABLED (CONFIG_MD_BITMAP )) { + mddev->bitmap_id =ID_BITMAP_NONE ; + }else { + mddev->bitmap_id =ID_BITMAP ; +if (!mddev_set_bitmap_ops (mddev)) +return -EINVAL ; + }if (percpu_ref_init (&mddev->active_io, active_io_release,PERCPU_REF_ALLOW_REINIT ,GFP_KERNEL )) {
drivers/md/md.h +2 -0
@@ -40,6 +40,7 @@ enum md_submodule_id {ID_CLUSTER ,ID_BITMAP ,ID_LLBITMAP ,/* TODO */ + ID_BITMAP_NONE, }; struct md_submodule_head {
@@ -565,6 +566,7 @@ struct mddev {struct percpu_ref writes_pending;int sync_checkers;/* # of threads checking writes_pending */ +enum md_submodule_id bitmap_id;void *bitmap;/* the bitmap for the device */ struct bitmap_operations *bitmap_ops;struct {
Signature · 1 line
2.43.0
PATCH v4 5/11 md/md-bitmap: add a new sysfs api bitmap_type
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
The api will be used by mdadm to set bitmap_type while creating new array or assembling array, prepare to add a new bitmap.
Currently available options are:
cat /sys/block/md0/md/bitmap_type
none [bitmap]
--- Documentation/admin-guide/md.rst | 73 ++++++++++++++++------------ drivers/md/md.c | 81 ++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+), 30 deletions(-)
Documentation/admin-guide/md.rst +43 -30
@@ -347,6 +347,49 @@ All md devices contain: active-idle like active, but no writes have been seen for a while (safe_mode_delay). + consistency_policy + This indicates how the array maintains consistency in case of unexpected + shutdown. It can be: + + none + Array has no redundancy information, e.g. raid0, linear. + + resync + Full resync is performed and all redundancy is regenerated when the + array is started after unclean shutdown. + + bitmap + Resync assisted by a write-intent bitmap. + + journal + For raid4/5/6, journal device is used to log transactions and replay + after unclean shutdown. + + ppl + For raid5 only, Partial Parity Log is used to close the write hole and + eliminate resync. + + The accepted values when writing to this file are ``ppl`` and ``resync``, + used to enable and disable PPL. + + uuid + This indicates the UUID of the array in the following format: + xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + + bitmap_type + [RW] When read, this file will display the current and available + bitmap for this array. The currently active bitmap will be enclosed + in [] brackets. Writing an bitmap name or ID to this file will switch + control of this array to that new bitmap. Note that writing a new + bitmap for created array is forbidden. + + none + No bitmap + bitmap + The default internal bitmap + +If bitmap_type is bitmap, then the md device will also contain: + bitmap/location This indicates where the write-intent bitmap for the array is stored.
@@ -401,36 +444,6 @@ All md devices contain: once the array becomes non-degraded, and this fact has been recorded in the metadata. - consistency_policy - This indicates how the array maintains consistency in case of unexpected - shutdown. It can be: - - none - Array has no redundancy information, e.g. raid0, linear. - - resync - Full resync is performed and all redundancy is regenerated when the - array is started after unclean shutdown. - - bitmap - Resync assisted by a write-intent bitmap. - - journal - For raid4/5/6, journal device is used to log transactions and replay - after unclean shutdown. - - ppl - For raid5 only, Partial Parity Log is used to close the write hole and - eliminate resync. - - The accepted values when writing to this file are ``ppl`` and ``resync``, - used to enable and disable PPL. - - uuid - This indicates the UUID of the array in the following format: - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx - - As component devices are added to an md array, they appear in the ``md`` directory as new directories named::
drivers/md/md.c +81 -0
93 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v4 6/11 md/md-bitmap: delay registration of bitmap_ops until creating bitmap
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Currently bitmap_ops is registered while allocating mddev, this is fine when there is only one bitmap_ops.
Delay setting bitmap_ops until creating bitmap, so that user can choose which bitmap to use before running the array.
--- Documentation/admin-guide/md.rst | 3 ++ drivers/md/md.c | 81 ++++++++++++++++++++------------ 2 files changed, 55 insertions(+), 29 deletions(-)
Documentation/admin-guide/md.rst +3 -0
@@ -388,6 +388,9 @@ All md devices contain: bitmap The default internal bitmap +If bitmap_type is not none, then additional bitmap attributes bitmap/xxx or +llbitmap/xxx will be created after md device KOBJ_CHANGE event. + If bitmap_type is bitmap, then the md device will also contain: bitmap/location
drivers/md/md.c +52 -29
186 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v4 06/11] md/md-bitmap: delay registration of bitmap_ops until creating bitmap
Quoted 5 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Currently
bitmap_opsis registered while allocating mddev, this is fine when there is only onebitmap_ops.
Quoted 9 lines
Delay setting
bitmap_opsuntil creating bitmap, so that user can choose which bitmap to use before running the array.Signed-off-by Yu Kuai <[email protected]>--- Documentation/admin-guide/md.rst | 3 ++ drivers/md/md.c | 81 ++++++++++++++++++++------------ 2 files changed, 55 insertions(+), 29 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v4 7/11 md/md-bitmap: add a new method skip_sync_blocks() in bitmap_operations
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
This method is used to check if blocks can be skipped before calling into pers->sync_request(), llbitmap will use this method to skip resync for unwritten/clean data blocks, and recovery/check/repair for unwritten data blocks;
--- drivers/md/md-bitmap.h | 1 + drivers/md/md.c | 7 +++++++ 2 files changed, 8 insertions(+)
drivers/md/md-bitmap.h +1 -0
@@ -89,6 +89,7 @@ struct bitmap_operations {md_bitmap_fn *start_discard;md_bitmap_fn *end_discard; +sector_t (*skip_sync_blocks)(struct mddev *mddev,sector_t offset);bool (*start_sync)(struct mddev *mddev,sector_t offset,sector_t *blocks,bool degraded);void (*end_sync)(struct mddev *mddev,sector_t offset,sector_t *blocks);
drivers/md/md.c +7 -0
@@ -9396,6 +9396,12 @@ void md_do_sync(struct md_thread *thread)if (test_bit (MD_RECOVERY_INTR , &mddev->recovery))break ; +if (mddev->bitmap_ops && mddev->bitmap_ops->skip_sync_blocks) { + sectors = mddev->bitmap_ops->skip_sync_blocks (mddev, j); +if (sectors) + goto update; + } + sectors = mddev->pers->sync_request (mddev, j, max_sectors, &skipped);if (sectors ==0 ) {
@@ -9411,6 +9417,7 @@ void md_do_sync(struct md_thread *thread)if (test_bit (MD_RECOVERY_INTR , &mddev->recovery))break ; +update: j += sectors;if (j > max_sectors)/* when skipping, extra large numbers can be returned. */
Signature · 1 line
2.43.0
PATCH v4 8/11 md/md-bitmap: add a new method blocks_synced() in bitmap_operations
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Currently, raid456 must perform a whole array initial recovery to build initail xor data, then IO to the array won't have to read all the blocks in underlying disks.
This behavior will affect IO performance a lot, and nowadays there are huge disks and the initial recovery can take a long time. Hence llbitmap will support lazy initial recovery in following patches. This method is used to check if data blocks is synced or not, if not then IO will still have to read all blocks for raid456.
--- drivers/md/md-bitmap.h | 1 + drivers/md/raid5.c | 6 ++++++ 2 files changed, 7 insertions(+)
drivers/md/md-bitmap.h +1 -0
@@ -90,6 +90,7 @@ struct bitmap_operations {md_bitmap_fn *end_discard;sector_t (*skip_sync_blocks)(struct mddev *mddev,sector_t offset); +bool (*blocks_synced)(struct mddev *mddev,sector_t offset);bool (*start_sync)(struct mddev *mddev,sector_t offset,sector_t *blocks,bool degraded);void (*end_sync)(struct mddev *mddev,sector_t offset,sector_t *blocks);
drivers/md/raid5.c +6 -0
@@ -3748,6 +3748,7 @@ static int want_replace(struct stripe_head *sh, int disk_idx)static int need_this_block (struct stripe_head *sh,struct stripe_head_state *s,int disk_idx,int disks) { +struct mddev *mddev = sh->raid_conf->mddev;struct r5dev *dev = &sh->dev[disk_idx];struct r5dev *fdev[2 ] = { &sh->dev[s->failed_num[0 ]], &sh->dev[s->failed_num[1 ]] };
@@ -3762,6 +3763,11 @@ static int need_this_block(struct stripe_head *sh, struct stripe_head_state *s,*/ return 0 ; +/* The initial recover is not done, must read everything */ +if (mddev->bitmap_ops && mddev->bitmap_ops->blocks_synced && + !mddev->bitmap_ops->blocks_synced (mddev, sh->sector)) +return 1 ; +if (dev->toread || (dev->towrite && !test_bit (R5_OVERWRITE , &dev->flags)))/* We need this block to directly satisfy a request */
Signature · 1 line
2.43.0
PATCH v4 9/11 md: add a new recovery_flag MD_RECOVERY_LAZY_RECOVER
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
This flag is used by llbitmap in later patches to skip raid456 initial recover and delay building initial xor data to first write.
--- drivers/md/md.c | 12 +++++++++++- drivers/md/md.h | 2 ++ 2 files changed, 13 insertions(+), 1 deletion(-)
drivers/md/md.c +11 -1
@@ -9166,6 +9166,14 @@ static sector_t md_sync_position(struct mddev *mddev, enum sync_action action) start = rdev->recovery_offset;rcu_read_unlock (); +/* +* If there are no spares, and raid456 lazy initial recover is +* requested. +*/ +if (test_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery) && + start == MaxSector) + start =0 ; +/* If there is a bitmap, we need to make sure all * writes that started before we added a spare * complete before we start doing a recovery.
@@ -9723,6 +9731,7 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)if (mddev->recovery_cp < MaxSector) {set_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_RECOVER , &mddev->recovery); +clear_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery);return true; }
@@ -9732,7 +9741,7 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)* re-add. */ *spares =remove_and_add_spares (mddev,NULL ); -if (*spares) { +if (*spares ||test_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery)) {clear_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_CHECK , &mddev->recovery);clear_bit (MD_RECOVERY_REQUESTED , &mddev->recovery);
@@ -10055,6 +10064,7 @@ void md_reap_sync_thread(struct mddev *mddev)clear_bit (MD_RECOVERY_RESHAPE , &mddev->recovery);clear_bit (MD_RECOVERY_REQUESTED , &mddev->recovery);clear_bit (MD_RECOVERY_CHECK , &mddev->recovery); +clear_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery);/* * We call mddev->cluster_ops->update_size here because sync_size could * be changed by md_update_sb, and MD_RECOVERY_RESHAPE is cleared,
drivers/md/md.h +2 -0
@@ -667,6 +667,8 @@ enum recovery_flags {MD_RECOVERY_RESHAPE ,/* remote node is running resync thread */ MD_RESYNCING_REMOTE , +/* raid456 lazy initial recover */ + MD_RECOVERY_LAZY_RECOVER, }; enum md_ro_state {
Signature · 1 line
2.43.0
PATCH v4 10/11 md/md-bitmap: make method bitmap_ops->daemon_work optional
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
daemon_work() will be called by daemon thread, on the one hand, daemon thread doesn't have strict wake-up time; on the other hand, too much work are put to daemon thread, like handle sync IO, handle failed or specail normal IO, handle recovery, and so on. Hence daemon thread may be too busy to clear dirty bits in time.
Make bitmap_ops->daemon_work() optional and following patches will use separate async work to clear dirty bits for the new bitmap.
--- drivers/md/md.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/md/md.c +1 -1
@@ -9887,7 +9887,7 @@ static void unregister_sync_thread(struct mddev *mddev)*/ void md_check_recovery (struct mddev *mddev) { -if (md_bitmap_enabled (mddev, false)) +if (md_bitmap_enabled (mddev, false) && mddev->bitmap_ops->daemon_work) mddev->bitmap_ops->daemon_work (mddev);if (signal_pending (current)) {
Signature · 1 line
2.43.0
PATCH v4 11/11 md/md-llbitmap: introduce new lockless bitmap
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Redundant data is used to enhance data fault tolerance, and the storage method for redundant data vary depending on the RAID levels. And it's important to maintain the consistency of redundant data.
Bitmap is used to record which data blocks have been synchronized and which ones need to be resynchronized or recovered. Each bit in the bitmap represents a segment of data in the array. When a bit is set, it indicates that the multiple redundant copies of that data segment may not be consistent. Data synchronization can be performed based on the bitmap after power failure or readding a disk. If there is no bitmap, a full disk synchronization is required.
Key Features:
- IO fastpath is lockless, if user issues lots of write IO to the same bitmap bit in a short time, only the first write have additional overhead to update bitmap bit, no additional overhead for the following writes;
- support only resync or recover written data, means in the case creating new array or replacing with a new disk, there is no need to do a full disk resync/recovery;
Key Concept:
- State Machine:
Each bit is one byte, contain 6 difference state, see llbitmap_state. And there are total 8 differenct actions, see llbitmap_action, can change state:
llbitmap state machine: transitions between states
| | Startwrite | Startsync | Endsync | Abortsync| | --------- | ---------- | --------- | ------- | ------- | | Unwritten | Dirty | x | x | x | | Clean | Dirty | x | x | x | | Dirty | x | x | x | x | | NeedSync | x | Syncing | x | x | | Syncing | x | Syncing | Dirty | NeedSync |
| | Reload | Daemon | Discard | Stale | | --------- | -------- | ------ | --------- | --------- | | Unwritten | x | x | x | x | | Clean | x | x | Unwritten | NeedSync | | Dirty | NeedSync | Clean | Unwritten | NeedSync | | NeedSync | x | x | Unwritten | x | | Syncing | NeedSync | x | Unwritten | NeedSync |
Typical scenarios:
1) Create new array
All bits will be set to Unwritten by default, if --assume-clean is set, all bits will be set to Clean instead.
2) write data, raid1/raid10 have full copy of data, while raid456 doesn't and rely on xor data
2.1) write new data to raid1/raid10:
Unwritten --StartWrite--> Dirty
2.2) write new data to raid456:
Unwritten --StartWrite--> NeedSync
Because the initial recover for raid456 is skipped, the xor data is not build yet, the bit must set to NeedSync first and after lazy initial recover is finished, the bit will finially set to Dirty(see 5.1 and 5.4);
2.3) cover write
Clean --StartWrite--> Dirty
3) daemon, if the array is not degraded:
Dirty --Daemon--> Clean
For degraded array, the Dirty bit will never be cleared, prevent full disk recovery while readding a removed disk.
4) discard
{Clean, Dirty, NeedSync, Syncing} --Discard--> Unwritten
5) resync and recover
5.1) common process
NeedSync --Startsync--> Syncing --Endsync--> Dirty --Daemon--> Clean
5.2) resync after power failure
Dirty --Reload--> NeedSync
5.3) recover while replacing with a new disk
By default, the old bitmap framework will recover all data, and llbitmap implement this by a new helper, see llbitmap_skip_sync_blocks:
skip recover for bits other than dirty or clean;
5.4) lazy initial recover for raid5:
By default, the old bitmap framework will only allow new recover when there are spares(new disk), a new recovery flag MD_RECOVERY_LAZY_RECOVER is add to perform raid456 lazy recover for set bits(from 2.2).
Bitmap IO:
- Chunksize
The default bitmap size is 128k, incluing 1k bitmap super block, and the default size of segment of data in the array each bit(chunksize) is 64k, and chunksize will adjust to twice the old size each time if the total number bits is not less than 127k.(see llbitmap_init)
- READ
While creating bitmap, all pages will be allocated and read for llbitmap, there won't be read afterwards
- WRITE
WRITE IO is divided into logical_block_size of the array, the dirty state of each block is tracked independently, for example:
each page is 4k, contain 8 blocks; each block is 512 bytes contain 512 bit;
| page0 | page1 | ... | page 31 | | | | \-----------------------\ | | | block0 | block1 | ... | block 8| | | | \-----------------\ | | | bit0 | bit1 | ... | bit511 |
From IO path, if one bit is changed to Dirty or NeedSync, the corresponding subpage will be marked dirty, such block must write first before the IO is issued. This behaviour will affect IO performance, to reduce the impact, if multiple bits are changed in the same block in a short time, all bits in this block will be changed to Dirty/NeedSync, so that there won't be any overhead until daemon clears dirty bits.
Dirty Bits syncronization:
IO fast path will set bits to dirty, and those dirty bits will be cleared by daemon after IO is done. llbitmap_page_ctl is used to synchronize between IO path and daemon;
IO path:
1) try to grab a reference, if succeed, set expire time after 5s and return;
2) if failed to grab a reference, wait for daemon to finish clearing dirty bits;
Daemon(Daemon will be waken up every daemon_sleep seconds):
For each page:
1) check if page expired, if not skip this page; for expired page:
2) suspend the page and wait for inflight write IO to be done;
3) change dirty page to clean;
4) resume the page;
--- Documentation/admin-guide/md.rst | 20 + drivers/md/Kconfig | 11 + drivers/md/Makefile | 1 + drivers/md/md-bitmap.c | 9 - drivers/md/md-bitmap.h | 31 +- drivers/md/md-llbitmap.c | 1598 ++++++++++++++++++++++++++++++ drivers/md/md.c | 6 + drivers/md/md.h | 4 +- 8 files changed, 1668 insertions(+), 12 deletions(-) create mode 100644 drivers/md/md-llbitmap.c
Documentation/admin-guide/md.rst +20 -0
@@ -387,6 +387,8 @@ All md devices contain: No bitmap bitmap The default internal bitmap + llbitmap + The lockless internal bitmap If bitmap_type is not none, then additional bitmap attributes bitmap/xxx or llbitmap/xxx will be created after md device KOBJ_CHANGE event.
@@ -447,6 +449,24 @@ If bitmap_type is bitmap, then the md device will also contain: once the array becomes non-degraded, and this fact has been recorded in the metadata. +If bitmap_type is llbitmap, then the md device will also contain: + + llbitmap/bits + This is readonly, show status of bitmap bits, the number of each + value. + + llbitmap/metadata + This is readonly, show bitmap metadata, include chunksize, chunkshift, + chunks, offset and daemon_sleep. + + llbitmap/daemon_sleep + This is readwrite, time in seconds that daemon function will be + triggered to clear dirty bits. + + llbitmap/barrier_idle + This is readwrite, time in seconds that page barrier will be idled, + means dirty bits in the page will be cleared. + As component devices are added to an md array, they appear in the ``md`` directory as new directories named::
drivers/md/Kconfig +11 -0
@@ -52,6 +52,17 @@ config MD_BITMAP If unsure, say Y. +config MD_LLBITMAP + bool "MD RAID lockless bitmap support" + depends on BLK_DEV_MD + help + If you say Y here, support for the lockless write intent bitmap will + be enabled. + + Note, this is an experimental feature. + + If unsure, say N. + config MD_AUTODETECT bool "Autodetect RAID arrays during kernel boot" depends on BLK_DEV_MD=y
drivers/md/Makefile +1 -0
@@ -29,6 +29,7 @@ dm-zoned-y += dm-zoned-target.o dm-zoned-metadata.o dm-zoned-reclaim.o md-mod-y += md.o md-mod-$(CONFIG_MD_BITMAP) += md-bitmap.o +md-mod-$(CONFIG_MD_LLBITMAP) += md-llbitmap.o raid456-y += raid5.o raid5-cache.o raid5-ppl.o linear-y += md-linear.o
drivers/md/md-bitmap.c +0 -9
@@ -34,15 +34,6 @@#include "md-bitmap.h" #include "md-cluster.h" -#define BITMAP_MAJOR_LO 3 -/* version 4 insists the bitmap is in little-endian order -* with version 3, it is host-endian which is non-portable -* Version 5 is currently set only for clustered devices -*/ -#define BITMAP_MAJOR_HI 4 -#define BITMAP_MAJOR_CLUSTERED 5 -#define BITMAP_MAJOR_HOSTENDIAN 3 -/* * in-memory bitmap: *
drivers/md/md-bitmap.h +30 -1
@@ -9,10 +9,26 @@#define BITMAP_MAGIC 0x6d746962 +/* +* version 3 is host-endian order, this is deprecated and not used for new +* array +*/ +#define BITMAP_MAJOR_LO 3 +#define BITMAP_MAJOR_HOSTENDIAN 3 +/* version 4 is little-endian order, the default value */ +#define BITMAP_MAJOR_HI 4 +/* version 5 is only used for cluster */ +#define BITMAP_MAJOR_CLUSTERED 5 +/* version 6 is only used for lockless bitmap */ +#define BITMAP_MAJOR_LOCKLESS 6 +/* use these for bitmap->flags and bitmap->sb->state bit-fields */ enum bitmap_state { -BITMAP_STALE =1 ,/* the bitmap file is out of date or had -EIO */ +BITMAP_STALE =1 ,/* the bitmap file is out of date or had -EIO */ BITMAP_WRITE_ERROR =2 ,/* A write error has occurred */ +BITMAP_FIRST_USE =3 ,/* llbitmap is just created */ +BITMAP_CLEAN =4 ,/* llbitmap is created with assume_clean */ +BITMAP_DAEMON_BUSY =5 ,/* llbitmap daemon is not finished after daemon_sleep */ BITMAP_HOSTENDIAN =15 , };
@@ -166,4 +182,17 @@ static inline void md_bitmap_exit(void) }#endif +#ifdef CONFIG_MD_LLBITMAP +int md_llbitmap_init (void ); +void md_llbitmap_exit (void ); +#else +static inline int md_llbitmap_init (void ) +{ +return 0 ; +} +static inline void md_llbitmap_exit (void ) +{ +} +#endif +#endif
drivers/md/md-llbitmap.c new file +1598 -0
1,598 lines not on this page · raw message
drivers/md/md.c +6 -0
@@ -10217,6 +10217,10 @@ static int __init md_init(void)if (ret)return ret; + ret =md_llbitmap_init (); +if (ret) + goto err_bitmap; + ret = -ENOMEM ; md_wq =alloc_workqueue ("md" ,WQ_MEM_RECLAIM ,0 );if (!md_wq)
@@ -10248,6 +10252,8 @@ static int __init md_init(void) err_misc_wq:destroy_workqueue (md_wq); err_wq: +md_llbitmap_exit (); +err_bitmap:md_bitmap_exit ();return ret; }
drivers/md/md.h +2 -2
@@ -26,7 +26,7 @@enum md_submodule_type {MD_PERSONALITY =0 ,MD_CLUSTER , -MD_BITMAP ,/* TODO */ +MD_BITMAP , };enum md_submodule_id {
@@ -39,7 +39,7 @@ enum md_submodule_id {ID_RAID10 =10 ,ID_CLUSTER ,ID_BITMAP , -ID_LLBITMAP ,/* TODO */ +ID_LLBITMAP , ID_BITMAP_NONE, };
Signature · 1 line
2.43.0
Re: [PATCH v4 11/11] md/md-llbitmap: introduce new lockless bitmap
Hi,
Quoted 9 lines · Yu Kuai
drivers/md/md-llbitmap.c new file +9
-0+static int llbitmap_read_sb (struct llbitmap *llbitmap) +{ +struct mddev *mddev = llbitmap->mddev; +unsigned long daemon_sleep; +unsigned long chunksize; +unsigned long events; +struct page *sb_page; +bitmap_super_t *sb; +int ret = -EINVAL ;
drivers/md/md-llbitmap.c line 789
Quoted 10 lines
drivers/md/md-llbitmap.c new file +10
-0+ +if (!mddev->bitmap_info.offset) { +pr_err ("md/llbitmap: %s: no super block found" ,mdname (mddev)); +return -EINVAL ; + } + + sb_page =llbitmap_read_page (llbitmap,0 ); +if (IS_ERR (sb_page)) { +pr_err ("md/llbitmap: %s: read super block failed" , +mdname (mddev));
There should return -EIO directly here.
drivers/md/md-llbitmap.c line 791
Quoted 2 lines
drivers/md/md-llbitmap.c new file +2
-0+ ret = -EIO ; + goto out;
And the out tag can be removed.
Thanks,
Kuai
Quoted 80 lines
drivers/md/md-llbitmap.c new file +80
-0+ } + + sb =kmap_local_page (sb_page); +if (sb->magic !=cpu_to_le32 (BITMAP_MAGIC )) { +pr_err ("md/llbitmap: %s: invalid super block magic number" , +mdname (mddev)); + goto out_put_page; + } + +if (sb->version !=cpu_to_le32 (BITMAP_MAJOR_LOCKLESS )) { +pr_err ("md/llbitmap: %s: invalid super block version" , +mdname (mddev)); + goto out_put_page; + } + +if (memcmp (sb->uuid, mddev->uuid,16 )) { +pr_err ("md/llbitmap: %s: bitmap superblock UUID mismatch\n" , +mdname (mddev)); + goto out_put_page; + } + +if (mddev->bitmap_info.space ==0 ) { +int room =le32_to_cpu (sb->sectors_reserved); + +if (room) + mddev->bitmap_info.space = room; +else + mddev->bitmap_info.space = mddev->bitmap_info.default_space; + } + llbitmap->flags =le32_to_cpu (sb->state); +if (test_and_clear_bit (BITMAP_FIRST_USE , &llbitmap->flags)) { + ret =llbitmap_init (llbitmap); + goto out_put_page; + } + + chunksize =le32_to_cpu (sb->chunksize); +if (!is_power_of_2 (chunksize)) { +pr_err ("md/llbitmap: %s: chunksize not a power of 2" , +mdname (mddev)); + goto out_put_page; + } + +if (chunksize <DIV_ROUND_UP (mddev->resync_max_sectors, + mddev->bitmap_info.space <<SECTOR_SHIFT )) { +pr_err ("md/llbitmap: %s: chunksize too small %lu < %llu / %lu" , +mdname (mddev), chunksize, mddev->resync_max_sectors, + mddev->bitmap_info.space); + goto out_put_page; + } + + daemon_sleep =le32_to_cpu (sb->daemon_sleep); +if (daemon_sleep <1 || daemon_sleep >MAX_SCHEDULE_TIMEOUT /HZ ) { +pr_err ("md/llbitmap: %s: daemon sleep %lu period out of range" , +mdname (mddev), daemon_sleep); + goto out_put_page; + } + + events =le64_to_cpu (sb->events); +if (events < mddev->events) { +pr_warn ("md/llbitmap :%s: bitmap file is out of date (%lu < %llu) -- forcing full recovery" , +mdname (mddev), events, mddev->events); +set_bit (BITMAP_STALE , &llbitmap->flags); + } + + sb->sync_size =cpu_to_le64 (mddev->resync_max_sectors); + mddev->bitmap_info.chunksize = chunksize; + mddev->bitmap_info.daemon_sleep = daemon_sleep; + + llbitmap->barrier_idle =DEFAULT_BARRIER_IDLE ; + llbitmap->chunksize = chunksize; + llbitmap->chunks =DIV_ROUND_UP (mddev->resync_max_sectors, chunksize); + llbitmap->chunkshift =ffz (~chunksize); + ret =llbitmap_cache_pages (llbitmap); + +out_put_page: +__free_page (sb_page); +out: +kunmap_local (sb); +return ret; +}
Quoted 7 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Changes from v3:
- fix redundant settingmddev->bitmap_idin patch 6;
- add explanation of bitmap attributes in Documentation;
- add llbitmap/barrier_idle in patch 11;
- add some comments in patch 11;
[...]
Patches 01–10 look good:
Quoted 26 lines
Yu Kuai (11):
md: add a new parameter 'offset' tomd_super_write()
md: factor out a helperraid_is_456()
md/md-bitmap: support discard for bitmap ops
md: add a new mddev field 'bitmap_id'
md/md-bitmap: add a new sysfs apibitmap_typemd/md-bitmap: delay registration ofbitmap_opsuntil creating bitmap md/md-bitmap: add a new methodskip_sync_blocks()inbitmap_operationsmd/md-bitmap: add a new methodblocks_synced()inbitmap_operations
md: add a newrecovery_flagMD_RECOVERY_LAZY_RECOVERmd/md-bitmap: make methodbitmap_ops->daemon_workoptional md/md-llbitmap: introduce new lockless bitmapDocumentation/admin-guide/md.rst | 86 +- drivers/md/Kconfig | 11 + drivers/md/Makefile | 1 + drivers/md/md-bitmap.c | 15 +- drivers/md/md-bitmap.h | 45 +- drivers/md/md-llbitmap.c | 1598 ++++++++++++++++++++++++++++++ drivers/md/md.c | 288 ++++-- drivers/md/md.h | 20 +- drivers/md/raid5.c | 6 + 9 files changed, 1963 insertions(+), 107 deletions(-)create mode 100644
drivers/md/md-llbitmap.c
Signature · 2 lines
Thanks, Nan
Re: [PATCH v4 04/11] md: add a new mddev field 'bitmap_id'
In reply to Yu KuaiThe patch looks good to me.
Re: [PATCH v4 04/11] md: add a new mddev field 'bitmap_id'
Hi,
Quoted 4 lines · Xiao Ni
The patch looks good to me.
Reviewed-by Xiao Ni <[email protected]>
Thanks for the review :)
Kuai
Re: [PATCH v4 05/11] md/md-bitmap: add a new sysfs api bitmap_type
In reply to Yu KuaiThe patch looks good to me.
Re: [PATCH v4 06/11] md/md-bitmap: delay registration of bitmap_ops until creating bitmap
In reply to Yu KuaiThe patch looks good to me.
Re: [PATCH v4 08/11] md/md-bitmap: add a new method blocks_synced() in bitmap_operations
In reply to Yu KuaiThe patch looks good to me.