fsck.erofs: implement thread-safe global LRU metadata cache
PATCH 1/1 fsck.erofs: implement thread-safe global LRU metadata cache
This patch introduces a thread-safe metadata cache to reduce redundant I/O and decompression overhead during fsck extraction.
To ensure it remains highly concurrent for worker threads extracting pclusters, the cache utilizes a bucketed, rw-semaphore protected architecture modeled after the existing fragment cache.
Furthermore, to prevent out-of-memory (OOM) scenarios on exceptionally large EROFS images, the cache implements a strict Global Least Recently Used (LRU) eviction policy. The maximum cache size is dynamically configurable via the new '--cache-size' parameter, which defaults to a safe, fixed threshold of 32 MB.
--- fsck/main.c | 12 ++++ include/erofs/internal.h | 2 + lib/data.c | 149 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 160 insertions(+), 3 deletions(-)
fsck/main.c +12 -0
@@ -67,6 +67,7 @@ static struct option long_options[] = { {"no-xattrs", no_argument, 0, 14}, {"nid", required_argument, 0, 15}, {"path", required_argument, 0, 16}, + {"cache-size", required_argument, 0, 17}, {"no-sbcrc", no_argument, 0, 512}, {0, 0, 0, 0}, };
@@ -120,6 +121,7 @@ static void usage(int argc, char **argv)" --offset=# skip # bytes at the beginning of IMAGE\n" " --nid=# check or extract from the target inode of nid #\n" " --path=X check or extract from the target inode of path X\n" +" --cache-size=# set maximum metadata cache size in bytes (default 32MB)\n" " --no-sbcrc bypass the superblock checksum verification\n" " --[no-]xattrs whether to dump extended attributes (default off)\n" "\n"
@@ -261,6 +263,16 @@ static int erofsfsck_parse_options_cfg(int argc, char **argv)case 16 : fsckcfg.inode_path = optarg;break ; +case 17 : { +char *endptr; +unsigned long cache_size =strtoul (optarg, &endptr,0 ); +if (*endptr !='\0' ) { +erofs_err ("invalid metadata cache size %s" , optarg); +return -EINVAL ; + } +erofs_meta_cache_set_capacity (cache_size); +break ; + }case 512 : fsckcfg.nosbcrc = true;break ;
include/erofs/internal.h +2 -0
@@ -459,6 +459,8 @@ struct z_erofs_read_ctx {void z_erofs_read_ctx_enqueue (struct z_erofs_read_ctx *ctx); +void erofs_meta_cache_set_capacity (unsigned long bytes); +int liberofs_global_init (void );void liberofs_global_exit (void );
lib/data.c +146 -3
160 lines not on this page · raw message
Signature · 1 line
2.52.0
Quoted 7 lines · Nithurshen
This patch introduces a thread-safe metadata cache to reduce redundant I/O and decompression overhead during fsck extraction.
To ensure it remains highly concurrent for worker threads extracting pclusters, the cache utilizes a bucketed, rw-semaphore protected architecture modeled after the existing fragment cache.
Quoted 7 lines
Furthermore, to prevent out-of-memory (OOM) scenarios on exceptionally large EROFS images, the cache implements a strict Global Least Recently Used (LRU) eviction policy. The maximum cache size is dynamically configurable via the new '--cache-size' parameter, which defaults to a safe, fixed threshold of 32 MB.
Signed-off-by Nithurshen <[email protected]>
why malloc() will prevent out-of-memory (OOM)?
First, either erofs_read_metadata_nid() or erofs_read_metadata_bdi() will read file to the page cache; and currently there is no cache.
But you introduce another cache using malloc(), since it increases the memory overhead, why it prevents out-of-memory (OOM)?
Thanks,
Gao Xiang
PATCH v2 1/1 fsck.erofs: implement thread-safe global LRU metadata cache
In reply to NithurshenThis patch introduces a thread-safe userspace metadata cache to reduce redundant decompression cycles and the overhead of repetitive pread() syscalls across multiple background worker threads.
To ensure it remains highly concurrent for worker threads extracting pclusters, the cache utilizes a bucketed, rw-semaphore protected architecture modeled after the existing fragment cache.
While the introduction of a userspace cache inherently increases the memory footprint compared to relying solely on the kernel's page cache, this patch implements a strict Global Least Recently Used (LRU) eviction policy to safely bound this additional memory overhead. This prevents the cache from growing unbounded on exceptionally large EROFS images. The maximum cache capacity is dynamically configurable via the new '--cache-size' parameter, which defaults to a safe threshold of 32 MB.
--- fsck/main.c | 12 ++++ include/erofs/internal.h | 2 + lib/data.c | 150 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 161 insertions(+), 3 deletions(-)
fsck/main.c +12 -0
@@ -67,6 +67,7 @@ static struct option long_options[] = { {"no-xattrs", no_argument, 0, 14}, {"nid", required_argument, 0, 15}, {"path", required_argument, 0, 16}, + {"cache-size", required_argument, 0, 17}, {"no-sbcrc", no_argument, 0, 512}, {0, 0, 0, 0}, };
@@ -120,6 +121,7 @@ static void usage(int argc, char **argv)" --offset=# skip # bytes at the beginning of IMAGE\n" " --nid=# check or extract from the target inode of nid #\n" " --path=X check or extract from the target inode of path X\n" +" --cache-size=# set maximum metadata cache size in bytes (default 32MB)\n" " --no-sbcrc bypass the superblock checksum verification\n" " --[no-]xattrs whether to dump extended attributes (default off)\n" "\n"
@@ -261,6 +263,16 @@ static int erofsfsck_parse_options_cfg(int argc, char **argv)case 16 : fsckcfg.inode_path = optarg;break ; +case 17 : { +char *endptr; +unsigned long cache_size =strtoul (optarg, &endptr,0 ); +if (*endptr !='\0' ) { +erofs_err ("invalid metadata cache size %s" , optarg); +return -EINVAL ; + } +erofs_meta_cache_set_capacity (cache_size); +break ; + }case 512 : fsckcfg.nosbcrc = true;break ;
include/erofs/internal.h +2 -0
@@ -459,6 +459,8 @@ struct z_erofs_read_ctx {void z_erofs_read_ctx_enqueue (struct z_erofs_read_ctx *ctx); +void erofs_meta_cache_set_capacity (unsigned long bytes); +int liberofs_global_init (void );void liberofs_global_exit (void );
lib/data.c +147 -3
161 lines not on this page · raw message
Signature · 1 line
2.52.0
Hi Xiang,
What if rather than caching the raw metadata bytes, the userspace cache will focus on caching the parsed structures (such as the computed erofs_map_blocks mappings) to save the worker threads from redundantly parsing chunk indexes and calculating offsets.
Can I replace this malloc-based raw buffer approach and focus v3 patch on an LRU cache for the parsed block mappings to reduce the CPU overhead during the concurrent directory traversal?
Thanks,
Nithurshen
Quoted 10 lines · Nithurshen
Hi Xiang,
What if rather than caching the raw metadata bytes, the userspace cache will focus on caching the parsed structures (such as the computed
erofs_map_blocksmappings) to save the worker threads from redundantly parsing chunk indexes and calculating offsets.Can I replace this malloc-based raw buffer approach and focus v3 patch on an LRU cache for the parsed block mappings to reduce the CPU overhead during the concurrent directory traversal?
I don't think it's worth unless you provide enough number to show which really reduces the CPU overhead.
decompressed metadata cache is only useful for metabox, but I don't think it's prioritized for now.
Thanks,
Gao Xiang
Quoted 3 lines
Thanks,
Nithurshen
Hi Xiang,
Understood. I will drop the userspace metadata caching patches for now, as it is not the current priority.
I will shift my focus entirely to the concurrent directory traversal patch (Phase 3) that I submitted recently. Please let me know your thoughts on this.
Thanks,
Nithurshen