Skip to contentSkip to conversation

fsck.erofs: implement thread-safe global LRU metadata cache

PATCH Superseded Nithurshen 2 participants 6 messages Jun 11 – Jun 15 Library code

lore
Share & tools
lore

Nithurshen

author
#0 lore raw

PATCH 1/1 fsck.erofs: implement thread-safe global LRU metadata cache

This patch introduces a thread-safe metadata cache to reduce redundant I/O and decompression overhead during fsck extraction.

To ensure it remains highly concurrent for worker threads extracting pclusters, the cache utilizes a bucketed, rw-semaphore protected architecture modeled after the existing fragment cache.

Furthermore, to prevent out-of-memory (OOM) scenarios on exceptionally large EROFS images, the cache implements a strict Global Least Recently Used (LRU) eviction policy. The maximum cache size is dynamically configurable via the new '--cache-size' parameter, which defaults to a safe, fixed threshold of 32 MB.

---
 fsck/main.c              |  12 ++++
 include/erofs/internal.h |   2 +
 lib/data.c               | 149 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 160 insertions(+), 3 deletions(-)
fsck/main.c +12 -0
index ffe7e29..7a1e573 100644
@@ -67,6 +67,7 @@ static struct option long_options[] = {
 	{"no-xattrs", no_argument, 0, 14},
 	{"nid", required_argument, 0, 15},
 	{"path", required_argument, 0, 16},
+	{"cache-size", required_argument, 0, 17},
 	{"no-sbcrc", no_argument, 0, 512},
 	{0, 0, 0, 0},
 };
@@ -120,6 +121,7 @@ static void usage(int argc, char **argv)
 		" --offset=#             skip # bytes at the beginning of IMAGE\n"
 		" --nid=#                check or extract from the target inode of nid #\n"
 		" --path=X               check or extract from the target inode of path X\n"
+		" --cache-size=#        set maximum metadata cache size in bytes (default 32MB)\n"
 		" --no-sbcrc             bypass the superblock checksum verification\n"
 		" --[no-]xattrs          whether to dump extended attributes (default off)\n"
 		"\n"
@@ -261,6 +263,16 @@ static int erofsfsck_parse_options_cfg(int argc, char **argv)
 		case 16:
 			fsckcfg.inode_path = optarg;
 			break;
+		case 17: {
+			char *endptr;
+			unsigned long cache_size = strtoul(optarg, &endptr, 0);
+			if (*endptr != '\0') {
+				erofs_err("invalid metadata cache size %s", optarg);
+				return -EINVAL;
+			}
+			erofs_meta_cache_set_capacity(cache_size);
+			break;
+		}
 		case 512:
 			fsckcfg.nosbcrc = true;
 			break;
include/erofs/internal.h +2 -0
index 94f14da..34b7eb3 100644
@@ -459,6 +459,8 @@ struct z_erofs_read_ctx {
 
 void z_erofs_read_ctx_enqueue(struct z_erofs_read_ctx *ctx);
 
+void erofs_meta_cache_set_capacity(unsigned long bytes);
+
 int liberofs_global_init(void);
 void liberofs_global_exit(void);
 
lib/data.c +146 -3

160 lines not on this page · raw message

Signature · 1 line
2.52.0
#1 lore raw
Quoted 7 lines · Nithurshen

This patch introduces a thread-safe metadata cache to reduce redundant I/O and decompression overhead during fsck extraction.

To ensure it remains highly concurrent for worker threads extracting pclusters, the cache utilizes a bucketed, rw-semaphore protected architecture modeled after the existing fragment cache.

Quoted 7 lines

Furthermore, to prevent out-of-memory (OOM) scenarios on exceptionally large EROFS images, the cache implements a strict Global Least Recently Used (LRU) eviction policy. The maximum cache size is dynamically configurable via the new '--cache-size' parameter, which defaults to a safe, fixed threshold of 32 MB.

why malloc() will prevent out-of-memory (OOM)?

First, either erofs_read_metadata_nid() or erofs_read_metadata_bdi() will read file to the page cache; and currently there is no cache.

But you introduce another cache using malloc(), since it increases the memory overhead, why it prevents out-of-memory (OOM)?

Thanks,
Gao Xiang

author
#5 lore raw

PATCH v2 1/1 fsck.erofs: implement thread-safe global LRU metadata cache

In reply to Nithurshen

This patch introduces a thread-safe userspace metadata cache to reduce redundant decompression cycles and the overhead of repetitive pread() syscalls across multiple background worker threads.

To ensure it remains highly concurrent for worker threads extracting pclusters, the cache utilizes a bucketed, rw-semaphore protected architecture modeled after the existing fragment cache.

While the introduction of a userspace cache inherently increases the memory footprint compared to relying solely on the kernel's page cache, this patch implements a strict Global Least Recently Used (LRU) eviction policy to safely bound this additional memory overhead. This prevents the cache from growing unbounded on exceptionally large EROFS images. The maximum cache capacity is dynamically configurable via the new '--cache-size' parameter, which defaults to a safe threshold of 32 MB.

---
 fsck/main.c              |  12 ++++
 include/erofs/internal.h |   2 +
 lib/data.c               | 150 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 161 insertions(+), 3 deletions(-)
fsck/main.c +12 -0
index ffe7e29..7a1e573 100644
@@ -67,6 +67,7 @@ static struct option long_options[] = {
 	{"no-xattrs", no_argument, 0, 14},
 	{"nid", required_argument, 0, 15},
 	{"path", required_argument, 0, 16},
+	{"cache-size", required_argument, 0, 17},
 	{"no-sbcrc", no_argument, 0, 512},
 	{0, 0, 0, 0},
 };
@@ -120,6 +121,7 @@ static void usage(int argc, char **argv)
 		" --offset=#             skip # bytes at the beginning of IMAGE\n"
 		" --nid=#                check or extract from the target inode of nid #\n"
 		" --path=X               check or extract from the target inode of path X\n"
+		" --cache-size=#        set maximum metadata cache size in bytes (default 32MB)\n"
 		" --no-sbcrc             bypass the superblock checksum verification\n"
 		" --[no-]xattrs          whether to dump extended attributes (default off)\n"
 		"\n"
@@ -261,6 +263,16 @@ static int erofsfsck_parse_options_cfg(int argc, char **argv)
 		case 16:
 			fsckcfg.inode_path = optarg;
 			break;
+		case 17: {
+			char *endptr;
+			unsigned long cache_size = strtoul(optarg, &endptr, 0);
+			if (*endptr != '\0') {
+				erofs_err("invalid metadata cache size %s", optarg);
+				return -EINVAL;
+			}
+			erofs_meta_cache_set_capacity(cache_size);
+			break;
+		}
 		case 512:
 			fsckcfg.nosbcrc = true;
 			break;
include/erofs/internal.h +2 -0
index 94f14da..34b7eb3 100644
@@ -459,6 +459,8 @@ struct z_erofs_read_ctx {
 
 void z_erofs_read_ctx_enqueue(struct z_erofs_read_ctx *ctx);
 
+void erofs_meta_cache_set_capacity(unsigned long bytes);
+
 int liberofs_global_init(void);
 void liberofs_global_exit(void);
 
lib/data.c +147 -3

161 lines not on this page · raw message

Signature · 1 line
2.52.0
author
#2 lore raw
In reply to Gao Xiang

Hi Xiang,

What if rather than caching the raw metadata bytes, the userspace cache will focus on caching the parsed structures (such as the computed erofs_map_blocks mappings) to save the worker threads from redundantly parsing chunk indexes and calculating offsets.

Can I replace this malloc-based raw buffer approach and focus v3 patch on an LRU cache for the parsed block mappings to reduce the CPU overhead during the concurrent directory traversal?

Thanks,
Nithurshen

#3 lore raw
Quoted 10 lines · Nithurshen

Hi Xiang,

What if rather than caching the raw metadata bytes, the userspace cache will focus on caching the parsed structures (such as the computed erofs_map_blocks mappings) to save the worker threads from redundantly parsing chunk indexes and calculating offsets.

Can I replace this malloc-based raw buffer approach and focus v3 patch on an LRU cache for the parsed block mappings to reduce the CPU overhead during the concurrent directory traversal?

I don't think it's worth unless you provide enough number to show which really reduces the CPU overhead.

decompressed metadata cache is only useful for metabox, but I don't think it's prioritized for now.

Thanks,
Gao Xiang

Quoted 3 lines

Thanks,
Nithurshen

author
#4 lore raw

Hi Xiang,

Understood. I will drop the userspace metadata caching patches for now, as it is not the current priority.

I will shift my focus entirely to the concurrent directory traversal patch (Phase 3) that I submitted recently. Please let me know your thoughts on this.

Thanks,
Nithurshen