Skip to contentSkip to conversation

fscache,erofs: fscache-based demand-read semantics

PATCH Superseded Jeffle Xu 5 participants 44 messages Dec 27, 2021 – Jan 20, 2022

Subsystems (5)

v1 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Jeffle Xu

author
#0 lore raw

cover fscache,erofs: fscache-based demand-read semantics

changes since RFC:
- patch set organization:
patch 1-16 implement the data plane over fscache without demand reading support, while the demand reading support is left to patch 17-23.
- implement a new devnode ("/dev/cachefiles_demand") for the new mode (patch 1,20-22)
- use flag bit inside cache->flags to distinguish modes (patch 2) (David Howells)
- user daemon is responsible for placing blob files under corresponding fan sub directory, and setting "CacheFiles.cache" xattr in advance, so that cachefiles backend doesn't need modification to adapt to the new mode. (David Howells)
- erofs will allocate an anonymous inode for each backed file. This anonymous inode is responsible for managing page cache of backed file, so that netfs API doesn't need modification to adapt to the new mode. (Gao Xiang)

RFC: https://lore.kernel.org/all/[email protected]/t/

[Background]
============
erofs (Enhanced Read-Only File System) is a filesystem specially optimised for read-only scenarios. (Documentation/filesystem/erofs.rst)

Recently we are focusing on erofs in container images distribution scenario (https://sched.co/pcdL). In this case, erofs can be mounted from one bootstrap file (metadata) with (optional) multiple data blob files (data) stored on another local filesystem. (All these files are actually image files in erofs disk format.)

To accelerate the container startup (fetching container image from remote and then start the container), we do hope that the bootstrap blob file could support demand read. That is, erofs can be mounted and accessed even when the bootstrap/data blob files have not been fully downloaded.

That means we have to manage the cache state of the bootstrap/data blob files (if cache hit, read directly from the local cache; if cache miss, fetch the data somehow). It would be painful and may be dumb for erofs to implement the cache management itself. Thus we prefer fscache/cachefiles to do the cache management. Besides, the demand-read feature shall be general and it can benefit other using scenarios if it can be implemented in fscache level.

[Overall Design]
================
The upper fs uses a backing file on the local fs as the local cache (exactly the "cachefiles" way), and relies on fscache to detect if data is ready or not (cache hit/miss). Since currently fscache detects cache hit/miss by detecting the hole of the backing files, our demand-read mechanism also relies on the hole detecting.

1. initial phase
On the first beginning, the user daemon will touch the backing files (bootstrap/data blob files) under corresponding directory (under <root>/cache/<volume>/<fan>/) in advance. These backing files are completely sparse files (with zero disk usage). Since these backing files are all read-only and the file size is known prior mounting, user daemon will set corresponding file size and thus create all these sparse backing files in advance.

2. cache miss
When a file range (of bootstrap/data blob file) is accessed for the first time, a cache miss will be triggered and then .issue_op() will be called to fetch the data somehow.

In the demand-read case, we relies on a user daemon to fetch the data from local/remote. In this case, .issue_op() just packages the file range into a message and informs the user daemon. User daemon needs to poll and wait on the devnode (/dev/cachefiles_demand). Once awaken, the user daemon will read the devnode to get the file range information, and then fetch the data corresponding to the file range somehow, e.g. download from remote through network. Once data ready, the user daemon will write the fetched data into the backing file and then inform cachefiles backend by writing to the devnode. Cachefiles backend getting blocked on the previous .issue_op() calling will be awaken then. By then the data has been ready in the backing file, and the netfs API will re-initiate a read request from the backing file.

3. cache hit
Once data is already ready in the backing file, netfs API will read from the backing file directly.

[Advantage of fscache-based demand-read]
========================================
1. Asynchronous Prefetch
In current mechanism, fscache is responsible for cache state management, while the data plane (fetch data from local/remote on cache miss) is done on the user daemon side.

If data has already been ready in the backing file, netfs API will read from the backing file directly and won't be trapped to user space anymore. Thus the user daemon could fetch data (from remote) asynchronously on the background, and thus accelerate the backing file accessing in some degree.

2. Support massive blob files
Besides this mechanism supports a large amount of backing files, and thus can benefit the densely employed scenario.

In our using scenario, one container image can correspond to one bootstrap file (required) and multiple data blob files (optional). For example, one container image for node.js will corresponds to ~20 files in total. In densely employed environment, there could be as many as hundreds of containers and thus thousands of backing files on one machine.

[Test]
======
1. create erofs image (bootstrap)
mkfs.erofs --chunksize=1048576 --blobdev=Dblob1.img -Eforce-chunk-indexes Dtest.img tmp/

2. create sparse blob files under fscache root directory ("/root") truncate -s 28672 cache/Ierofs/@9c/Dtest.img
truncate -s 8040448 cache/Ierofs/@b5/Dblob1.img

3. set "CacheFiles.cache" xattr for blob files in advance (refer to
https://github.com/lostjeffle/demand-read-cachefilesd/blob/main/setxattr.c)

4. run user daemon
(https://github.com/lostjeffle/demand-read-cachefilesd/blob/main/cachefilesd2.c)
./cachefilesd2

5. mount erofs from bootstrap
mount -t erofs none -o uuid=test.img /mnt/

Jeffle Xu (23):
cachefiles: add cachefiles_demand devnode
cachefiles: add mode command to distinguish modes
cachefiles: detect backing file size in demand-read mode
netfs: make ops->init_rreq() optional
netfs: add inode parameter to netfs_alloc_read_request()
erofs: export erofs_map_blocks()
erofs: add nodev mode
erofs: register global fscache volume
erofs: add cookie context helper functions
erofs: add anonymous inode managing page cache of blob file
erofs: register cookie context for bootstrap
erofs: implement fscache-based metadata read
erofs: implement fscache-based data read
erofs: register cookie context for data blobs
erofs: implement fscache-based data read for data blobs
erofs: add 'uuid' mount option
netfs: support on demand read
cachefiles: use idr tree managing pending demand read
cachefiles: implement .demand_read() for demand read
cachefiles: implement .poll() for demand read
cachefiles: implement .read() for demand read
cachefiles: add done command for demand read
erofs: support on demand read

 fs/cachefiles/daemon.c   | 124 +++++++++++++++++
 fs/cachefiles/internal.h |  18 +++
 fs/cachefiles/io.c       |  56 ++++++++
 fs/cachefiles/main.c     |  12 ++
 fs/cachefiles/namei.c    |  40 ++++++
 fs/ceph/addr.c           |   5 -
 fs/erofs/Makefile        |   2 +-
 fs/erofs/data.c          |  20 ++-
 fs/erofs/fscache.c       | 292 +++++++++++++++++++++++++++++++++++++++
 fs/erofs/inode.c         |   6 +-
 fs/erofs/internal.h      |  25 ++++
 fs/erofs/super.c         |  98 ++++++++++---
 fs/netfs/read_helper.c   |  41 +++++-
 include/linux/netfs.h    |   4 +
 14 files changed, 708 insertions(+), 35 deletions(-)
 create mode 100644 fs/erofs/fscache.c
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH 1/23 cachefiles: add cachefiles_demand devnode

fscache/cachefiles used to serve as a local cache for remote fs. The following patches will introduce a new use case, in which local read-only fs could implement demand reading with fscache. By then the user daemon needs to read and poll on the devnode, and thus the original cachefiles devnode can't be reused in this case.

Thus create a new devnode specifically for the new mode. The following patches will add more file_operations.

---
 fs/cachefiles/daemon.c   |  8 ++++++++
 fs/cachefiles/internal.h |  1 +
 fs/cachefiles/main.c     | 12 ++++++++++++
 3 files changed, 21 insertions(+)
fs/cachefiles/daemon.c +8 -0
index 40a792421fc1..871f1e0f423d 100644
@@ -56,6 +56,14 @@ const struct file_operations cachefiles_daemon_fops = {
 	.llseek		= noop_llseek,
 };
 
+const struct file_operations cachefiles_demand_fops = {
+	.owner		= THIS_MODULE,
+	.open		= cachefiles_daemon_open,
+	.release	= cachefiles_daemon_release,
+	.write		= cachefiles_daemon_write,
+	.llseek		= noop_llseek,
+};
+
 struct cachefiles_daemon_cmd {
 	char name[8];
 	int (*handler)(struct cachefiles_cache *cache, char *args);
fs/cachefiles/internal.h +1 -0
index 421423819d63..e0ed811d628d 100644
@@ -145,6 +145,7 @@ extern int cachefiles_has_space(struct cachefiles_cache *cache,
  * daemon.c
  */
 extern const struct file_operations cachefiles_daemon_fops;
+extern const struct file_operations cachefiles_demand_fops;
 
 /*
  * error_inject.c
fs/cachefiles/main.c +12 -0
index 3f369c6f816d..0a423274d283 100644
@@ -39,6 +39,12 @@ static struct miscdevice cachefiles_dev = {
 	.fops	= &cachefiles_daemon_fops,
 };
 
+static struct miscdevice cachefiles_demand_dev = {
+	.minor	= MISC_DYNAMIC_MINOR,
+	.name	= "cachefiles_demand",
+	.fops	= &cachefiles_demand_fops,
+};
+
 /*
  * initialise the fs caching module
  */
@@ -52,6 +58,9 @@ static int __init cachefiles_init(void)
 	ret = misc_register(&cachefiles_dev);
 	if (ret < 0)
 		goto error_dev;
+	ret = misc_register(&cachefiles_demand_dev);
+	if (ret < 0)
+		goto error_demand_dev;
 
 	/* create an object jar */
 	ret = -ENOMEM;
@@ -68,6 +77,8 @@ static int __init cachefiles_init(void)
 	return 0;
 
 error_object_jar:
+	misc_deregister(&cachefiles_demand_dev);
+error_demand_dev:
 	misc_deregister(&cachefiles_dev);
 error_dev:
 	cachefiles_unregister_error_injection();
@@ -86,6 +97,7 @@ static void __exit cachefiles_exit(void)
 	pr_info("Unloading\n");
 
 	kmem_cache_destroy(cachefiles_object_jar);
+	misc_deregister(&cachefiles_demand_dev);
 	misc_deregister(&cachefiles_dev);
 	cachefiles_unregister_error_injection();
 }
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH v1 01/23] cachefiles: add cachefiles_demand devnode

Quoted 22 lines · Jeffle Xu

fscache/cachefiles used to serve as a local cache for remote fs. The following patches will introduce a new use case, in which local read-only fs could implement demand reading with fscache. By then the user daemon needs to read and poll on the devnode, and thus the original cachefiles devnode can't be reused in this case.

Thus create a new devnode specifically for the new mode. The following patches will add more file_operations.

---
 fs/cachefiles/daemon.c   |  8 ++++++++
 fs/cachefiles/internal.h |  1 +
 fs/cachefiles/main.c     | 12 ++++++++++++
 3 files changed, 21 insertions(+)
fs/cachefiles/daemon.c +0 -0
index 40a792421fc1..871f1e0f423d 100644
@@ -56,6 +56,14 @@ const struct file_operations cachefiles_daemon_fops = {
 	.llseek		= noop_llseek,

fs/cachefiles/daemon.c const struct file_operations cachefiles_daemon_fops = { · line 66

Quoted 10 lines
fs/cachefiles/daemon.c +8 -0
 };
 
+const struct file_operations cachefiles_demand_fops = {
+	.owner		= THIS_MODULE,
+	.open		= cachefiles_daemon_open,
+	.release	= cachefiles_daemon_release,
+	.write		= cachefiles_daemon_write,
+	.llseek		= noop_llseek,
+};
+

Better to prepare the on-demand read() and poll() first, and then add the on-demand cachefiles dev.

Thanks,
Joseph

Quoted 59 lines
fs/cachefiles/daemon.c +0 -0
 struct cachefiles_daemon_cmd {
 	char name[8];
 	int (*handler)(struct cachefiles_cache *cache, char *args);
fs/cachefiles/internal.h +1 -0
index 421423819d63..e0ed811d628d 100644
@@ -145,6 +145,7 @@ extern int cachefiles_has_space(struct cachefiles_cache *cache,
  * daemon.c
  */
 extern const struct file_operations cachefiles_daemon_fops;
+extern const struct file_operations cachefiles_demand_fops;
 
 /*
  * error_inject.c
fs/cachefiles/main.c +12 -0
index 3f369c6f816d..0a423274d283 100644
@@ -39,6 +39,12 @@ static struct miscdevice cachefiles_dev = {
 	.fops	= &cachefiles_daemon_fops,
 };
 
+static struct miscdevice cachefiles_demand_dev = {
+	.minor	= MISC_DYNAMIC_MINOR,
+	.name	= "cachefiles_demand",
+	.fops	= &cachefiles_demand_fops,
+};
+
 /*
  * initialise the fs caching module
  */
@@ -52,6 +58,9 @@ static int __init cachefiles_init(void)
 	ret = misc_register(&cachefiles_dev);
 	if (ret < 0)
 		goto error_dev;
+	ret = misc_register(&cachefiles_demand_dev);
+	if (ret < 0)
+		goto error_demand_dev;
 
 	/* create an object jar */
 	ret = -ENOMEM;
@@ -68,6 +77,8 @@ static int __init cachefiles_init(void)
 	return 0;
 
 error_object_jar:
+	misc_deregister(&cachefiles_demand_dev);
+error_demand_dev:
 	misc_deregister(&cachefiles_dev);
 error_dev:
 	cachefiles_unregister_error_injection();
@@ -86,6 +97,7 @@ static void __exit cachefiles_exit(void)
 	pr_info("Unloading\n");
 
 	kmem_cache_destroy(cachefiles_object_jar);
+	misc_deregister(&cachefiles_demand_dev);
 	misc_deregister(&cachefiles_dev);
 	cachefiles_unregister_error_injection();
 }
author
#3 lore raw

Re: [PATCH v1 01/23] cachefiles: add cachefiles_demand devnode

Quoted 4 lines · Joseph Qi

Better to prepare the on-demand read() and poll() first, and then add the on-demand cachefiles dev.

Regards. Thanks.

Signature · 2 lines
Thanks,
Jeffle
author
#4 lore raw

PATCH 2/23 cachefiles: add mode command to distinguish modes

In reply to Jeffle Xu

Add one flag bit to distinguish the new introduced demand-read mode from the original mode. User daemon could set the specified mode with 'mode' command. If user daemon doesn't ever explicitly set the mode, then the behaviour is the same with that prior this patch, i.e. cachefiles serves as the local cache for remote fs by default.

---
 fs/cachefiles/daemon.c   | 32 ++++++++++++++++++++++++++++++++
 fs/cachefiles/internal.h |  1 +
 2 files changed, 33 insertions(+)
fs/cachefiles/daemon.c +32 -0
index 871f1e0f423d..892a9bdba53f 100644
@@ -41,6 +41,7 @@ static int cachefiles_daemon_dir(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_inuse(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_secctx(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_tag(struct cachefiles_cache *, char *);
+static int cachefiles_daemon_mode(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_bind(struct cachefiles_cache *, char *);
 static void cachefiles_daemon_unbind(struct cachefiles_cache *);
 
@@ -83,6 +84,7 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = {
 	{ "inuse",	cachefiles_daemon_inuse		},
 	{ "secctx",	cachefiles_daemon_secctx	},
 	{ "tag",	cachefiles_daemon_tag		},
+	{ "mode",	cachefiles_daemon_mode		},
 	{ "",		NULL				}
 };
 
@@ -671,6 +673,36 @@ static int cachefiles_daemon_inuse(struct cachefiles_cache *cache, char *args)
 	return -EINVAL;
 }
 
+/*
+ * Set the cache mode
+ * - command: "mode cache|demand"
+ */
+static int cachefiles_daemon_mode(struct cachefiles_cache *cache, char *args)
+{
+	_enter(",%s", args);
+
+	if (test_bit(CACHEFILES_READY, &cache->flags)) {
+		pr_err("Cache already started\n");
+		return -EINVAL;
+	}
+
+	if (!*args) {
+		pr_err("Empty mode specified\n");
+		return -EINVAL;
+	}
+
+	if (!strncmp(args, "cache", strlen("cache"))) {
+		clear_bit(CACHEFILES_DEMAND_MODE, &cache->flags);
+	} else if (!strncmp(args, "demand", strlen("demand"))) {
+		set_bit(CACHEFILES_DEMAND_MODE, &cache->flags);
+	} else {
+		pr_err("Invalid mode specified\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 /*
  * Bind a directory as a cache
  */
fs/cachefiles/internal.h +1 -0
index e0ed811d628d..a8e6500889d7 100644
@@ -98,6 +98,7 @@ struct cachefiles_cache {
 #define CACHEFILES_DEAD			1	/* T if cache dead */
 #define CACHEFILES_CULLING		2	/* T if cull engaged */
 #define CACHEFILES_STATE_CHANGED	3	/* T if state changed (poll trigger) */
+#define CACHEFILES_DEMAND_MODE		4	/* T if works in demand read mode for read-only fs */
 	char				*rootdirname;	/* name of cache root directory */
 	char				*secctx;	/* LSM security context */
 	char				*tag;		/* cache binding tag */
Signature · 1 line
2.27.0
author
#5 lore raw

PATCH 3/23 cachefiles: detect backing file size in demand-read mode

In reply to Jeffle Xu

When upper read-only fs uses fscache for demand reading, it has no idea on the size of the backed file. (You need to input the file size as the @object_size parameter when calling fscache_acquire_cookie().)

In this using scenario, user daemon is responsible for preparing all backing files with correct file size (though they can be all sparse files), and the upper fs shall guarantee that past EOF access will never happen.

Then with this precondition, cachefiles can detect the actual size of backing file, and set it as the size of backed file.

---
 fs/cachefiles/namei.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
fs/cachefiles/namei.c +40 -0
index 074722c21522..54123b2693cd 100644
@@ -511,9 +511,19 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
  */
 static bool cachefiles_create_file(struct cachefiles_object *object)
 {
+	struct cachefiles_cache *cache = object->volume->cache;
 	struct file *file;
 	int ret;
 
+	/*
+	 * Demand read mode requires that backing files have been prepared with
+	 * correct file size under corresponding directory. We can get here when
+	 * the backing file doesn't exist under corresponding directory, or the
+	 * file size is unexpected 0.
+	 */
+	if (test_bit(CACHEFILES_DEMAND_MODE, &cache->flags))
+		return false;
+
 	ret = cachefiles_has_space(object->volume->cache, 1, 0,
 				   cachefiles_has_space_for_create);
 	if (ret < 0)
@@ -530,6 +540,32 @@ static bool cachefiles_create_file(struct cachefiles_object *object)
 	return true;
 }
 
+/*
+ * Fs using fscache for demand reading may have no idea of the file size of
+ * backing files. Thus the demand read mode requires that backing files have
+ * been prepared with correct file size under corresponding directory. Then
+ * fscache backend is responsible for taking the file size of the backing file
+ * as the object size.
+ */
+static int cachefiles_recheck_size(struct cachefiles_object *object,
+				   struct file *file)
+{
+	loff_t size;
+	struct cachefiles_cache *cache = object->volume->cache;
+
+	if (!test_bit(CACHEFILES_DEMAND_MODE, &cache->flags))
+		return 0;
+
+	size = i_size_read(file_inode(file));
+	if (size) {
+		object->cookie->object_size = size;
+		return 0;
+	} else {
+		return -EINVAL;
+	}
+
+}
+
 /*
  * Open an existing file, checking its attributes and replacing it if it is
  * stale.
@@ -569,6 +605,10 @@ static bool cachefiles_open_file(struct cachefiles_object *object,
 	}
 	_debug("file -> %pd positive", dentry);
 
+	ret = cachefiles_recheck_size(object, file);
+	if (ret < 0)
+		goto check_failed;
+
 	ret = cachefiles_check_auxdata(object, file);
 	if (ret < 0)
 		goto check_failed;
Signature · 1 line
2.27.0
author
#6 lore raw

PATCH 4/23 netfs: make ops->init_rreq() optional

In reply to Jeffle Xu

There's already upper fs implementing empty .init_rreq() callback, and thus make it optional.

---
 fs/ceph/addr.c         | 5 -----
 fs/netfs/read_helper.c | 3 ++-
 2 files changed, 2 insertions(+), 6 deletions(-)
fs/ceph/addr.c +0 -5
index b3d9459c9bbd..c98e5238a1b6 100644
@@ -297,10 +297,6 @@ static void ceph_netfs_issue_op(struct netfs_read_subrequest *subreq)
 	dout("%s: result %d\n", __func__, err);
 }
 
-static void ceph_init_rreq(struct netfs_read_request *rreq, struct file *file)
-{
-}
-
 static void ceph_readahead_cleanup(struct address_space *mapping, void *priv)
 {
 	struct inode *inode = mapping->host;
@@ -312,7 +308,6 @@ static void ceph_readahead_cleanup(struct address_space *mapping, void *priv)
 }
 
 static const struct netfs_read_request_ops ceph_netfs_read_ops = {
-	.init_rreq		= ceph_init_rreq,
 	.is_cache_enabled	= ceph_is_cache_enabled,
 	.begin_cache_operation	= ceph_begin_cache_operation,
 	.issue_op		= ceph_netfs_issue_op,
fs/netfs/read_helper.c +2 -1
index 1659a55e9c1c..8c58cff420ba 100644
@@ -55,7 +55,8 @@ static struct netfs_read_request *netfs_alloc_read_request(
 		INIT_WORK(&rreq->work, netfs_rreq_work);
 		refcount_set(&rreq->usage, 1);
 		__set_bit(NETFS_RREQ_IN_PROGRESS, &rreq->flags);
-		ops->init_rreq(rreq, file);
+		if (ops->init_rreq)
+			ops->init_rreq(rreq, file);
 		netfs_stat(&netfs_n_rh_rreq);
 	}
 
Signature · 1 line
2.27.0
author
#7 lore raw

PATCH 5/23 netfs: add inode parameter to netfs_alloc_read_request()

In reply to Jeffle Xu

When working as the local cache, the @file parameter of netfs_alloc_read_request() represents the backed file inside netfs. It is for two use: 1) we can derive the corresponding inode from file,
2) works as the argument for ops->init_rreq().

In the new introduced demand-read mode, netfs_readpage() will be called by the upper fs to read from backing files. However in this new mode, the backed file may not be opened, and thus the @file argument is NULL in this case.

For netfs_readpage(), @file parameter represents the backed file inside netfs, while @folio parameter represents one page cache inside the address space of this backed file. We can still derive the inode from the @folio parameter, even when @file parameter is NULL.

Thus refactor netfs_alloc_read_request() somewhat for this change.

---
 fs/netfs/read_helper.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
fs/netfs/read_helper.c +7 -5
index 8c58cff420ba..ca84918b6b5d 100644
@@ -39,7 +39,7 @@ static void netfs_put_subrequest(struct netfs_read_subrequest *subreq,
 
 static struct netfs_read_request *netfs_alloc_read_request(
 	const struct netfs_read_request_ops *ops, void *netfs_priv,
-	struct file *file)
+	struct inode *inode, struct file *file)
 {
 	static atomic_t debug_ids;
 	struct netfs_read_request *rreq;
@@ -48,7 +48,7 @@ static struct netfs_read_request *netfs_alloc_read_request(
 	if (rreq) {
 		rreq->netfs_ops	= ops;
 		rreq->netfs_priv = netfs_priv;
-		rreq->inode	= file_inode(file);
+		rreq->inode	= inode;
 		rreq->i_size	= i_size_read(rreq->inode);
 		rreq->debug_id	= atomic_inc_return(&debug_ids);
 		INIT_LIST_HEAD(&rreq->subrequests);
@@ -870,6 +870,7 @@ void netfs_readahead(struct readahead_control *ractl,
 		     void *netfs_priv)
 {
 	struct netfs_read_request *rreq;
+	struct inode *inode = file_inode(ractl->file);
 	unsigned int debug_index = 0;
 	int ret;
 
@@ -878,7 +879,7 @@ void netfs_readahead(struct readahead_control *ractl,
 	if (readahead_count(ractl) == 0)
 		goto cleanup;
 
-	rreq = netfs_alloc_read_request(ops, netfs_priv, ractl->file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, ractl->file);
 	if (!rreq)
 		goto cleanup;
 	rreq->mapping	= ractl->mapping;
@@ -948,12 +949,13 @@ int netfs_readpage(struct file *file,
 		   void *netfs_priv)
 {
 	struct netfs_read_request *rreq;
+	struct inode *inode = folio_file_mapping(folio)->host;
 	unsigned int debug_index = 0;
 	int ret;
 
 	_enter("%lx", folio_index(folio));
 
-	rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file);
 	if (!rreq) {
 		if (netfs_priv)
 			ops->cleanup(folio_file_mapping(folio), netfs_priv);
@@ -1122,7 +1124,7 @@ int netfs_write_begin(struct file *file, struct address_space *mapping,
 	}
 
 	ret = -ENOMEM;
-	rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file);
 	if (!rreq)
 		goto error;
 	rreq->mapping		= folio_file_mapping(folio);
Signature · 1 line
2.27.0
#8 lore raw

Re: [PATCH v1 05/23] netfs: add inode parameter to netfs_alloc_read_request()

Quoted 10 lines · Jeffle Xu

When working as the local cache, the @file parameter of netfs_alloc_read_request() represents the backed file inside netfs. It is for two use: 1) we can derive the corresponding inode from file,
2) works as the argument for ops->init_rreq().

In the new introduced demand-read mode, netfs_readpage() will be called by the upper fs to read from backing files. However in this new mode, the backed file may not be opened, and thus the @file argument is NULL in this case.

Quoted 8 lines

For netfs_readpage(), @file parameter represents the backed file inside netfs, while @folio parameter represents one page cache inside the address space of this backed file. We can still derive the inode from the @folio parameter, even when @file parameter is NULL.

Thus refactor netfs_alloc_read_request() somewhat for this change.

I'm not sure how other folks think.

Yet in principle, personally I think it's reasonable that something like read_cache_page_gfp() could be used directly with fscache backend as well.

So for such internal read requests, @file argument is actually optional as a common practice.

Apart from the commit message itself (which I think it could be simplified a bit), generally it looks good to me.

Thanks,
Gao Xiang

Quoted 69 lines
---
 fs/netfs/read_helper.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
fs/netfs/read_helper.c +7 -5
index 8c58cff420ba..ca84918b6b5d 100644
@@ -39,7 +39,7 @@ static void netfs_put_subrequest(struct netfs_read_subrequest *subreq,
 
 static struct netfs_read_request *netfs_alloc_read_request(
 	const struct netfs_read_request_ops *ops, void *netfs_priv,
-	struct file *file)
+	struct inode *inode, struct file *file)
 {
 	static atomic_t debug_ids;
 	struct netfs_read_request *rreq;
@@ -48,7 +48,7 @@ static struct netfs_read_request *netfs_alloc_read_request(
 	if (rreq) {
 		rreq->netfs_ops	= ops;
 		rreq->netfs_priv = netfs_priv;
-		rreq->inode	= file_inode(file);
+		rreq->inode	= inode;
 		rreq->i_size	= i_size_read(rreq->inode);
 		rreq->debug_id	= atomic_inc_return(&debug_ids);
 		INIT_LIST_HEAD(&rreq->subrequests);
@@ -870,6 +870,7 @@ void netfs_readahead(struct readahead_control *ractl,
 		     void *netfs_priv)
 {
 	struct netfs_read_request *rreq;
+	struct inode *inode = file_inode(ractl->file);
 	unsigned int debug_index = 0;
 	int ret;
 
@@ -878,7 +879,7 @@ void netfs_readahead(struct readahead_control *ractl,
 	if (readahead_count(ractl) == 0)
 		goto cleanup;
 
-	rreq = netfs_alloc_read_request(ops, netfs_priv, ractl->file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, ractl->file);
 	if (!rreq)
 		goto cleanup;
 	rreq->mapping	= ractl->mapping;
@@ -948,12 +949,13 @@ int netfs_readpage(struct file *file,
 		   void *netfs_priv)
 {
 	struct netfs_read_request *rreq;
+	struct inode *inode = folio_file_mapping(folio)->host;
 	unsigned int debug_index = 0;
 	int ret;
 
 	_enter("%lx", folio_index(folio));
 
-	rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file);
 	if (!rreq) {
 		if (netfs_priv)
 			ops->cleanup(folio_file_mapping(folio), netfs_priv);
@@ -1122,7 +1124,7 @@ int netfs_write_begin(struct file *file, struct address_space *mapping,
 	}
 
 	ret = -ENOMEM;
-	rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file);
 	if (!rreq)
 		goto error;
 	rreq->mapping		= folio_file_mapping(folio);
Signature · 1 line
2.27.0
author
#9 lore raw

Re: [Linux-cachefs] [PATCH v1 05/23] netfs: add inode parameter to netfs_alloc_read_request()

In reply to Jeffle Xu

Hi David,

What would you think about this cleanup? We need this in prep for the following fscache-based on-demand reading feature. It would be great if it could be cherry picked in advance.

I also simplify the commit message as suggested by Gao Xiang. I could resend a v2 patch with the updated commit message if you'd like.

netfs: add inode parameter to netfs_alloc_read_request()

Make the @file parameter optional, and derive inode from the @folio parameter instead.

@file parameter can't be removed completely, since it also works as the private data of ops->init_rreq().

Quoted 86 lines · Jeffle Xu

When working as the local cache, the @file parameter of netfs_alloc_read_request() represents the backed file inside netfs. It is for two use: 1) we can derive the corresponding inode from file,
2) works as the argument for ops->init_rreq().

In the new introduced demand-read mode, netfs_readpage() will be called by the upper fs to read from backing files. However in this new mode, the backed file may not be opened, and thus the @file argument is NULL in this case.

For netfs_readpage(), @file parameter represents the backed file inside netfs, while @folio parameter represents one page cache inside the address space of this backed file. We can still derive the inode from the @folio parameter, even when @file parameter is NULL.

Thus refactor netfs_alloc_read_request() somewhat for this change.

---
 fs/netfs/read_helper.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
fs/netfs/read_helper.c +7 -5
index 8c58cff420ba..ca84918b6b5d 100644
@@ -39,7 +39,7 @@ static void netfs_put_subrequest(struct netfs_read_subrequest *subreq,
 
 static struct netfs_read_request *netfs_alloc_read_request(
 	const struct netfs_read_request_ops *ops, void *netfs_priv,
-	struct file *file)
+	struct inode *inode, struct file *file)
 {
 	static atomic_t debug_ids;
 	struct netfs_read_request *rreq;
@@ -48,7 +48,7 @@ static struct netfs_read_request *netfs_alloc_read_request(
 	if (rreq) {
 		rreq->netfs_ops	= ops;
 		rreq->netfs_priv = netfs_priv;
-		rreq->inode	= file_inode(file);
+		rreq->inode	= inode;
 		rreq->i_size	= i_size_read(rreq->inode);
 		rreq->debug_id	= atomic_inc_return(&debug_ids);
 		INIT_LIST_HEAD(&rreq->subrequests);
@@ -870,6 +870,7 @@ void netfs_readahead(struct readahead_control *ractl,
 		     void *netfs_priv)
 {
 	struct netfs_read_request *rreq;
+	struct inode *inode = file_inode(ractl->file);
 	unsigned int debug_index = 0;
 	int ret;
 
@@ -878,7 +879,7 @@ void netfs_readahead(struct readahead_control *ractl,
 	if (readahead_count(ractl) == 0)
 		goto cleanup;
 
-	rreq = netfs_alloc_read_request(ops, netfs_priv, ractl->file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, ractl->file);
 	if (!rreq)
 		goto cleanup;
 	rreq->mapping	= ractl->mapping;
@@ -948,12 +949,13 @@ int netfs_readpage(struct file *file,
 		   void *netfs_priv)
 {
 	struct netfs_read_request *rreq;
+	struct inode *inode = folio_file_mapping(folio)->host;
 	unsigned int debug_index = 0;
 	int ret;
 
 	_enter("%lx", folio_index(folio));
 
-	rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file);
 	if (!rreq) {
 		if (netfs_priv)
 			ops->cleanup(folio_file_mapping(folio), netfs_priv);
@@ -1122,7 +1124,7 @@ int netfs_write_begin(struct file *file, struct address_space *mapping,
 	}
 
 	ret = -ENOMEM;
-	rreq = netfs_alloc_read_request(ops, netfs_priv, file);
+	rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file);
 	if (!rreq)
 		goto error;
 	rreq->mapping		= folio_file_mapping(folio);
Signature · 2 lines
Thanks,
Jeffle
#10 lore raw

Re: [Linux-cachefs] [PATCH v1 05/23] netfs: add inode parameter to netfs_alloc_read_request()

Quoted 11 lines · JeffleXu

Hi David,

What would you think about this cleanup? We need this in prep for the following fscache-based on-demand reading feature. It would be great if it could be cherry picked in advance.

I also simplify the commit message as suggested by Gao Xiang. I could resend a v2 patch with the updated commit message if you'd like.

netfs: add inode parameter to netfs_alloc_read_request()

Quoted 7 lines

Make the @file parameter optional, and derive inode from the @folio parameter instead.

@file parameter can't be removed completely, since it also works as the private data of ops->init_rreq().

Hi,

IMHO, How about the following message:

netfs: make @file optional in netfs_alloc_read_request()

Make the @file parameter optional, and derive inode from the @folio parameter instead in order to support file system internal requests.

@file parameter can't be removed completely, since it also works as the private data of ops->init_rreq().

Thanks,
Gao Xiang

author
#11 lore raw

PATCH 6/23 erofs: export erofs_map_blocks()

In reply to Jeffle Xu

... so that it can be used in the following introduced fs/erofs/fscache.c.

---
 fs/erofs/data.c     | 4 ++--
 fs/erofs/internal.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
fs/erofs/data.c +2 -2
index 0e35ef3f9f3d..477aaff0c832 100644
@@ -77,8 +77,8 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
 	return err;
 }
 
-static int erofs_map_blocks(struct inode *inode,
-			    struct erofs_map_blocks *map, int flags)
+int erofs_map_blocks(struct inode *inode,
+		     struct erofs_map_blocks *map, int flags)
 {
 	struct super_block *sb = inode->i_sb;
 	struct erofs_inode *vi = EROFS_I(inode);
fs/erofs/internal.h +2 -0
index 3265688af7f9..45fb6f5d11b5 100644
@@ -447,6 +447,8 @@ struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr);
 int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *dev);
 int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 		 u64 start, u64 len);
+int erofs_map_blocks(struct inode *inode,
+		     struct erofs_map_blocks *map, int flags);
 
 /* inode.c */
 static inline unsigned long erofs_inode_hash(erofs_nid_t nid)
Signature · 1 line
2.27.0
author
#12 lore raw

PATCH 7/23 erofs: add nodev mode

In reply to Jeffle Xu

Until then erofs is exactly blockdev based filesystem. In other using scenarios (e.g. container image), erofs needs to run upon files.

This patch introduces a new nodev mode, in which erofs could be mounted from a bootstrap blob file containing the complete erofs image.

The following patch will introduce a new mount option "uuid", by which users could specify the bootstrap blob file.

---
 fs/erofs/data.c     | 13 ++++++++---
 fs/erofs/internal.h |  1 +
 fs/erofs/super.c    | 56 +++++++++++++++++++++++++++++++++------------
 3 files changed, 53 insertions(+), 17 deletions(-)
fs/erofs/data.c +10 -3
index 477aaff0c832..61fa431d0713 100644
@@ -11,11 +11,18 @@
 
 struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr)
 {
-	struct address_space *const mapping = sb->s_bdev->bd_inode->i_mapping;
+	struct address_space *mapping;
 	struct page *page;
 
-	page = read_cache_page_gfp(mapping, blkaddr,
-				   mapping_gfp_constraint(mapping, ~__GFP_FS));
+	if (sb->s_bdev) {
+		mapping = sb->s_bdev->bd_inode->i_mapping;
+		page = read_cache_page_gfp(mapping, blkaddr,
+				mapping_gfp_constraint(mapping, ~__GFP_FS));
+	} else {
+		/* TODO: data path in nodev mode */
+		page = ERR_PTR(-EINVAL);
+	}
+
 	/* should already be PageUptodate */
 	if (!IS_ERR(page))
 		lock_page(page);
fs/erofs/internal.h +1 -0
index 45fb6f5d11b5..c9ee8c247202 100644
@@ -67,6 +67,7 @@ struct erofs_mount_opts {
 	unsigned int max_sync_decompress_pages;
 #endif
 	unsigned int mount_opt;
+	char *uuid;
 };
 
 struct erofs_dev_context {
fs/erofs/super.c +42 -14

109 lines not on this page · raw message

Signature · 1 line
2.27.0
#13 lore raw

Re: [PATCH v1 07/23] erofs: add nodev mode

Quoted 10 lines · Jeffle Xu

Until then erofs is exactly blockdev based filesystem. In other using scenarios (e.g. container image), erofs needs to run upon files.

This patch introduces a new nodev mode, in which erofs could be mounted from a bootstrap blob file containing the complete erofs image.

The following patch will introduce a new mount option "uuid", by which users could specify the bootstrap blob file.

I think the order of some patches in this patchset can be improved.

Take this patch as an example. This patch introduces a new mount option called "uuid", so the kernel will just accept it (which generates a user-visible impact) after this patch but it doesn't actually work.

Therefore, we actually have three different behaviors here:
- kernel doesn't support "uuid" mount option completely;
- kernel support "uuid" but it doesn't work;
- kernel support "uuid" correctly (maybe after some random patch);

Actually that is bad for bisecting since there are some commits having temporary behaviors. And we don't know which commit actually fully implements this "uuid" mount option.

So personally I think the proper order is just like the bottom-up approach, and make sure each patch can be tested / bisected independently.

Quoted 11 lines
---
 fs/erofs/data.c     | 13 ++++++++---
 fs/erofs/internal.h |  1 +
 fs/erofs/super.c    | 56 +++++++++++++++++++++++++++++++++------------
 3 files changed, 53 insertions(+), 17 deletions(-)
fs/erofs/data.c +0 -0
index 477aaff0c832..61fa431d0713 100644

fs/erofs/data.c line 18

Quoted 9 lines
fs/erofs/data.c +1 -3
@@ -11,11 +11,18 @@
 
 struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr)
 {
-	struct address_space *const mapping = sb->s_bdev->bd_inode->i_mapping;
+	struct address_space *mapping;
 	struct page *page;
 
-	page = read_cache_page_gfp(mapping, blkaddr,
-				   mapping_gfp_constraint(mapping, ~__GFP_FS));

Apart from the recommendation above, if my understanding is correct, I think after we implement fscache_aops, read_cache_page_gfp() can work with proper fscache mapping.

So no need to implement something like erofs_readpage_from_fscache() later (at least for the case here.)

Thanks,
Gao Xiang

Quoted 9 lines
fs/erofs/data.c +9 -0
+	if (sb->s_bdev) {
+		mapping = sb->s_bdev->bd_inode->i_mapping;
+		page = read_cache_page_gfp(mapping, blkaddr,
+				mapping_gfp_constraint(mapping, ~__GFP_FS));
+	} else {
+		/* TODO: data path in nodev mode */
+		page = ERR_PTR(-EINVAL);
+	}
+
#14 lore raw

Re: [PATCH v1 07/23] erofs: add nodev mode

Quoted 24 lines · Gao Xiang
Quoted 10 lines · Jeffle Xu

Until then erofs is exactly blockdev based filesystem. In other using scenarios (e.g. container image), erofs needs to run upon files.

This patch introduces a new nodev mode, in which erofs could be mounted from a bootstrap blob file containing the complete erofs image.

The following patch will introduce a new mount option "uuid", by which users could specify the bootstrap blob file.

I think the order of some patches in this patchset can be improved.

Take this patch as an example. This patch introduces a new mount option called "uuid", so the kernel will just accept it (which generates a user-visible impact) after this patch but it doesn't actually work.

Therefore, we actually have three different behaviors here:
- kernel doesn't support "uuid" mount option completely;
- kernel support "uuid" but it doesn't work;
- kernel support "uuid" correctly (maybe after some random patch);

Quoted 7 lines

Actually that is bad for bisecting since there are some commits having temporary behaviors. And we don't know which commit actually fully implements this "uuid" mount option.

So personally I think the proper order is just like the bottom-up approach, and make sure each patch can be tested / bisected independently.

Oh, I may misread this patch, but I still think we'd better to avoid dead paths "TODO" like this as much as possible.

Just do in the bottom-up way.

Thanks,
Gao Xiang

author
#15 lore raw

Re: [PATCH v1 07/23] erofs: add nodev mode

Quoted 29 lines · Gao Xiang
Quoted 28 lines · Gao Xiang
Quoted 10 lines · Jeffle Xu

Until then erofs is exactly blockdev based filesystem. In other using scenarios (e.g. container image), erofs needs to run upon files.

This patch introduces a new nodev mode, in which erofs could be mounted from a bootstrap blob file containing the complete erofs image.

The following patch will introduce a new mount option "uuid", by which users could specify the bootstrap blob file.

I think the order of some patches in this patchset can be improved.

Take this patch as an example. This patch introduces a new mount option called "uuid", so the kernel will just accept it (which generates a user-visible impact) after this patch but it doesn't actually work.

Therefore, we actually have three different behaviors here:
- kernel doesn't support "uuid" mount option completely;
- kernel support "uuid" but it doesn't work;
- kernel support "uuid" correctly (maybe after some random patch);

Actually that is bad for bisecting since there are some commits having temporary behaviors. And we don't know which commit actually fully implements this "uuid" mount option.

Quoted 9 lines
Quoted 3 lines

So personally I think the proper order is just like the bottom-up approach, and make sure each patch can be tested / bisected independently.

Oh, I may misread this patch, but I still think we'd better to avoid dead paths "TODO" like this as much as possible.

Just do in the bottom-up way.

OK, it is better to be put at the latter part of the whole patch set. Would be fixed in the next version. Thanks.

Signature · 2 lines
Thanks,
Jeffle
author
#16 lore raw

PATCH 8/23 erofs: register global fscache volume

In reply to Jeffle Xu

All erofs instances will share one global fscache volume.

In this using scenario, one erofs instance could be mounted from one (or multiple) blob files instead of blkdev. The number of blob files that each erofs instance could correspond to is limited, since these blob files are quite large in size. For example, when used for container image distribution, one erofs instance used for container image for node.js will correspond to ~20 blob files in total. Thus in densely employed environment, there could be as many as hundreds of containers and thus thousands of fscache cookies under one fscache volume.

Then as for cachefiles backend, the hash table managing all cookies under one volume contains 32K slots. Thus the hashing functionality shall scale well in this case. Besides, cachefiles backend will scatter backing files under 256 fan sub-directoris, and thus the scalability of looking up backing files shall also not be an issue.

---
 fs/erofs/Makefile   |  2 +-
 fs/erofs/fscache.c  | 21 +++++++++++++++++++++
 fs/erofs/internal.h |  5 +++++
 fs/erofs/super.c    |  7 +++++++
 4 files changed, 34 insertions(+), 1 deletion(-)
 create mode 100644 fs/erofs/fscache.c
fs/erofs/Makefile +1 -1
index 756fe2d65272..f9a3609625aa 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 obj-$(CONFIG_EROFS_FS) += erofs.o
-erofs-objs := super.o inode.o data.o namei.o dir.o utils.o pcpubuf.o
+erofs-objs := super.o inode.o data.o namei.o dir.o utils.o pcpubuf.o fscache.o
 erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o
 erofs-$(CONFIG_EROFS_FS_ZIP) += decompressor.o zmap.o zdata.o
 erofs-$(CONFIG_EROFS_FS_ZIP_LZMA) += decompressor_lzma.o
fs/erofs/fscache.c new file +21 -0
new file mode 100644
index 000000000000..cdbbe3f15a20
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2021, Alibaba Cloud
+ */
+#include "internal.h"
+
+static struct fscache_volume *volume;
+
+int __init erofs_fscache_init(void)
+{
+	volume = fscache_acquire_volume("erofs", NULL, NULL, 0);
+	if (!volume)
+		return -EINVAL;
+
+	return 0;
+}
+
+void erofs_fscache_cleanup(void)
+{
+	fscache_relinquish_volume(volume, NULL, false);
+}
fs/erofs/internal.h +5 -0
index c9ee8c247202..d464642e9d35 100644
@@ -17,6 +17,7 @@
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 #include <linux/iomap.h>
+#include <linux/fscache.h>
 #include "erofs_fs.h"
 
 /* redefine pr_fmt "erofs: " */
@@ -569,6 +570,10 @@ static inline int z_erofs_load_lzma_config(struct super_block *sb,
 }
 #endif	/* !CONFIG_EROFS_FS_ZIP */
 
+/* fscache.c */
+int erofs_fscache_init(void);
+void erofs_fscache_cleanup(void);
+
 #define EFSCORRUPTED    EUCLEAN         /* Filesystem is corrupted */
 
 #endif	/* __EROFS_INTERNAL_H */
fs/erofs/super.c +7 -0
index 80c00c34eafc..517d74f3c303 100644
@@ -876,6 +876,10 @@ static int __init erofs_module_init(void)
 	if (err)
 		goto zip_err;
 
+	err = erofs_fscache_init();
+	if (err)
+		goto fscache_err;
+
 	err = register_filesystem(&erofs_fs_type);
 	if (err)
 		goto fs_err;
@@ -883,6 +887,8 @@ static int __init erofs_module_init(void)
 	return 0;
 
 fs_err:
+	erofs_fscache_cleanup();
+fscache_err:
 	z_erofs_exit_zip_subsystem();
 zip_err:
 	z_erofs_lzma_exit();
@@ -901,6 +907,7 @@ static void __exit erofs_module_exit(void)
 	/* Ensure all RCU free inodes / pclusters are safe to be destroyed. */
 	rcu_barrier();
 
+	erofs_fscache_cleanup();
 	z_erofs_exit_zip_subsystem();
 	z_erofs_lzma_exit();
 	erofs_exit_shrinker();
Signature · 1 line
2.27.0
author
#17 lore raw

PATCH 9/23 erofs: add cookie context helper functions

In reply to Jeffle Xu

Introduce 'struct erofs_cookie_ctx' for managing cookie for backing file, and the following introduced API for reading from backing file.

Besides, introduce two helper functions for initializing and cleaning up erofs_cookie_ctx.

struct erofs_cookie_ctx * erofs_fscache_get_ctx(struct super_block *sb, char *path);

void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx);

---
 fs/erofs/fscache.c  | 75 +++++++++++++++++++++++++++++++++++++++++++++
 fs/erofs/internal.h |  8 +++++
 2 files changed, 83 insertions(+)
fs/erofs/fscache.c +75 -0
index cdbbe3f15a20..15c5bb0f8ea5 100644
@@ -6,6 +6,81 @@
 
 static struct fscache_volume *volume;
 
+static int erofs_fscache_init_cookie(struct erofs_cookie_ctx *ctx, char *path)
+{
+	struct fscache_cookie *cookie;
+
+	/*
+	 * @object_size shall be non-zero to avoid
+	 * FSCACHE_COOKIE_NO_DATA_TO_READ.
+	 */
+	cookie = fscache_acquire_cookie(volume, 0,
+					path, strlen(path),
+					NULL, 0, -1);
+	if (!cookie)
+		return -EINVAL;
+
+	fscache_use_cookie(cookie, false);
+	ctx->cookie = cookie;
+	return 0;
+}
+
+static void erofs_fscache_cleanup_cookie(struct erofs_cookie_ctx *ctx)
+{
+	struct fscache_cookie *cookie = ctx->cookie;
+
+	fscache_unuse_cookie(cookie, NULL, NULL);
+	fscache_relinquish_cookie(cookie, false);
+	ctx->cookie = NULL;
+}
+
+static int erofs_fscahce_init_ctx(struct erofs_cookie_ctx *ctx,
+				  struct super_block *sb, char *path)
+{
+	int ret;
+
+	ret = erofs_fscache_init_cookie(ctx, path);
+	if (ret) {
+		erofs_err(sb, "failed to init cookie\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void erofs_fscache_cleanup_ctx(struct erofs_cookie_ctx *ctx)
+{
+	erofs_fscache_cleanup_cookie(ctx);
+}
+
+struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb,
+					       char *path)
+{
+	struct erofs_cookie_ctx *ctx;
+	int ret;
+
+	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)
+		return ERR_PTR(-ENOMEM);
+
+	ret = erofs_fscahce_init_ctx(ctx, sb, path);
+	if (ret) {
+		kfree(ctx);
+		return ERR_PTR(ret);
+	}
+
+	return ctx;
+}
+
+void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx)
+{
+	if (!ctx)
+		return;
+
+	erofs_fscache_cleanup_ctx(ctx);
+	kfree(ctx);
+}
+
 int __init erofs_fscache_init(void)
 {
 	volume = fscache_acquire_volume("erofs", NULL, NULL, 0);
fs/erofs/internal.h +8 -0
index d464642e9d35..4179c3dcb7f9 100644
@@ -91,6 +91,10 @@ struct erofs_sb_lz4_info {
 	u16 max_pclusterblks;
 };
 
+struct erofs_cookie_ctx {
+	struct fscache_cookie *cookie;
+};
+
 struct erofs_sb_info {
 	struct erofs_mount_opts opt;	/* options */
 #ifdef CONFIG_EROFS_FS_ZIP
@@ -574,6 +578,10 @@ static inline int z_erofs_load_lzma_config(struct super_block *sb,
 int erofs_fscache_init(void);
 void erofs_fscache_cleanup(void);
 
+struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb,
+					       char *path);
+void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx);
+
 #define EFSCORRUPTED    EUCLEAN         /* Filesystem is corrupted */
 
 #endif	/* __EROFS_INTERNAL_H */
Signature · 1 line
2.27.0
author
#18 lore raw

PATCH 10/23 erofs: add anonymous inode managing page cache of blob file

In reply to Jeffle Xu

Introduce one anonymous inode for managing page cache of corresponding blob file. Then erofs could read directly from the address space of the anonymous inode when cache hit.

---
 fs/erofs/fscache.c  | 35 +++++++++++++++++++++++++++++++++++
 fs/erofs/internal.h |  1 +
 2 files changed, 36 insertions(+)
fs/erofs/fscache.c +35 -0
index 15c5bb0f8ea5..4dfca7c95710 100644
@@ -34,6 +34,33 @@ static void erofs_fscache_cleanup_cookie(struct erofs_cookie_ctx *ctx)
 	ctx->cookie = NULL;
 }
 
+static const struct address_space_operations erofs_fscache_aops = {
+};
+
+static int erofs_fscache_get_inode(struct erofs_cookie_ctx *ctx,
+				   struct super_block *sb)
+{
+	struct inode *const inode = new_inode(sb);
+
+	if (!inode)
+		return -ENOMEM;
+
+	set_nlink(inode, 1);
+	inode->i_size = OFFSET_MAX;
+
+	inode->i_mapping->a_ops = &erofs_fscache_aops;
+	mapping_set_gfp_mask(inode->i_mapping,
+			GFP_NOFS | __GFP_HIGHMEM | __GFP_MOVABLE);
+	ctx->inode = inode;
+	return 0;
+}
+
+static void erofs_fscache_put_inode(struct erofs_cookie_ctx *ctx)
+{
+	iput(ctx->inode);
+	ctx->inode = NULL;
+}
+
 static int erofs_fscahce_init_ctx(struct erofs_cookie_ctx *ctx,
 				  struct super_block *sb, char *path)
 {
@@ -45,12 +72,20 @@ static int erofs_fscahce_init_ctx(struct erofs_cookie_ctx *ctx,
 		return ret;
 	}
 
+	ret = erofs_fscache_get_inode(ctx, sb);
+	if (ret) {
+		erofs_err(sb, "failed to get anonymous inode\n");
+		erofs_fscache_cleanup_cookie(ctx);
+		return ret;
+	}
+
 	return 0;
 }
 
 static void erofs_fscache_cleanup_ctx(struct erofs_cookie_ctx *ctx)
 {
 	erofs_fscache_cleanup_cookie(ctx);
+	erofs_fscache_put_inode(ctx);
 }
 
 struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb,
fs/erofs/internal.h +1 -0
index 4179c3dcb7f9..2e4f267b37e7 100644
@@ -93,6 +93,7 @@ struct erofs_sb_lz4_info {
 
 struct erofs_cookie_ctx {
 	struct fscache_cookie *cookie;
+	struct inode *inode;
 };
 
 struct erofs_sb_info {
Signature · 1 line
2.27.0
author
#19 lore raw

PATCH 11/23 erofs: register cookie context for bootstrap

In reply to Jeffle Xu

Something worth mentioning about the cleanup routine.

1. The init routine is prior to when the root inode gets initialized, and thus the corresponding cleanup routine shall be placed under .kill_sb() callback.

2. The init routine will instantiate anonymous inodes under the super_block, and thus .put_super() callback shall also contain the cleanup routine. Or we'll get "VFS: Busy inodes after unmount." warning.

---
 fs/erofs/internal.h |  2 ++
 fs/erofs/super.c    | 13 +++++++++++++
 2 files changed, 15 insertions(+)
fs/erofs/internal.h +2 -0
index 2e4f267b37e7..4ee4ff6774ba 100644
@@ -141,6 +141,8 @@ struct erofs_sb_info {
 	u8 volume_name[16];             /* volume name */
 	u32 feature_compat;
 	u32 feature_incompat;
+
+	struct erofs_cookie_ctx *bootstrap;
 };
 
 #define EROFS_SB(sb) ((struct erofs_sb_info *)(sb)->s_fs_info)
fs/erofs/super.c +13 -0
index 517d74f3c303..141cabd01d32 100644
@@ -663,6 +663,16 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc)
 	else
 		sbi->dax_dev = NULL;
 
+	if (!sb->s_bdev) {
+		struct erofs_cookie_ctx *bootstrap;
+
+		bootstrap = erofs_fscache_get_ctx(sb, ctx->opt.uuid);
+		if (IS_ERR(bootstrap))
+			return PTR_ERR(bootstrap);
+
+		sbi->bootstrap = bootstrap;
+	}
+
 	err = erofs_read_superblock(sb);
 	if (err)
 		return err;
@@ -820,6 +830,7 @@ static void erofs_kill_sb(struct super_block *sb)
 		return;
 
 	erofs_free_dev_context(sbi->devs);
+	erofs_fscache_put_ctx(sbi->bootstrap);
 	fs_put_dax(sbi->dax_dev);
 	kfree(sbi);
 	sb->s_fs_info = NULL;
@@ -837,6 +848,8 @@ static void erofs_put_super(struct super_block *sb)
 	iput(sbi->managed_cache);
 	sbi->managed_cache = NULL;
 #endif
+	erofs_fscache_put_ctx(sbi->bootstrap);
+	sbi->bootstrap = NULL;
 }
 
 static struct file_system_type erofs_fs_type = {
Signature · 1 line
2.27.0
author
#20 lore raw

PATCH 12/23 erofs: implement fscache-based metadata read

In reply to Jeffle Xu

This patch implements the data plane of reading metadata from bootstrap blob file over fscache.

Be noted that currently it only supports the scenario where the backing file has no hole. Once it hits a hole of the backing file, erofs will fail the IO with -EOPNOTSUPP for now. The following patch will fix this issue, i.e. implementing the demand reading mode.

---
 fs/erofs/data.c     |  4 +--
 fs/erofs/fscache.c  | 59 +++++++++++++++++++++++++++++++++++++++++++++
 fs/erofs/internal.h |  3 +++
 fs/erofs/super.c    |  7 ++----
 4 files changed, 66 insertions(+), 7 deletions(-)
fs/erofs/data.c +2 -2
index 61fa431d0713..f0857c285fac 100644
@@ -19,8 +19,8 @@ struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr)
 		page = read_cache_page_gfp(mapping, blkaddr,
 				mapping_gfp_constraint(mapping, ~__GFP_FS));
 	} else {
-		/* TODO: data path in nodev mode */
-		page = ERR_PTR(-EINVAL);
+		page = erofs_readpage_from_fscache(EROFS_SB(sb)->bootstrap,
+						   blkaddr);
 	}
 
 	/* should already be PageUptodate */
fs/erofs/fscache.c +59 -0

65 lines not on this page · raw message

fs/erofs/internal.h +3 -0
index 4ee4ff6774ba..10fb7ef26ddf 100644
@@ -585,6 +585,9 @@ struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb,
 					       char *path);
 void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx);
 
+struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx,
+					 pgoff_t index);
+
 #define EFSCORRUPTED    EUCLEAN         /* Filesystem is corrupted */
 
 #endif	/* __EROFS_INTERNAL_H */
fs/erofs/super.c +2 -5
index 141cabd01d32..0e5964d8b24b 100644
@@ -334,25 +334,22 @@ static int erofs_init_devices(struct super_block *sb,
 
 static int erofs_read_superblock(struct super_block *sb)
 {
-	struct erofs_sb_info *sbi;
+	struct erofs_sb_info *sbi = EROFS_SB(sb);
 	struct page *page;
 	struct erofs_super_block *dsb;
 	unsigned int blkszbits;
 	void *data;
 	int ret;
 
-	/* TODO: metadata path in nodev mode */
 	if (sb->s_bdev)
 		page = read_mapping_page(sb->s_bdev->bd_inode->i_mapping, 0, NULL);
 	else
-		page = ERR_PTR(-EOPNOTSUPP);
+		page = erofs_readpage_from_fscache(sbi->bootstrap, 0);
 	if (IS_ERR(page)) {
 		erofs_err(sb, "cannot read erofs superblock");
 		return PTR_ERR(page);
 	}
 
-	sbi = EROFS_SB(sb);
-
 	data = kmap(page);
 	dsb = (struct erofs_super_block *)(data + EROFS_SUPER_OFFSET);
 
Signature · 1 line
2.27.0
#21 lore raw

Re: [PATCH v1 12/23] erofs: implement fscache-based metadata read

Thank you for the patch! Yet something to improve:

Hi Jeffle,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on dhowells-fs/fscache-next]
[cannot apply to xiang-erofs/dev-test ceph-client/for-linus linus/master v5.16-rc7 next-20211224] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742
base:   https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git fscache-next
config: m68k-defconfig (https://download.01.org/0day-ci/archive/20211228/202112280115.O0H8Ow1Q-lkp(a)intel.com/config)
compiler: m68k-linux-gcc (GCC) 11.2.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/c3453b91df3b4e89c3336453437f761d6cb6bca3
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742 git checkout c3453b91df3b4e89c3336453437f761d6cb6bca3
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=m68k SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>, old ones prefixed by <<):

ERROR: modpost: "netfs_subreq_terminated" [fs/erofs/erofs.ko] undefined!

>> ERROR: modpost: "netfs_readpage" [fs/erofs/erofs.ko] undefined!

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#22 lore raw

Re: [PATCH v1 12/23] erofs: implement fscache-based metadata read

In reply to Jeffle Xu
Thank you for the patch! Yet something to improve:

Hi Jeffle,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on dhowells-fs/fscache-next]
[cannot apply to xiang-erofs/dev-test ceph-client/for-linus linus/master v5.16-rc7 next-20211224] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742
base:   https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git fscache-next
config: x86_64-randconfig-a014-20211227 (https://download.01.org/0day-ci/archive/20211228/202112280132.0kJ8Vjql-lkp(a)intel.com/config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/c3453b91df3b4e89c3336453437f761d6cb6bca3
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742
        git checkout c3453b91df3b4e89c3336453437f761d6cb6bca3
        # save the config file to linux build tree
        mkdir build_dir
        make W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

vmlinux.o: warning: objtool: do_machine_check()+0x89f: call to queue_task_work() leaves .noinstr.text section vmlinux.o: warning: objtool: enter_from_user_mode()+0x4e: call to on_thread_stack() leaves .noinstr.text section vmlinux.o: warning: objtool: syscall_enter_from_user_mode()+0x59: call to on_thread_stack() leaves .noinstr.text section vmlinux.o: warning: objtool: syscall_enter_from_user_mode_prepare()+0x4e: call to on_thread_stack() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_enter_from_user_mode()+0x4e: call to on_thread_stack() leaves .noinstr.text section
ld: fs/erofs/fscache.o: in function `erofs_issue_op':
fs/erofs/fscache.c:27: undefined reference to `netfs_subreq_terminated'
ld: fs/erofs/fscache.o: in function `erofs_readpage_from_fscache':

>> fs/erofs/fscache.c:59: undefined reference to `netfs_readpage'

vim +59 fs/erofs/fscache.c

    35	
    36	struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx,
    37						 pgoff_t index)
    38	{
    39		struct folio *folio;
    40		struct page *page;
    41		struct super_block *sb = ctx->inode->i_sb;
    42		int ret;
    43	
    44		page = find_or_create_page(ctx->inode->i_mapping, index, GFP_KERNEL);
    45		if (unlikely(!page)) {
    46			erofs_err(sb, "failed to allocate page");
    47			return ERR_PTR(-ENOMEM);
    48		}
    49	
    50		/* The content is already buffered in the address space */
    51		if (PageUptodate(page)) {
    52			unlock_page(page);
    53			return page;
    54		}
    55	
    56		/* Or a new page cache is created, then read the content from fscache */
    57		folio = page_folio(page);
    58	
  > 59		ret = netfs_readpage(NULL, folio, &erofs_req_ops, ctx->cookie);
    60		if (unlikely(ret || !PageUptodate(page))) {
    61			erofs_err(sb, "failed to read from fscache");
    62			return ERR_PTR(-EINVAL);
    63		}
    64	
    65		return page;
    66	}
    67	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#23 lore raw

PATCH 13/23 erofs: implement fscache-based data read

In reply to Jeffle Xu

This patch implements the data plane of reading data from bootstrap blob file over fscache.

Be noted that currently compressed layout is not supported yet.

---
 fs/erofs/fscache.c  | 91 +++++++++++++++++++++++++++++++++++++++++++++
 fs/erofs/inode.c    |  6 ++-
 fs/erofs/internal.h |  1 +
 3 files changed, 97 insertions(+), 1 deletion(-)
fs/erofs/fscache.c +91 -0
index 325f5663836b..bfcec831d58a 100644
@@ -65,6 +65,97 @@ struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx,
 	return page;
 }
 
+static inline void do_copy_page(struct page *from, struct page *to,
+				size_t offset, size_t len)
+{
+	char *vfrom, *vto;
+
+	vfrom = kmap_atomic(from);
+	vto = kmap_atomic(to);
+	memcpy(vto, vfrom + offset, len);
+	kunmap_atomic(vto);
+	kunmap_atomic(vfrom);
+}
+
+static int erofs_fscache_do_readpage(struct file *file, struct page *page)
+{
+	struct inode *inode = page->mapping->host;
+	struct erofs_inode *vi = EROFS_I(inode);
+	struct super_block *sb = inode->i_sb;
+	struct erofs_map_blocks map;
+	erofs_off_t o_la, pa;
+	size_t offset, len;
+	struct page *ipage;
+	int ret;
+
+	if (erofs_inode_is_data_compressed(vi->datalayout)) {
+		erofs_info(sb, "compressed layout not supported yet");
+		return -EOPNOTSUPP;
+	}
+
+	o_la = page_offset(page);
+	map.m_la = o_la;
+
+	ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
+	if (ret)
+		return ret;
+
+	if (!(map.m_flags & EROFS_MAP_MAPPED)) {
+		zero_user(page, 0, PAGE_SIZE);
+		return 0;
+	}
+
+	/*
+	 * 1) For FLAT_PLAIN/FLAT_INLINE layout, the output map.m_la shall be
+	 * equal to o_la, and the output map.m_pa is exactly the physical
+	 * address of o_la.
+	 * 2) For CHUNK_BASED layout, the output map.m_la is rounded down to the
+	 * nearest chunk boundary, and the output map.m_pa is actually the
+	 * physical address of this chunk boundary. So we need to recalculate
+	 * the actual physical address of o_la.
+	 */
+	pa = map.m_pa + o_la - map.m_la;
+
+	ipage = erofs_get_meta_page(sb, erofs_blknr(pa));
+	if (IS_ERR(ipage))
+		return PTR_ERR(ipage);
+
+	/*
+	 * @offset refers to the page offset inside @ipage.
+	 * 1) Except for the inline layout, the offset shall all be 0, and @pa
+	 * shall be aligned with EROFS_BLKSIZ in this case. Thus we can
+	 * conveniently get the offset from @pa.
+	 * 2) While for the inline layout, the offset may be non-zero. Since
+	 * currently only flat layout supports inline, we can calculate the
+	 * offset from the corresponding physical address.
+	 */
+	offset = erofs_blkoff(pa);
+	len = min_t(u64, map.m_llen, PAGE_SIZE);
+
+	do_copy_page(ipage, page, offset, len);
+
+	unlock_page(ipage);
+	return 0;
+}
+
+static int erofs_fscache_readpage(struct file *file, struct page *page)
+{
+	int ret;
+
+	ret = erofs_fscache_do_readpage(file, page);
+	if (ret)
+		SetPageError(page);
+	else
+		SetPageUptodate(page);
+
+	unlock_page(page);
+	return ret;
+}
+
+const struct address_space_operations erofs_fscache_access_aops = {
+	.readpage = erofs_fscache_readpage,
+};
+
 static int erofs_fscache_init_cookie(struct erofs_cookie_ctx *ctx, char *path)
 {
 	struct fscache_cookie *cookie;
fs/erofs/inode.c +5 -1
index 2345f1de438e..452d147277c4 100644
@@ -299,7 +299,11 @@ static int erofs_fill_inode(struct inode *inode, int isdir)
 		err = z_erofs_fill_inode(inode);
 		goto out_unlock;
 	}
-	inode->i_mapping->a_ops = &erofs_raw_access_aops;
+
+	if (inode->i_sb->s_bdev)
+		inode->i_mapping->a_ops = &erofs_raw_access_aops;
+	else
+		inode->i_mapping->a_ops = &erofs_fscache_access_aops;
 
 out_unlock:
 	unlock_page(page);
fs/erofs/internal.h +1 -0
index 10fb7ef26ddf..f3a1aa429a93 100644
@@ -357,6 +357,7 @@ struct page *erofs_grab_cache_page_nowait(struct address_space *mapping,
 extern const struct super_operations erofs_sops;
 
 extern const struct address_space_operations erofs_raw_access_aops;
+extern const struct address_space_operations erofs_fscache_access_aops;
 extern const struct address_space_operations z_erofs_aops;
 
 /*
Signature · 1 line
2.27.0
author
#24 lore raw

Re: [PATCH v1 13/23] erofs: implement fscache-based data read

fs/erofs/fscache.c struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx, · line 79

Quoted 13 lines · Jeffle Xu
fs/erofs/fscache.c +12 -0
+static inline void do_copy_page(struct page *from, struct page *to,
+				size_t offset, size_t len)
+{
+	char *vfrom, *vto;
+
+	vfrom = kmap_atomic(from);
+	vto = kmap_atomic(to);
+	memcpy(vto, vfrom + offset, len);
+	kunmap_atomic(vto);
+	kunmap_atomic(vfrom);
+}
+

It seems that this private function can be replaced by memcpy_page(). Will be done in the next version.

Signature · 2 lines
Thanks,
Jeffle
#25 lore raw

Re: [PATCH v1 13/23] erofs: implement fscache-based data read

In reply to Jeffle Xu
Quoted 78 lines · Jeffle Xu

This patch implements the data plane of reading data from bootstrap blob file over fscache.

Be noted that currently compressed layout is not supported yet.

---
 fs/erofs/fscache.c  | 91 +++++++++++++++++++++++++++++++++++++++++++++
 fs/erofs/inode.c    |  6 ++-
 fs/erofs/internal.h |  1 +
 3 files changed, 97 insertions(+), 1 deletion(-)
fs/erofs/fscache.c +58 -0
index 325f5663836b..bfcec831d58a 100644
@@ -65,6 +65,97 @@ struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx,
 	return page;
 }
 
+static inline void do_copy_page(struct page *from, struct page *to,
+				size_t offset, size_t len)
+{
+	char *vfrom, *vto;
+
+	vfrom = kmap_atomic(from);
+	vto = kmap_atomic(to);
+	memcpy(vto, vfrom + offset, len);
+	kunmap_atomic(vto);
+	kunmap_atomic(vfrom);
+}
+
+static int erofs_fscache_do_readpage(struct file *file, struct page *page)
+{
+	struct inode *inode = page->mapping->host;
+	struct erofs_inode *vi = EROFS_I(inode);
+	struct super_block *sb = inode->i_sb;
+	struct erofs_map_blocks map;
+	erofs_off_t o_la, pa;
+	size_t offset, len;
+	struct page *ipage;
+	int ret;
+
+	if (erofs_inode_is_data_compressed(vi->datalayout)) {
+		erofs_info(sb, "compressed layout not supported yet");
+		return -EOPNOTSUPP;
+	}
+
+	o_la = page_offset(page);
+	map.m_la = o_la;
+
+	ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
+	if (ret)
+		return ret;
+
+	if (!(map.m_flags & EROFS_MAP_MAPPED)) {
+		zero_user(page, 0, PAGE_SIZE);
+		return 0;
+	}
+
+	/*
+	 * 1) For FLAT_PLAIN/FLAT_INLINE layout, the output map.m_la shall be
+	 * equal to o_la, and the output map.m_pa is exactly the physical
+	 * address of o_la.
+	 * 2) For CHUNK_BASED layout, the output map.m_la is rounded down to the
+	 * nearest chunk boundary, and the output map.m_pa is actually the
+	 * physical address of this chunk boundary. So we need to recalculate
+	 * the actual physical address of o_la.
+	 */
+	pa = map.m_pa + o_la - map.m_la;
+
+	ipage = erofs_get_meta_page(sb, erofs_blknr(pa));
+	if (IS_ERR(ipage))
+		return PTR_ERR(ipage);
+
+	/*
+	 * @offset refers to the page offset inside @ipage.
+	 * 1) Except for the inline layout, the offset shall all be 0, and @pa

fs/erofs/fscache.c struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx, · line 135

Quoted 10 lines
fs/erofs/fscache.c +10 -0
+	 * shall be aligned with EROFS_BLKSIZ in this case. Thus we can
+	 * conveniently get the offset from @pa.
+	 * 2) While for the inline layout, the offset may be non-zero. Since
+	 * currently only flat layout supports inline, we can calculate the
+	 * offset from the corresponding physical address.
+	 */
+	offset = erofs_blkoff(pa);
+	len = min_t(u64, map.m_llen, PAGE_SIZE);
+
+	do_copy_page(ipage, page, offset, len);

If my understanding is correct, I still have no idea why we need to copy data here even if fscache can do direct I/O for us without extra efforts.

I think the only case would be tail-packing inline (which should go through metadata path), otherwise, all data is block-aligned. So fscache can handle it directly.

Thanks,
Gao Xiang

author
#26 lore raw

Re: [PATCH v1 13/23] erofs: implement fscache-based data read

Quoted 94 lines · Gao Xiang
Quoted 78 lines · Jeffle Xu

This patch implements the data plane of reading data from bootstrap blob file over fscache.

Be noted that currently compressed layout is not supported yet.

---
 fs/erofs/fscache.c  | 91 +++++++++++++++++++++++++++++++++++++++++++++
 fs/erofs/inode.c    |  6 ++-
 fs/erofs/internal.h |  1 +
 3 files changed, 97 insertions(+), 1 deletion(-)
fs/erofs/fscache.c +58 -0
index 325f5663836b..bfcec831d58a 100644
@@ -65,6 +65,97 @@ struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx,
 	return page;
 }
 
+static inline void do_copy_page(struct page *from, struct page *to,
+				size_t offset, size_t len)
+{
+	char *vfrom, *vto;
+
+	vfrom = kmap_atomic(from);
+	vto = kmap_atomic(to);
+	memcpy(vto, vfrom + offset, len);
+	kunmap_atomic(vto);
+	kunmap_atomic(vfrom);
+}
+
+static int erofs_fscache_do_readpage(struct file *file, struct page *page)
+{
+	struct inode *inode = page->mapping->host;
+	struct erofs_inode *vi = EROFS_I(inode);
+	struct super_block *sb = inode->i_sb;
+	struct erofs_map_blocks map;
+	erofs_off_t o_la, pa;
+	size_t offset, len;
+	struct page *ipage;
+	int ret;
+
+	if (erofs_inode_is_data_compressed(vi->datalayout)) {
+		erofs_info(sb, "compressed layout not supported yet");
+		return -EOPNOTSUPP;
+	}
+
+	o_la = page_offset(page);
+	map.m_la = o_la;
+
+	ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
+	if (ret)
+		return ret;
+
+	if (!(map.m_flags & EROFS_MAP_MAPPED)) {
+		zero_user(page, 0, PAGE_SIZE);
+		return 0;
+	}
+
+	/*
+	 * 1) For FLAT_PLAIN/FLAT_INLINE layout, the output map.m_la shall be
+	 * equal to o_la, and the output map.m_pa is exactly the physical
+	 * address of o_la.
+	 * 2) For CHUNK_BASED layout, the output map.m_la is rounded down to the
+	 * nearest chunk boundary, and the output map.m_pa is actually the
+	 * physical address of this chunk boundary. So we need to recalculate
+	 * the actual physical address of o_la.
+	 */
+	pa = map.m_pa + o_la - map.m_la;
+
+	ipage = erofs_get_meta_page(sb, erofs_blknr(pa));
+	if (IS_ERR(ipage))
+		return PTR_ERR(ipage);
+
+	/*
+	 * @offset refers to the page offset inside @ipage.
+	 * 1) Except for the inline layout, the offset shall all be 0, and @pa

fs/erofs/fscache.c struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx, · line 135

Quoted 10 lines
fs/erofs/fscache.c +10 -0
+	 * shall be aligned with EROFS_BLKSIZ in this case. Thus we can
+	 * conveniently get the offset from @pa.
+	 * 2) While for the inline layout, the offset may be non-zero. Since
+	 * currently only flat layout supports inline, we can calculate the
+	 * offset from the corresponding physical address.
+	 */
+	offset = erofs_blkoff(pa);
+	len = min_t(u64, map.m_llen, PAGE_SIZE);
+
+	do_copy_page(ipage, page, offset, len);

If my understanding is correct, I still have no idea why we need to copy data here even if fscache can do direct I/O for us without extra efforts.

Quoted 4 lines

I think the only case would be tail-packing inline (which should go through metadata path), otherwise, all data is block-aligned. So fscache can handle it directly.

Right, only tail packing need special handling here. Would be fixed in the next version. Thanks.

Signature · 2 lines
Thanks,
Jeffle
author
#27 lore raw

PATCH 14/23 erofs: register cookie context for data blobs

In reply to Jeffle Xu

Similar to the multi device mode, erofs could be mounted from multiple blob files (one bootstrap blob file and optional multiple data blob files). In this case, each device slot contains the path of corresponding data blob file.

This patch registers corresponding cookie context for each data blob file. Similarly, the cleanup routine shall be contained in both .kill_sb() and .put_super() callback for the same reason.

---
 fs/erofs/internal.h |  1 +
 fs/erofs/super.c    | 12 +++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
fs/erofs/internal.h +1 -0
index f3a1aa429a93..f60577a7aade 100644
@@ -52,6 +52,7 @@ struct erofs_device_info {
 	char *path;
 	struct block_device *bdev;
 	struct dax_device *dax_dev;
+	struct erofs_cookie_ctx *ctx;
 
 	u32 blocks;
 	u32 mapped_blkaddr;
fs/erofs/super.c +11 -1
index 0e5964d8b24b..ea56122f7a35 100644
@@ -287,6 +287,7 @@ static int erofs_init_devices(struct super_block *sb,
 	idr_for_each_entry(&sbi->devs->tree, dif, id) {
 		erofs_blk_t blk = erofs_blknr(pos);
 		struct block_device *bdev;
+		struct erofs_cookie_ctx *ctx;
 
 		if (!page || page->index != blk) {
 			if (page) {
@@ -315,7 +316,12 @@ static int erofs_init_devices(struct super_block *sb,
 			dif->bdev = bdev;
 			dif->dax_dev = fs_dax_get_by_bdev(bdev);
 		} else {
-			/* TODO: multi devs in nodev mode */
+			ctx = erofs_fscache_get_ctx(sb, dif->path);
+			if (IS_ERR(ctx)) {
+				err = PTR_ERR(ctx);
+				goto err_out;
+			}
+			dif->ctx = ctx;
 		}
 		dif->blocks = le32_to_cpu(dis->blocks);
 		dif->mapped_blkaddr = le32_to_cpu(dis->mapped_blkaddr);
@@ -755,6 +761,7 @@ static int erofs_release_device_info(int id, void *ptr, void *data)
 {
 	struct erofs_device_info *dif = ptr;
 
+	erofs_fscache_put_ctx(dif->ctx);
 	fs_put_dax(dif->dax_dev);
 	if (dif->bdev)
 		blkdev_put(dif->bdev, FMODE_READ | FMODE_EXCL);
@@ -845,6 +852,9 @@ static void erofs_put_super(struct super_block *sb)
 	iput(sbi->managed_cache);
 	sbi->managed_cache = NULL;
 #endif
+	erofs_free_dev_context(sbi->devs);
+	sbi->devs = NULL;
+
 	erofs_fscache_put_ctx(sbi->bootstrap);
 	sbi->bootstrap = NULL;
 }
Signature · 1 line
2.27.0
author
#28 lore raw

PATCH 15/23 erofs: implement fscache-based data read for data blobs

In reply to Jeffle Xu

This patch implements the data plane of reading data from data blob file over fscache.

---
 fs/erofs/data.c     |  3 +++
 fs/erofs/fscache.c  | 25 +++++++++++++++++++++++--
 fs/erofs/internal.h |  1 +
 3 files changed, 27 insertions(+), 2 deletions(-)
fs/erofs/data.c +3 -0
index f0857c285fac..621769ab5753 100644
@@ -166,6 +166,7 @@ int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map)
 	/* primary device by default */
 	map->m_bdev = sb->s_bdev;
 	map->m_daxdev = EROFS_SB(sb)->dax_dev;
+	map->m_ctx = EROFS_SB(sb)->bootstrap;
 
 	if (map->m_deviceid) {
 		down_read(&devs->rwsem);
@@ -176,6 +177,7 @@ int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map)
 		}
 		map->m_bdev = dif->bdev;
 		map->m_daxdev = dif->dax_dev;
+		map->m_ctx = dif->ctx;
 		up_read(&devs->rwsem);
 	} else if (devs->extra_devices) {
 		down_read(&devs->rwsem);
@@ -192,6 +194,7 @@ int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map)
 				map->m_pa -= startoff;
 				map->m_bdev = dif->bdev;
 				map->m_daxdev = dif->dax_dev;
+				map->m_ctx = dif->ctx;
 				break;
 			}
 		}
fs/erofs/fscache.c +23 -2
index bfcec831d58a..7d4f6682e521 100644
@@ -77,12 +77,24 @@ static inline void do_copy_page(struct page *from, struct page *to,
 	kunmap_atomic(vfrom);
 }
 
+static struct page *erofs_fscache_get_page(struct erofs_cookie_ctx *ctx,
+					   erofs_blk_t blkaddr)
+{
+	struct page *page;
+
+	page = erofs_readpage_from_fscache(ctx, blkaddr);
+	if (!IS_ERR(page))
+		lock_page(page);
+	return page;
+}
+
 static int erofs_fscache_do_readpage(struct file *file, struct page *page)
 {
 	struct inode *inode = page->mapping->host;
 	struct erofs_inode *vi = EROFS_I(inode);
 	struct super_block *sb = inode->i_sb;
 	struct erofs_map_blocks map;
+	struct erofs_map_dev mdev;
 	erofs_off_t o_la, pa;
 	size_t offset, len;
 	struct page *ipage;
@@ -105,6 +117,15 @@ static int erofs_fscache_do_readpage(struct file *file, struct page *page)
 		return 0;
 	}
 
+	mdev = (struct erofs_map_dev) {
+		.m_deviceid = map.m_deviceid,
+		.m_pa = map.m_pa,
+	};
+
+	ret = erofs_map_dev(inode->i_sb, &mdev);
+	if (ret)
+		return ret;
+
 	/*
 	 * 1) For FLAT_PLAIN/FLAT_INLINE layout, the output map.m_la shall be
 	 * equal to o_la, and the output map.m_pa is exactly the physical
@@ -114,9 +135,9 @@ static int erofs_fscache_do_readpage(struct file *file, struct page *page)
 	 * physical address of this chunk boundary. So we need to recalculate
 	 * the actual physical address of o_la.
 	 */
-	pa = map.m_pa + o_la - map.m_la;
+	pa = mdev.m_pa + o_la - map.m_la;
 
-	ipage = erofs_get_meta_page(sb, erofs_blknr(pa));
+	ipage = erofs_fscache_get_page(mdev.m_ctx, erofs_blknr(pa));
 	if (IS_ERR(ipage))
 		return PTR_ERR(ipage);
 
fs/erofs/internal.h +1 -0
index f60577a7aade..b21c3751e88d 100644
@@ -447,6 +447,7 @@ static inline int z_erofs_map_blocks_iter(struct inode *inode,
 struct erofs_map_dev {
 	struct block_device *m_bdev;
 	struct dax_device *m_daxdev;
+	struct erofs_cookie_ctx *m_ctx;
 
 	erofs_off_t m_pa;
 	unsigned int m_deviceid;
Signature · 1 line
2.27.0
author
#29 lore raw

PATCH 16/23 erofs: add 'uuid' mount option

In reply to Jeffle Xu

Introduce 'uuid' mount option to enable the nodev mode, in which erofs could be mounted from blob files instead of blkdev. By then users could specify the path of bootstrap blob file containing the complete erofs image.

---
 fs/erofs/super.c | 9 +++++++++
 1 file changed, 9 insertions(+)
fs/erofs/super.c +9 -0
index ea56122f7a35..4f17aedc4acd 100644
@@ -451,6 +451,7 @@ enum {
 	Opt_dax,
 	Opt_dax_enum,
 	Opt_device,
+	Opt_uuid,
 	Opt_err
 };
 
@@ -475,6 +476,7 @@ static const struct fs_parameter_spec erofs_fs_parameters[] = {
 	fsparam_flag("dax",             Opt_dax),
 	fsparam_enum("dax",		Opt_dax_enum, erofs_dax_param_enums),
 	fsparam_string("device",	Opt_device),
+	fsparam_string("uuid",		Opt_uuid),
 	{}
 };
 
@@ -570,6 +572,12 @@ static int erofs_fc_parse_param(struct fs_context *fc,
 		}
 		++ctx->devs->extra_devices;
 		break;
+	case Opt_uuid:
+		kfree(ctx->opt.uuid);
+		ctx->opt.uuid = kstrdup(param->string, GFP_KERNEL);
+		if (!ctx->opt.uuid)
+			return -ENOMEM;
+		break;
 	default:
 		return -ENOPARAM;
 	}
@@ -784,6 +792,7 @@ static void erofs_fc_free(struct fs_context *fc)
 	struct erofs_fs_context *ctx = fc->fs_private;
 
 	erofs_free_dev_context(ctx->devs);
+	kfree(ctx->opt.uuid);
 	kfree(ctx);
 }
 
Signature · 1 line
2.27.0
author
#30 lore raw

PATCH 17/23 netfs: support on demand read

In reply to Jeffle Xu

Add demand_read() callback to netfs_cache_ops to implement demand reading.

The precondition for implementing demand reading semantic is that, all blob files are sparse and have been placed under corresponding directory on the first beginning. When upper fs starts to access the blob file, it will "cache miss" (hit the hole) and then .issue_op() callback will be called to prepare the data.

Then the working flow is as described below. The .issue_op() callback could be implemented by netfs_demand_read() helper, which will in turn call .demand_read() callback of corresponding fscache backend to prepare the data.

The implementation of .demand_read() callback can be backend specific. The following patch will introduce an implementation of .demand_read() callback for cachefiles, which will notify user daemon the requested file range to read. The .demand_read() callback will get blocked until the user daemon has prepared the corresponding data.

Then once .demand_read() callback returns with 0, it is guaranteed that the requested data has been ready. In this case, transform this IO request to NETFS_READ_FROM_CACHE state, initiate an incomplete completion and then retry to read from backing file.

---
 fs/netfs/read_helper.c | 26 ++++++++++++++++++++++++++
 include/linux/netfs.h  |  4 ++++
 2 files changed, 30 insertions(+)
fs/netfs/read_helper.c +26 -0
index ca84918b6b5d..8aac65132b67 100644
@@ -1006,6 +1006,32 @@ int netfs_readpage(struct file *file,
 }
 EXPORT_SYMBOL(netfs_readpage);
 
+void netfs_demand_read(struct netfs_read_subrequest *subreq)
+{
+	struct netfs_read_request *rreq = subreq->rreq;
+	struct netfs_cache_resources *cres = &rreq->cache_resources;
+	loff_t start_pos;
+	size_t len;
+	int ret;
+
+	start_pos = subreq->start + subreq->transferred;
+	len = subreq->len - subreq->transferred;
+
+	/*
+	 * In success case (ret == 0), user daemon has downloaded data for us,
+	 * thus transform to NETFS_READ_FROM_CACHE state and advertise that 0
+	 * byte readed, so that the request will enter into INCOMPLETE state and
+	 * retry to read from backing file.
+	 */
+	ret = cres->ops->demand_read(cres, start_pos, len);
+	if (!ret) {
+		subreq->source = NETFS_READ_FROM_CACHE;
+		__clear_bit(NETFS_SREQ_WRITE_TO_CACHE, &subreq->flags);
+	}
+
+	netfs_subreq_terminated(subreq, ret, false);
+}
+
 /*
  * Prepare a folio for writing without reading first
  * @folio: The folio being prepared
include/linux/netfs.h +4 -0
index b46c39d98bbd..80a738762deb 100644
@@ -244,6 +244,9 @@ struct netfs_cache_ops {
 	int (*prepare_write)(struct netfs_cache_resources *cres,
 			     loff_t *_start, size_t *_len, loff_t i_size,
 			     bool no_space_allocated_yet);
+
+	int (*demand_read)(struct netfs_cache_resources *cres,
+			   loff_t start_pos, size_t len);
 };
 
 struct readahead_control;
@@ -259,6 +262,7 @@ extern int netfs_write_begin(struct file *, struct address_space *,
 			     void **,
 			     const struct netfs_read_request_ops *,
 			     void *);
+extern void netfs_demand_read(struct netfs_read_subrequest *);
 
 extern void netfs_subreq_terminated(struct netfs_read_subrequest *, ssize_t, bool);
 extern void netfs_stats_show(struct seq_file *);
Signature · 1 line
2.27.0
author
#31 lore raw

PATCH 18/23 cachefiles: use idr tree managing pending demand read

In reply to Jeffle Xu

This is in prep for implementing .demand_read() callback for cachefiles.

In the following patch, .demand_read() callback is responsible for notifying user daemon the requested file range (hole) to read. Then .demand_read() callback will get blocked until user daemon has prepared data and filled the hole.

This patch introduces an idr tree to manage all these pending demand-read IO.

---
 fs/cachefiles/daemon.c   | 3 +++
 fs/cachefiles/internal.h | 4 ++++
 2 files changed, 7 insertions(+)
fs/cachefiles/daemon.c +3 -0
index 892a9bdba53f..01496fa8c263 100644
@@ -118,6 +118,8 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)
 	INIT_LIST_HEAD(&cache->volumes);
 	INIT_LIST_HEAD(&cache->object_list);
 	spin_lock_init(&cache->object_list_lock);
+	idr_init(&cache->reqs);
+	spin_lock_init(&cache->reqs_lock);
 
 	/* set default caching limits
 	 * - limit at 1% free space and/or free files
@@ -152,6 +154,7 @@ static int cachefiles_daemon_release(struct inode *inode, struct file *file)
 	cachefiles_daemon_unbind(cache);
 
 	/* clean up the control file interface */
+	idr_destroy(&cache->reqs);
 	cache->cachefilesd = NULL;
 	file->private_data = NULL;
 	cachefiles_open = 0;
fs/cachefiles/internal.h +4 -0
index a8e6500889d7..af8ac8107f77 100644
@@ -15,6 +15,7 @@
 #include <linux/fscache-cache.h>
 #include <linux/cred.h>
 #include <linux/security.h>
+#include <linux/idr.h>
 
 #define CACHEFILES_DIO_BLOCK_SIZE 4096
 
@@ -102,6 +103,9 @@ struct cachefiles_cache {
 	char				*rootdirname;	/* name of cache root directory */
 	char				*secctx;	/* LSM security context */
 	char				*tag;		/* cache binding tag */
+
+	struct idr			reqs;
+	spinlock_t			reqs_lock;
 };
 
 #include <trace/events/cachefiles.h>
Signature · 1 line
2.27.0
author
#32 lore raw

PATCH 19/23 cachefiles: implement .demand_read() for demand read

In reply to Jeffle Xu

Implement .demand_read() callback for cachefiels backend.

.demand_read() callback is responsible for notifying user daemon the pending request to process, and will get blocked until user daemon has prepared data and filled the hole.

---
 fs/cachefiles/internal.h | 12 +++++++++
 fs/cachefiles/io.c       | 56 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
fs/cachefiles/internal.h +12 -0
index af8ac8107f77..eeb6ad7dcd49 100644
@@ -61,6 +61,18 @@ struct cachefiles_object {
 #define CACHEFILES_OBJECT_USING_TMPFILE	0		/* Have an unlinked tmpfile */
 };
 
+struct cachefiles_req_in {
+	uint64_t id;
+	uint64_t off;
+	uint64_t len;
+	char path[NAME_MAX];
+};
+
+struct cachefiles_req {
+	struct completion done;
+	struct cachefiles_req_in req_in;
+};
+
 /*
  * Cache files cache definition
  */
fs/cachefiles/io.c +56 -0

68 lines not on this page · raw message

Signature · 1 line
2.27.0
#33 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

Thank you for the patch! Perhaps something to improve:

Hi Jeffle,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on dhowells-fs/fscache-next]
[cannot apply to xiang-erofs/dev-test ceph-client/for-linus linus/master v5.16-rc7 next-20211224] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742
base:   https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git fscache-next
config: csky-defconfig (https://download.01.org/0day-ci/archive/20211227/202112272340.1ho7sEjG-lkp(a)intel.com/config)
compiler: csky-linux-gcc (GCC) 11.2.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/54c300b4598e3f2836d8233681da387fe388cfda
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742 git checkout 54c300b4598e3f2836d8233681da387fe388cfda
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=csky SHELL=/bin/bash fs/cachefiles/

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

fs/cachefiles/io.c:564:5: warning: no previous prototype for 'cachefiles_demand_read' [-Wmissing-prototypes]
564 | int cachefiles_demand_read(struct netfs_cache_resources *cres,
| ^~~~~~~~~~~~~~~~~~~~~~
In function 'cachefiles_alloc_req',
inlined from 'cachefiles_demand_read' at fs/cachefiles/io.c:575:8:

>> fs/cachefiles/io.c:557:9: warning: 'strncpy' specified bound 255 equals destination size [-Wstringop-truncation]
     557 |         strncpy(req_in->path, object->d_name, sizeof(req_in->path));
         |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

vim +/strncpy +557 fs/cachefiles/io.c

541
542 static struct cachefiles_req *cachefiles_alloc_req(struct cachefiles_object *object,
543 loff_t start_pos,
544 size_t len)
545 {
546 struct cachefiles_req *req;
547 struct cachefiles_req_in *req_in;
548
549 req = kzalloc(sizeof(*req), GFP_KERNEL);
550 if (!req)
551 return NULL;
552
553 req_in = &req->req_in;
554
555 req_in->off = start_pos;
556 req_in->len = len;

 > 557		strncpy(req_in->path, object->d_name, sizeof(req_in->path));

558
559 init_completion(&req->done);
560
561 return req;
562 }
563

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#34 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

In reply to Jeffle Xu

fs/cachefiles/io.c static void cachefiles_end_operation(struct netfs_cache_resources *cres) · line 580

Quoted 2 lines · Jeffle Xu
fs/cachefiles/io.c +2 -0
+	spin_lock(&cache->reqs_lock);
+	ret = idr_alloc(&cache->reqs, req, 0, 0, GFP_KERNEL);

GFP_KERNEL while holding a spinlock?

You should be using an XArray instead of an IDR in new code anyway.

author
#35 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

Quoted 5 lines · Matthew Wilcox

fs/cachefiles/io.c static void cachefiles_end_operation(struct netfs_cache_resources *cres) · line 580

Quoted 2 lines · Jeffle Xu
fs/cachefiles/io.c +2 -0
+	spin_lock(&cache->reqs_lock);
+	ret = idr_alloc(&cache->reqs, req, 0, 0, GFP_KERNEL);

GFP_KERNEL while holding a spinlock?

Right. Thanks for pointing it out.

Quoted 3 lines

You should be using an XArray instead of an IDR in new code anyway.

Regards.

Signature · 2 lines
Thanks,
Jeffle
author
#36 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

Quoted 9 lines · JeffleXu
Quoted 5 lines · Matthew Wilcox

fs/cachefiles/io.c static void cachefiles_end_operation(struct netfs_cache_resources *cres) · line 580

Quoted 2 lines · Jeffle Xu
fs/cachefiles/io.c +2 -0
+	spin_lock(&cache->reqs_lock);
+	ret = idr_alloc(&cache->reqs, req, 0, 0, GFP_KERNEL);

GFP_KERNEL while holding a spinlock?

Quoted 8 lines

Right. Thanks for pointing it out.

Quoted 3 lines

You should be using an XArray instead of an IDR in new code anyway.

Regards.

Hi Matthew,

I'm afraid IDR can't be replaced by xarray here. Because we need an 'ID' for each pending read request, so that after fetching data from remote, user daemon could notify kernel which read request has finished by this 'ID'.

Currently this 'ID' is get from idr_alloc(), and actually identifies the position of corresponding read request inside the IDR tree. I can't find similar API of xarray implementing similar function, i.e., returning an 'ID'.

As for the issue of GFP_KERNEL while holding a spinlock, I'm going to fix this with idr_preload(), somehing like

+ idr_preload(GFP_KERNEL);
+ idr_lock(&cache->reqs);
+
+ ret = idr_alloc(&cache->reqs, req, 0, 0, GFP_ATOMIC);
+ if (ret >= 0)
+ req->req_in.id = ret;
+
+ idr_unlock(&cache->reqs);
+ idr_preload_end();

Please correct me if I'm wrong.

Signature · 2 lines
Thanks,
Jeffle
#37 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

Quoted 9 lines · JeffleXu

I'm afraid IDR can't be replaced by xarray here. Because we need an 'ID' for each pending read request, so that after fetching data from remote, user daemon could notify kernel which read request has finished by this 'ID'.

Currently this 'ID' is get from idr_alloc(), and actually identifies the position of corresponding read request inside the IDR tree. I can't find similar API of xarray implementing similar function, i.e., returning an 'ID'.

xa_alloc().

author
#38 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · JeffleXu

I'm afraid IDR can't be replaced by xarray here. Because we need an 'ID' for each pending read request, so that after fetching data from remote, user daemon could notify kernel which read request has finished by this 'ID'.

Quoted 7 lines
Quoted 4 lines

Currently this 'ID' is get from idr_alloc(), and actually identifies the position of corresponding read request inside the IDR tree. I can't find similar API of xarray implementing similar function, i.e., returning an 'ID'.

xa_alloc().

Oh yes. Thanks. I will try to convert to xarray API...

Signature · 2 lines
Thanks,
Jeffle
#39 lore raw

Re: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read

In reply to Jeffle Xu
Thank you for the patch! Perhaps something to improve:

Hi Jeffle,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on dhowells-fs/fscache-next]
[cannot apply to xiang-erofs/dev-test ceph-client/for-linus linus/master v5.16-rc7 next-20211224] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742
base:   https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git fscache-next
config: i386-randconfig-r005-20211227 (https://download.01.org/0day-ci/archive/20211227/202112272353.ekKESQX6-lkp(a)intel.com/config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 511726c64d3b6cca66f7c54d457d586aa3129f67)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/54c300b4598e3f2836d8233681da387fe388cfda
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jeffle-Xu/fscache-erofs-fscache-based-demand-read-semantics/20211227-205742 git checkout 54c300b4598e3f2836d8233681da387fe388cfda
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 SHELL=/bin/bash fs/cachefiles/

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> fs/cachefiles/io.c:564:5: warning: no previous prototype for function 'cachefiles_demand_read' [-Wmissing-prototypes]
   int cachefiles_demand_read(struct netfs_cache_resources *cres,
       ^

fs/cachefiles/io.c:564:1: note: declare 'static' if the function is not intended to be used outside of this translation unit int cachefiles_demand_read(struct netfs_cache_resources *cres,
^
static
1 warning generated.

vim +/cachefiles_demand_read +564 fs/cachefiles/io.c

563

 > 564	int cachefiles_demand_read(struct netfs_cache_resources *cres,

565 loff_t start_pos, size_t len)
566 {
567 struct cachefiles_object *object;
568 struct cachefiles_cache *cache;
569 struct cachefiles_req *req;
570 int ret;
571
572 object = cachefiles_cres_object(cres);
573 cache = object->volume->cache;
574
575 req = cachefiles_alloc_req(object, start_pos, len);
576 if (!req)
577 return -ENOMEM;
578
579 spin_lock(&cache->reqs_lock);
580 ret = idr_alloc(&cache->reqs, req, 0, 0, GFP_KERNEL);
581 if (ret >= 0)
582 req->req_in.id = ret;
583 spin_unlock(&cache->reqs_lock);
584 if (ret < 0) {
585 kfree(req);
586 return -ENOMEM;
587 }
588
589 wake_up_all(&cache->daemon_pollwq);
590
591 wait_for_completion(&req->done);
592 kfree(req);
593
594 return 0;
595 }
596

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#40 lore raw

PATCH 20/23 cachefiles: implement .poll() for demand read

In reply to Jeffle Xu

User daemon needs to poll on the devnode, and will be notified once there's pending request to process.

---
 fs/cachefiles/daemon.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
fs/cachefiles/daemon.c +18 -0
index 01496fa8c263..311dcd911a85 100644
@@ -29,6 +29,8 @@ static ssize_t cachefiles_daemon_write(struct file *, const char __user *,
 				       size_t, loff_t *);
 static __poll_t cachefiles_daemon_poll(struct file *,
 					   struct poll_table_struct *);
+static __poll_t cachefiles_demand_poll(struct file *,
+					   struct poll_table_struct *);
 static int cachefiles_daemon_frun(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_fcull(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_fstop(struct cachefiles_cache *, char *);
@@ -62,6 +64,7 @@ const struct file_operations cachefiles_demand_fops = {
 	.open		= cachefiles_daemon_open,
 	.release	= cachefiles_daemon_release,
 	.write		= cachefiles_daemon_write,
+	.poll		= cachefiles_demand_poll,
 	.llseek		= noop_llseek,
 };
 
@@ -319,6 +322,21 @@ static __poll_t cachefiles_daemon_poll(struct file *file,
 	return mask;
 }
 
+static __poll_t cachefiles_demand_poll(struct file *file,
+					   struct poll_table_struct *poll)
+{
+	struct cachefiles_cache *cache = file->private_data;
+	__poll_t mask;
+
+	poll_wait(file, &cache->daemon_pollwq, poll);
+	mask = 0;
+
+	if (!idr_is_empty(&cache->reqs))
+		mask |= EPOLLIN;
+
+	return mask;
+}
+
 /*
  * Give a range error for cache space constraints
  * - can be tail-called
Signature · 1 line
2.27.0
author
#41 lore raw

PATCH 21/23 cachefiles: implement .read() for demand read

In reply to Jeffle Xu

Once notified, user daemon need to get more details of the request by reading the devnode. The readed information includes the file range of the request, with which user daemon could somehow prepare corresponding data (e.g. download from remote through network) and fill the hole.

---
 fs/cachefiles/daemon.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
fs/cachefiles/daemon.c +29 -0
index 311dcd911a85..ce4cc5617dfc 100644
@@ -29,6 +29,8 @@ static ssize_t cachefiles_daemon_write(struct file *, const char __user *,
 				       size_t, loff_t *);
 static __poll_t cachefiles_daemon_poll(struct file *,
 					   struct poll_table_struct *);
+static ssize_t cachefiles_demand_read(struct file *, char __user *, size_t,
+				      loff_t *);
 static __poll_t cachefiles_demand_poll(struct file *,
 					   struct poll_table_struct *);
 static int cachefiles_daemon_frun(struct cachefiles_cache *, char *);
@@ -63,6 +65,7 @@ const struct file_operations cachefiles_demand_fops = {
 	.owner		= THIS_MODULE,
 	.open		= cachefiles_daemon_open,
 	.release	= cachefiles_daemon_release,
+	.read		= cachefiles_demand_read,
 	.write		= cachefiles_daemon_write,
 	.poll		= cachefiles_demand_poll,
 	.llseek		= noop_llseek,
@@ -322,6 +325,32 @@ static __poll_t cachefiles_daemon_poll(struct file *file,
 	return mask;
 }
 
+static ssize_t cachefiles_demand_read(struct file *file, char __user *_buffer,
+				      size_t buflen, loff_t *pos)
+{
+	struct cachefiles_cache *cache = file->private_data;
+	struct cachefiles_req *req;
+	int n, id = 0;
+
+	if (!test_bit(CACHEFILES_READY, &cache->flags))
+		return 0;
+
+	spin_lock(&cache->reqs_lock);
+	req = idr_get_next(&cache->reqs, &id);
+	spin_unlock(&cache->reqs_lock);
+	if (!req)
+		return 0;
+
+	n = sizeof(req->req_in);
+	if (n > buflen)
+		return -EMSGSIZE;
+
+	if (copy_to_user(_buffer, &req->req_in, n) != 0)
+		return -EFAULT;
+
+	return n;
+}
+
 static __poll_t cachefiles_demand_poll(struct file *file,
 					   struct poll_table_struct *poll)
 {
Signature · 1 line
2.27.0
author
#42 lore raw

PATCH 22/23 cachefiles: add done command for demand read

In reply to Jeffle Xu

Once has prepared data and filled the hole, user daemon could notify cachefiles backend by writing 'done' command to devnode.

With done command, the id originally readed from devnode is transferred back to cachefiles backend. This id identifies the position of the completed request inside the idr tree, and thus cachefiles backend could find the corresponding completed request once gets notified.

---
 fs/cachefiles/daemon.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
fs/cachefiles/daemon.c +34 -0
index ce4cc5617dfc..36d0bcf5fc3f 100644
@@ -48,6 +48,7 @@ static int cachefiles_daemon_tag(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_mode(struct cachefiles_cache *, char *);
 static int cachefiles_daemon_bind(struct cachefiles_cache *, char *);
 static void cachefiles_daemon_unbind(struct cachefiles_cache *);
+static int cachefiles_daemon_done(struct cachefiles_cache *, char *);
 
 static unsigned long cachefiles_open;
 
@@ -91,6 +92,7 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = {
 	{ "secctx",	cachefiles_daemon_secctx	},
 	{ "tag",	cachefiles_daemon_tag		},
 	{ "mode",	cachefiles_daemon_mode		},
+	{ "done",	cachefiles_daemon_done		},
 	{ "",		NULL				}
 };
 
@@ -604,6 +606,38 @@ static int cachefiles_daemon_tag(struct cachefiles_cache *cache, char *args)
 	return 0;
 }
 
+/*
+ * Request completion
+ * - command: "done <id>"
+ */
+static int cachefiles_daemon_done(struct cachefiles_cache *cache, char *args)
+{
+	unsigned long id;
+	int ret;
+	struct cachefiles_req *req;
+
+	_enter(",%s", args);
+
+	if (!*args) {
+		pr_err("Empty id specified\n");
+		return -EINVAL;
+	}
+
+	ret = kstrtoul(args, 0, &id);
+	if (ret)
+		return ret;
+
+	spin_lock(&cache->reqs_lock);
+	req = idr_remove(&cache->reqs, id);
+	spin_unlock(&cache->reqs_lock);
+	if (!req)
+		return -EINVAL;
+
+	complete(&req->done);
+
+	return 0;
+}
+
 /*
  * Request a node in the cache be culled from the current working directory
  * - command: "cull <name>"
Signature · 1 line
2.27.0
author
#43 lore raw

PATCH 23/23 erofs: support on demand read

In reply to Jeffle Xu

Implement .issue_op() callback to support demand reading.

---
 fs/erofs/fscache.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
fs/erofs/fscache.c +1 -7
index 7d4f6682e521..bd64dbe1b0be 100644
@@ -18,13 +18,7 @@ static void erofs_priv_cleanup(struct address_space *mapping, void *netfs_priv)
 
 static void erofs_issue_op(struct netfs_read_subrequest *subreq)
 {
-	/*
-	 * TODO: implement demand-read logic later.
-	 * We rely on user daemon to prepare blob files under corresponding
-	 * directory, and we can reach here if blob files don't exist.
-	 */
-
-	netfs_subreq_terminated(subreq, -EOPNOTSUPP, false);
+	netfs_demand_read(subreq);
 }
 
 const struct netfs_read_request_ops erofs_req_ops = {
Signature · 1 line
2.27.0