Skip to contentSkip to conversation

Convert NFS with fscache to the netfs API

PATCH Superseded Dave Wysochanski 6 participants 31 messages Oct 17, 2022 – Feb 9, 2023 1 message not in the archive

Subsystems (3)

v9 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v9 cover Convert NFS with fscache to the netfs API

This patchset converts NFS with fscache non-direct READ IO paths to use the netfs API with a non-invasive approach. The existing NFS pgio layer does not need extensive changes, and is the best way so far I've found to address Trond's concerns about modifying the IO path [1] as well as only enabling netfs when fscache is configured and enabled [2]. I have not attempted performance comparisions to address Chuck Lever's concern [3] because we are not converting the non-fscache enabled NFS IO paths to netfs.

No new issues found at BakeAthon testing NFSv4.2 with Hammerspace (pNFS flexfiles) and NetApp (pNFS filelayout).

The patchset is based on 6.0 and has been pushed to github at:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/c222ed8cd042d003ca22d890af98d9c76ec8650e

Changes since v8 [7]
====================
PATCH3-PATCH5: Add Jeff Layton Revewied-by

Testing
=======
The patches are fairly stable as evidenced with xfstests generic with various servers, both with and without fscache enabled, with no hangs or crashes seen:
- hammerspace(pNFS flexfiles): NFS4.1, NFS4.2
- NetApp(pNFS filelayout): NFS3, NFS4.0, NFS4.1, NFS4.2
- RHEL8: NFS3,NFS4.0,NFS4.2

Known issues
============
1. Unit test setting rsize < readahead does not properly read from fscache but re-reads data from the NFS server
* This will be fixed with another linux-cachefs [4] patch to resolve "Stop read optimisation when folio removed from pagecache"
* Daire Byrne also verified the patch fixes his issue as well

2. "Cache volume key already in use" after xfstest runs
* xfstests (hammerspace with vers=4.2,fsc) shows the following on the
console after some tests:
"NFS: Cache volume key already in use (nfs,4.1,2,c50,cfe0100a,3,,,8000,100000,100000,bb8,ea60,7530,ea60,1)"
* This may be fixed with another patch [5] that is in progress

3. Daire Byrne reported a NULL pointer oops at cachefiles_prepare_write+0x28/0x90
* harder to reproduce/debug but under investigation [6]
* only reproduced on RHEL7.9 based NFS re-export server using fscache with upstream kernel plus the previous patches
* thought to be a bug in netfs or cachefiles exposed by NFS patches

[58710.346376] BUG: kernel NULL pointer dereference, address: 0000000000000008
[58710.371212] CPU: 12 PID: 9134 Comm: kworker/u129:0 Tainted: G E      6.0.0-2.dneg.x86_64 #1
...
[58710.389995] Workqueue: events_unbound netfs_rreq_write_to_cache_work [netfs]
[58710.397188] RIP: 0010:cachefiles_prepare_write+0x28/0x90 [cachefiles]
...
[58710.500316] Call Trace:
[58710.502894]  <TASK>
[58710.505126]  netfs_rreq_write_to_cache_work+0x11c/0x320 [netfs]
[58710.511201]  process_one_work+0x217/0x3e0
[58710.515358]  worker_thread+0x4a/0x3b0
[58710.519152]  ? process_one_work+0x3e0/0x3e0
[58710.523467]  kthread+0xd6/0x100
[58710.526740]  ? kthread_complete_and_exit+0x20/0x20
[58710.531659]  ret_from_fork+0x1f/0x30

References
==========
[1] https://lore.kernel.org/linux-nfs/[email protected]/
[2] https://lore.kernel.org/linux-nfs/[email protected]/
[3] https://marc.info/?l=linux-nfs&m=160597917525083&w=4
[4] https://www.mail-archive.com/[email protected]/msg03043.html
[5] https://marc.info/?l=linux-nfs&m=165962662200679&w=4
[6] https://listman.redhat.com/archives/linux-cachefs/2022-September/007183.html
[7] https://marc.info/?l=linux-nfs&m=166385481714542&w=4

Dave Wysochanski (5):
NFS: Rename readpage_async_filler to nfs_pageio_add_page
NFS: Configure support for netfs when NFS fscache is configured
NFS: Convert buffered read paths to use netfs when fscache is enabled
NFS: Remove all NFSIOS_FSCACHE counters due to conversion to netfs API
NFS: Remove fscache specific trace points and NFS_INO_FSCACHE bit

 fs/nfs/Kconfig             |   1 +
 fs/nfs/delegation.c        |   2 +-
 fs/nfs/dir.c               |   2 +-
 fs/nfs/fscache.c           | 242 ++++++++++++++++++++++---------------
 fs/nfs/fscache.h           | 111 +++++++++++------
 fs/nfs/inode.c             |   8 +-
 fs/nfs/internal.h          |  11 +-
 fs/nfs/iostat.h            |  17 ---
 fs/nfs/nfstrace.h          |  91 --------------
 fs/nfs/pagelist.c          |  12 ++
 fs/nfs/pnfs.c              |  12 +-
 fs/nfs/read.c              | 110 +++++++++--------
 fs/nfs/super.c             |  11 --
 fs/nfs/write.c             |   2 +-
 include/linux/nfs_fs.h     |  35 ++++--
 include/linux/nfs_iostat.h |  12 --
 include/linux/nfs_page.h   |   3 +
 include/linux/nfs_xdr.h    |   3 +
 18 files changed, 335 insertions(+), 350 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v9 1/5 NFS: Rename readpage_async_filler to nfs_pageio_add_page

Rename readpage_async_filler to nfs_pageio_add_page to better reflect what this function does (add a page to the nfs_pageio_descriptor), and simplify arguments to this function by removing struct nfs_readdesc.

---
 fs/nfs/read.c | 60 +++++++++++++++++++++++++--------------------------
 1 file changed, 30 insertions(+), 30 deletions(-)
fs/nfs/read.c +30 -30
index 8ae2c8d1219d..525e82ea9a9e 100644
@@ -127,11 +127,6 @@ static void nfs_readpage_release(struct nfs_page *req, int error)
 	nfs_release_request(req);
 }
 
-struct nfs_readdesc {
-	struct nfs_pageio_descriptor pgio;
-	struct nfs_open_context *ctx;
-};
-
 static void nfs_page_group_set_uptodate(struct nfs_page *req)
 {
 	if (nfs_page_group_sync_on_bit(req, PG_UPTODATE))
@@ -153,7 +148,8 @@ static void nfs_read_completion(struct nfs_pgio_header *hdr)
 
 		if (test_bit(NFS_IOHDR_EOF, &hdr->flags)) {
 			/* note: regions of the page not covered by a
-			 * request are zeroed in readpage_async_filler */
+			 * request are zeroed in nfs_pageio_add_page
+			 */
 			if (bytes > hdr->good_bytes) {
 				/* nothing in this request was good, so zero
 				 * the full extent of the request */
@@ -281,8 +277,10 @@ static void nfs_readpage_result(struct rpc_task *task,
 		nfs_readpage_retry(task, hdr);
 }
 
-static int
-readpage_async_filler(struct nfs_readdesc *desc, struct page *page)
+int
+nfs_pageio_add_page(struct nfs_pageio_descriptor *pgio,
+		    struct nfs_open_context *ctx,
+		    struct page *page)
 {
 	struct inode *inode = page_file_mapping(page)->host;
 	unsigned int rsize = NFS_SERVER(inode)->rsize;
@@ -302,15 +300,15 @@ readpage_async_filler(struct nfs_readdesc *desc, struct page *page)
 			goto out_unlock;
 	}
 
-	new = nfs_create_request(desc->ctx, page, 0, aligned_len);
+	new = nfs_create_request(ctx, page, 0, aligned_len);
 	if (IS_ERR(new))
 		goto out_error;
 
 	if (len < PAGE_SIZE)
 		zero_user_segment(page, len, PAGE_SIZE);
-	if (!nfs_pageio_add_request(&desc->pgio, new)) {
+	if (!nfs_pageio_add_request(pgio, new)) {
 		nfs_list_remove_request(new);
-		error = desc->pgio.pg_error;
+		error = pgio->pg_error;
 		nfs_readpage_release(new, error);
 		goto out;
 	}
@@ -332,7 +330,8 @@ readpage_async_filler(struct nfs_readdesc *desc, struct page *page)
 int nfs_read_folio(struct file *file, struct folio *folio)
 {
 	struct page *page = &folio->page;
-	struct nfs_readdesc desc;
+	struct nfs_pageio_descriptor pgio;
+	struct nfs_open_context *ctx;
 	struct inode *inode = page_file_mapping(page)->host;
 	int ret;
 
@@ -358,29 +357,29 @@ int nfs_read_folio(struct file *file, struct folio *folio)
 
 	if (file == NULL) {
 		ret = -EBADF;
-		desc.ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
-		if (desc.ctx == NULL)
+		ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
+		if (ctx == NULL)
 			goto out_unlock;
 	} else
-		desc.ctx = get_nfs_open_context(nfs_file_open_context(file));
+		ctx = get_nfs_open_context(nfs_file_open_context(file));
 
-	xchg(&desc.ctx->error, 0);
-	nfs_pageio_init_read(&desc.pgio, inode, false,
+	xchg(&ctx->error, 0);
+	nfs_pageio_init_read(&pgio, inode, false,
 			     &nfs_async_read_completion_ops);
 
-	ret = readpage_async_filler(&desc, page);
+	ret = nfs_pageio_add_page(&pgio, ctx, page);
 	if (ret)
 		goto out;
 
-	nfs_pageio_complete_read(&desc.pgio);
-	ret = desc.pgio.pg_error < 0 ? desc.pgio.pg_error : 0;
+	nfs_pageio_complete_read(&pgio);
+	ret = pgio.pg_error < 0 ? pgio.pg_error : 0;
 	if (!ret) {
 		ret = wait_on_page_locked_killable(page);
 		if (!PageUptodate(page) && !ret)
-			ret = xchg(&desc.ctx->error, 0);
+			ret = xchg(&ctx->error, 0);
 	}
 out:
-	put_nfs_open_context(desc.ctx);
+	put_nfs_open_context(ctx);
 	trace_nfs_aop_readpage_done(inode, page, ret);
 	return ret;
 out_unlock:
@@ -391,9 +390,10 @@ int nfs_read_folio(struct file *file, struct folio *folio)
 
 void nfs_readahead(struct readahead_control *ractl)
 {
+	struct nfs_pageio_descriptor pgio;
+	struct nfs_open_context *ctx;
 	unsigned int nr_pages = readahead_count(ractl);
 	struct file *file = ractl->file;
-	struct nfs_readdesc desc;
 	struct inode *inode = ractl->mapping->host;
 	struct page *page;
 	int ret;
@@ -407,25 +407,25 @@ void nfs_readahead(struct readahead_control *ractl)
 
 	if (file == NULL) {
 		ret = -EBADF;
-		desc.ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
-		if (desc.ctx == NULL)
+		ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
+		if (ctx == NULL)
 			goto out;
 	} else
-		desc.ctx = get_nfs_open_context(nfs_file_open_context(file));
+		ctx = get_nfs_open_context(nfs_file_open_context(file));
 
-	nfs_pageio_init_read(&desc.pgio, inode, false,
+	nfs_pageio_init_read(&pgio, inode, false,
 			     &nfs_async_read_completion_ops);
 
 	while ((page = readahead_page(ractl)) != NULL) {
-		ret = readpage_async_filler(&desc, page);
+		ret = nfs_pageio_add_page(&pgio, ctx, page);
 		put_page(page);
 		if (ret)
 			break;
 	}
 
-	nfs_pageio_complete_read(&desc.pgio);
+	nfs_pageio_complete_read(&pgio);
 
-	put_nfs_open_context(desc.ctx);
+	put_nfs_open_context(ctx);
 out:
 	trace_nfs_aop_readahead_done(inode, nr_pages, ret);
 }
Signature · 1 line
2.31.1
maintainer
#2 lore raw

Re: [PATCH v9 1/5] NFS: Rename readpage_async_filler to nfs_pageio_add_page

Quoted 47 lines · Dave Wysochanski

Rename readpage_async_filler to nfs_pageio_add_page to better reflect what this function does (add a page to the nfs_pageio_descriptor), and simplify arguments to this function by removing struct nfs_readdesc.

---
 fs/nfs/read.c | 60 +++++++++++++++++++++++++------------------------ --

 1 file changed, 30 insertions(+), 30 deletions(-)
fs/nfs/read.c +0 -0
index 8ae2c8d1219d..525e82ea9a9e 100644
@@ -127,11 +127,6 @@ static void nfs_readpage_release(struct nfs_page
 *req, int error)
         nfs_release_request(req);
  }

fs/nfs/read.c +0 -5
-struct nfs_readdesc {
-       struct nfs_pageio_descriptor pgio;
-       struct nfs_open_context *ctx;
-};
-

 static void nfs_page_group_set_uptodate(struct nfs_page *req) {
        if (nfs_page_group_sync_on_bit(req, PG_UPTODATE))

fs/nfs/read.c +0 -0
@@ -153,7 +148,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)


                if (test_bit(NFS_IOHDR_EOF, &hdr->flags)) { /* note: regions of the page not covered by a -                        * request are zeroed in readpage_async_filler */
+                        * request are zeroed in nfs_pageio_add_page +                        */
                        if (bytes > hdr->good_bytes) {
                                /* nothing in this request was good, so zero
                                 * the full extent of the request */

fs/nfs/read.c +0 -0
@@ -281,8 +277,10 @@ static void nfs_readpage_result(struct rpc_task
 *task,
                 nfs_readpage_retry(task, hdr);
  }

fs/nfs/read.c static void nfs_readpage_result(struct rpc_task *task, · line 283

Quoted 6 lines
fs/nfs/read.c +4 -2
-static int
-readpage_async_filler(struct nfs_readdesc *desc, struct page *page)
+int
+nfs_pageio_add_page(struct nfs_pageio_descriptor *pgio,
+                   struct nfs_open_context *ctx,
+                   struct page *page)

If we're going to rename this function, then let's not give it a name that suggests it belongs in pagelist.c. It's not a generic helper function, but is still very much specific to the pagecache read functionality.

Quoted 128 lines

 {
        struct inode *inode = page_file_mapping(page)->host; unsigned int rsize = NFS_SERVER(inode)->rsize;

fs/nfs/read.c +0 -0
@@ -302,15 +300,15 @@ readpage_async_filler(struct nfs_readdesc
 *desc, struct page *page)
                         goto out_unlock;
         }


-       new = nfs_create_request(desc->ctx, page, 0, aligned_len); +       new = nfs_create_request(ctx, page, 0, aligned_len); if (IS_ERR(new))
                goto out_error;

        if (len < PAGE_SIZE)
                zero_user_segment(page, len, PAGE_SIZE); -       if (!nfs_pageio_add_request(&desc->pgio, new)) { +       if (!nfs_pageio_add_request(pgio, new)) { nfs_list_remove_request(new);
-               error = desc->pgio.pg_error;
+               error = pgio->pg_error; nfs_readpage_release(new, error);
                goto out;
        }

fs/nfs/read.c +2 -1
@@ -332,7 +330,8 @@ readpage_async_filler(struct nfs_readdesc *desc,
 struct page *page)
  int nfs_read_folio(struct file *file, struct folio *folio)
  {
         struct page *page = &folio->page;
-       struct nfs_readdesc desc;
+       struct nfs_pageio_descriptor pgio;
+       struct nfs_open_context *ctx;
         struct inode *inode = page_file_mapping(page)->host;
         int ret;

fs/nfs/read.c +0 -0
@@ -358,29 +357,29 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)


        if (file == NULL) {
                ret = -EBADF;
-               desc.ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
-               if (desc.ctx == NULL)
+               ctx = nfs_find_open_context(inode, NULL, FMODE_READ); +               if (ctx == NULL)
                        goto out_unlock;
        } else
-               desc.ctx = get_nfs_open_context(nfs_file_open_context(file));
+               ctx = get_nfs_open_context(nfs_file_open_context(file));

-       xchg(&desc.ctx->error, 0);
-       nfs_pageio_init_read(&desc.pgio, inode, false,
+       xchg(&ctx->error, 0);
+       nfs_pageio_init_read(&pgio, inode, false, &nfs_async_read_completion_ops);
-       ret = readpage_async_filler(&desc, page);
+       ret = nfs_pageio_add_page(&pgio, ctx, page);
        if (ret)
                goto out;

-       nfs_pageio_complete_read(&desc.pgio);
-       ret = desc.pgio.pg_error < 0 ? desc.pgio.pg_error : 0; +       nfs_pageio_complete_read(&pgio);
+       ret = pgio.pg_error < 0 ? pgio.pg_error : 0;
        if (!ret) {
                ret = wait_on_page_locked_killable(page); if (!PageUptodate(page) && !ret)
-                       ret = xchg(&desc.ctx->error, 0);
+                       ret = xchg(&ctx->error, 0);
        }
 out:
-       put_nfs_open_context(desc.ctx);
+       put_nfs_open_context(ctx); trace_nfs_aop_readpage_done(inode, page, ret); return ret;
 out_unlock:

fs/nfs/read.c +0 -0
@@ -391,9 +390,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)


 void nfs_readahead(struct readahead_control *ractl) {
+       struct nfs_pageio_descriptor pgio;
+       struct nfs_open_context *ctx;
        unsigned int nr_pages = readahead_count(ractl); struct file *file = ractl->file;
-       struct nfs_readdesc desc;
        struct inode *inode = ractl->mapping->host; struct page *page;
        int ret;

fs/nfs/read.c +0 -0
@@ -407,25 +407,25 @@ void nfs_readahead(struct readahead_control
 *ractl)


        if (file == NULL) {
                ret = -EBADF;
-               desc.ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
-               if (desc.ctx == NULL)
+               ctx = nfs_find_open_context(inode, NULL, FMODE_READ); +               if (ctx == NULL)
                        goto out;
        } else
-               desc.ctx = get_nfs_open_context(nfs_file_open_context(file));
+               ctx = get_nfs_open_context(nfs_file_open_context(file));

-       nfs_pageio_init_read(&desc.pgio, inode, false,
+       nfs_pageio_init_read(&pgio, inode, false, &nfs_async_read_completion_ops);
        while ((page = readahead_page(ractl)) != NULL) {
-               ret = readpage_async_filler(&desc, page); +               ret = nfs_pageio_add_page(&pgio, ctx, page); put_page(page);
                if (ret)
                        break;
        }

-       nfs_pageio_complete_read(&desc.pgio);
+       nfs_pageio_complete_read(&pgio);

-       put_nfs_open_context(desc.ctx);
+       put_nfs_open_context(ctx);
 out:
        trace_nfs_aop_readahead_done(inode, nr_pages, ret); }

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#3 lore raw

Re: [PATCH v9 1/5] NFS: Rename readpage_async_filler to nfs_pageio_add_page

Quoted 49 lines · Trond Myklebust
Quoted 47 lines · Dave Wysochanski

Rename readpage_async_filler to nfs_pageio_add_page to better reflect what this function does (add a page to the nfs_pageio_descriptor), and simplify arguments to this function by removing struct nfs_readdesc.

---
 fs/nfs/read.c | 60 +++++++++++++++++++++++++------------------------
--
 1 file changed, 30 insertions(+), 30 deletions(-)
fs/nfs/read.c +0 -5
index 8ae2c8d1219d..525e82ea9a9e 100644
@@ -127,11 +127,6 @@ static void nfs_readpage_release(struct nfs_page
 *req, int error)
        nfs_release_request(req);
 }
 
-struct nfs_readdesc {
-       struct nfs_pageio_descriptor pgio;
-       struct nfs_open_context *ctx;
-};
-
 static void nfs_page_group_set_uptodate(struct nfs_page *req)
 {

if (nfs_page_group_sync_on_bit(req, PG_UPTODATE))

fs/nfs/read.c +0 -1
@@ -153,7 +148,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
 
                if (test_bit(NFS_IOHDR_EOF, &hdr->flags)) {
                        /* note: regions of the page not covered by a
-                        * request are zeroed in

readpage_async_filler */

fs/nfs/read.c +2 -0
+                        * request are zeroed in nfs_pageio_add_page
+                        */
                        if (bytes > hdr->good_bytes) {
                                /* nothing in this request was good,

so zero
* the full extent of the request */

fs/nfs/read.c +0 -0
@@ -281,8 +277,10 @@ static void nfs_readpage_result(struct rpc_task
 *task,
                nfs_readpage_retry(task, hdr);
 }
 
Quoted 12 lines

fs/nfs/read.c static void nfs_readpage_result(struct rpc_task *task, · line 283

Quoted 6 lines
fs/nfs/read.c +4 -2
-static int
-readpage_async_filler(struct nfs_readdesc *desc, struct page *page)
+int
+nfs_pageio_add_page(struct nfs_pageio_descriptor *pgio,
+                   struct nfs_open_context *ctx,
+                   struct page *page)

If we're going to rename this function, then let's not give it a name that suggests it belongs in pagelist.c. It's not a generic helper function, but is still very much specific to the pagecache read functionality.

How about nfs_read_add_page()?

Quoted 135 lines
Quoted 118 lines
fs/nfs/read.c +0 -0
 {
        struct inode *inode = page_file_mapping(page)->host;
        unsigned int rsize = NFS_SERVER(inode)->rsize;
fs/nfs/read.c +3 -3
@@ -302,15 +300,15 @@ readpage_async_filler(struct nfs_readdesc
 *desc, struct page *page)
                        goto out_unlock;
        }
 
-       new = nfs_create_request(desc->ctx, page, 0, aligned_len);
+       new = nfs_create_request(ctx, page, 0, aligned_len);
        if (IS_ERR(new))
                goto out_error;
 
        if (len < PAGE_SIZE)
                zero_user_segment(page, len, PAGE_SIZE);
-       if (!nfs_pageio_add_request(&desc->pgio, new)) {
+       if (!nfs_pageio_add_request(pgio, new)) {
                nfs_list_remove_request(new);
-               error = desc->pgio.pg_error;
+               error = pgio->pg_error;
                nfs_readpage_release(new, error);
                goto out;
        }
fs/nfs/read.c +14 -12
@@ -332,7 +330,8 @@ readpage_async_filler(struct nfs_readdesc *desc,
 struct page *page)
 int nfs_read_folio(struct file *file, struct folio *folio)
 {
        struct page *page = &folio->page;
-       struct nfs_readdesc desc;
+       struct nfs_pageio_descriptor pgio;
+       struct nfs_open_context *ctx;
        struct inode *inode = page_file_mapping(page)->host;
        int ret;
@@ -358,29 +357,29 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
 
        if (file == NULL) {
                ret = -EBADF;
-               desc.ctx = nfs_find_open_context(inode, NULL,
 FMODE_READ);
-               if (desc.ctx == NULL)
+               ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
+               if (ctx == NULL)
                        goto out_unlock;
        } else
-               desc.ctx =
 get_nfs_open_context(nfs_file_open_context(file));
+               ctx =
 get_nfs_open_context(nfs_file_open_context(file));
 
-       xchg(&desc.ctx->error, 0);
-       nfs_pageio_init_read(&desc.pgio, inode, false,
+       xchg(&ctx->error, 0);
+       nfs_pageio_init_read(&pgio, inode, false,
                             &nfs_async_read_completion_ops);
 
-       ret = readpage_async_filler(&desc, page);
+       ret = nfs_pageio_add_page(&pgio, ctx, page);
        if (ret)
                goto out;
 
-       nfs_pageio_complete_read(&desc.pgio);
-       ret = desc.pgio.pg_error < 0 ? desc.pgio.pg_error : 0;
+       nfs_pageio_complete_read(&pgio);
+       ret = pgio.pg_error < 0 ? pgio.pg_error : 0;
        if (!ret) {
                ret = wait_on_page_locked_killable(page);
                if (!PageUptodate(page) && !ret)
-                       ret = xchg(&desc.ctx->error, 0);
+                       ret = xchg(&ctx->error, 0);
        }
 out:
-       put_nfs_open_context(desc.ctx);
+       put_nfs_open_context(ctx);
        trace_nfs_aop_readpage_done(inode, page, ret);
        return ret;
 out_unlock:
@@ -391,9 +390,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
 
 void nfs_readahead(struct readahead_control *ractl)
 {
+       struct nfs_pageio_descriptor pgio;
+       struct nfs_open_context *ctx;
        unsigned int nr_pages = readahead_count(ractl);
        struct file *file = ractl->file;
-       struct nfs_readdesc desc;
        struct inode *inode = ractl->mapping->host;
        struct page *page;
        int ret;
fs/nfs/read.c +5 -5
@@ -407,25 +407,25 @@ void nfs_readahead(struct readahead_control
 *ractl)
 
        if (file == NULL) {
                ret = -EBADF;
-               desc.ctx = nfs_find_open_context(inode, NULL,
 FMODE_READ);
-               if (desc.ctx == NULL)
+               ctx = nfs_find_open_context(inode, NULL, FMODE_READ);
+               if (ctx == NULL)
                        goto out;
        } else
-               desc.ctx =
 get_nfs_open_context(nfs_file_open_context(file));
+               ctx =
 get_nfs_open_context(nfs_file_open_context(file));
 
-       nfs_pageio_init_read(&desc.pgio, inode, false,
+       nfs_pageio_init_read(&pgio, inode, false,
                             &nfs_async_read_completion_ops);
 
        while ((page = readahead_page(ractl)) != NULL) {
-               ret = readpage_async_filler(&desc, page);
+               ret = nfs_pageio_add_page(&pgio, ctx, page);
                put_page(page);
                if (ret)
                        break;

fs/nfs/read.c void nfs_readahead(struct readahead_control · line 431

Quoted 10 lines
fs/nfs/read.c +2 -2
        }
 
-       nfs_pageio_complete_read(&desc.pgio);
+       nfs_pageio_complete_read(&pgio);
 
-       put_nfs_open_context(desc.ctx);
+       put_nfs_open_context(ctx);
 out:
        trace_nfs_aop_readahead_done(inode, nr_pages, ret);
 }

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

maintainer
#4 lore raw

Re: [PATCH v9 1/5] NFS: Rename readpage_async_filler to nfs_pageio_add_page

Quoted 73 lines · David Wysochanski
Quoted 56 lines · Trond Myklebust
Quoted 54 lines · Dave Wysochanski

Rename readpage_async_filler to nfs_pageio_add_page to better reflect what this function does (add a page to the nfs_pageio_descriptor), and simplify arguments to this function by removing struct nfs_readdesc.

---
 fs/nfs/read.c | 60 +++++++++++++++++++++++++--------------------
----
--

 1 file changed, 30 insertions(+), 30 deletions(-)
fs/nfs/read.c +0 -0
index 8ae2c8d1219d..525e82ea9a9e 100644
@@ -127,11 +127,6 @@ static void nfs_readpage_release(struct

nfs_page
*req, int error)
        nfs_release_request(req);
 }

fs/nfs/read.c +0 -5
-struct nfs_readdesc {
-       struct nfs_pageio_descriptor pgio;
-       struct nfs_open_context *ctx;
-};
-

 static void nfs_page_group_set_uptodate(struct nfs_page *req) {
        if (nfs_page_group_sync_on_bit(req, PG_UPTODATE))

fs/nfs/read.c +0 -0
@@ -153,7 +148,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
 
                 if (test_bit(NFS_IOHDR_EOF, &hdr->flags)) {
                         /* note: regions of the page not covered

by a
-                        * request are zeroed in readpage_async_filler */
+                        * request are zeroed in nfs_pageio_add_page
+                        */
                        if (bytes > hdr->good_bytes) { /* nothing in this request was good,
so zero
                                 * the full extent of the request */

fs/nfs/read.c +0 -0
@@ -281,8 +277,10 @@ static void nfs_readpage_result(struct

rpc_task
*task,
                nfs_readpage_retry(task, hdr); }

Quoted 14 lines

fs/nfs/read.c static void nfs_readpage_result(struct rpc_task *task, · line 283

Quoted 7 lines
fs/nfs/read.c +0 -2
-static int
-readpage_async_filler(struct nfs_readdesc *desc, struct page

*page)

fs/nfs/read.c +4 -0
+int
+nfs_pageio_add_page(struct nfs_pageio_descriptor *pgio,
+                   struct nfs_open_context *ctx,
+                   struct page *page)

If we're going to rename this function, then let's not give it a name
that suggests it belongs in pagelist.c. It's not a generic helper function, but is still very much specific to the pagecache read functionality.

Quoted 3 lines

How about nfs_read_add_page()?

That's better.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#5 lore raw

PATCH v9 2/5 NFS: Configure support for netfs when NFS fscache is configured

In reply to Dave Wysochanski

As first steps for support of the netfs library when NFS_FSCACHE is configured, add NETFS_SUPPORT to Kconfig and add the required netfs_inode into struct nfs_inode.

Using netfs requires we move the VFS inode structure to be stored inside struct netfs_inode, along with the fscache_cookie. Thus, create a new helper, VFS_I(), which is defined differently depending on whether NFS_FSCACHE is configured. In addition, use the netfs_inode() and netfs_i_cookie() helpers, and remove our own helper, nfs_i_fscache().

Later patches will convert NFS fscache to fully use netfs.

---
 fs/nfs/Kconfig         |  1 +
 fs/nfs/delegation.c    |  2 +-
 fs/nfs/dir.c           |  2 +-
 fs/nfs/fscache.c       | 20 +++++++++-----------
 fs/nfs/fscache.h       | 15 ++++++---------
 fs/nfs/inode.c         |  6 +++---
 fs/nfs/internal.h      |  2 +-
 fs/nfs/pnfs.c          | 12 ++++++------
 fs/nfs/write.c         |  2 +-
 include/linux/nfs_fs.h | 34 +++++++++++++++++++++++-----------
 10 files changed, 52 insertions(+), 44 deletions(-)
fs/nfs/Kconfig +1 -0
index 14a72224b657..8fbb6caf3481 100644
@@ -171,6 +171,7 @@ config ROOT_NFS
 config NFS_FSCACHE
 	bool "Provide NFS client caching support"
 	depends on NFS_FS=m && FSCACHE || NFS_FS=y && FSCACHE=y
+	select NETFS_SUPPORT
 	help
 	  Say Y here if you want NFS data to be cached locally on disc through
 	  the general filesystem cache manager
fs/nfs/delegation.c +1 -1
index 5c97cad741a7..b5c492d40367 100644
@@ -306,7 +306,7 @@ nfs_start_delegation_return_locked(struct nfs_inode *nfsi)
 	}
 	spin_unlock(&delegation->lock);
 	if (ret)
-		nfs_clear_verifier_delegated(&nfsi->vfs_inode);
+		nfs_clear_verifier_delegated(VFS_I(nfsi));
 out:
 	return ret;
 }
fs/nfs/dir.c +1 -1
index 5d6c2ddc7ea6..b63c624cea6d 100644
@@ -2799,7 +2799,7 @@ nfs_do_access_cache_scan(unsigned int nr_to_scan)
 
 		if (nr_to_scan-- == 0)
 			break;
-		inode = &nfsi->vfs_inode;
+		inode = VFS_I(nfsi);
 		spin_lock(&inode->i_lock);
 		if (list_empty(&nfsi->access_cache_entry_lru))
 			goto remove_lru_entry;
fs/nfs/fscache.c +9 -11
index e861d7bae305..a6fc1c8b6644 100644
@@ -163,13 +163,14 @@ void nfs_fscache_init_inode(struct inode *inode)
 	struct nfs_server *nfss = NFS_SERVER(inode);
 	struct nfs_inode *nfsi = NFS_I(inode);
 
-	nfsi->fscache = NULL;
+	netfs_inode(inode)->cache = NULL;
 	if (!(nfss->fscache && S_ISREG(inode->i_mode)))
 		return;
 
 	nfs_fscache_update_auxdata(&auxdata, inode);
 
-	nfsi->fscache = fscache_acquire_cookie(NFS_SB(inode->i_sb)->fscache,
+	netfs_inode(inode)->cache = fscache_acquire_cookie(
+					       nfss->fscache,
 					       0,
 					       nfsi->fh.data, /* index_key */
 					       nfsi->fh.size,
@@ -183,11 +184,8 @@ void nfs_fscache_init_inode(struct inode *inode)
  */
 void nfs_fscache_clear_inode(struct inode *inode)
 {
-	struct nfs_inode *nfsi = NFS_I(inode);
-	struct fscache_cookie *cookie = nfs_i_fscache(inode);
-
-	fscache_relinquish_cookie(cookie, false);
-	nfsi->fscache = NULL;
+	fscache_relinquish_cookie(netfs_i_cookie(&NFS_I(inode)->netfs), false);
+	netfs_inode(inode)->cache = NULL;
 }
 
 /*
@@ -212,7 +210,7 @@ void nfs_fscache_clear_inode(struct inode *inode)
 void nfs_fscache_open_file(struct inode *inode, struct file *filp)
 {
 	struct nfs_fscache_inode_auxdata auxdata;
-	struct fscache_cookie *cookie = nfs_i_fscache(inode);
+	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
 	bool open_for_write = inode_is_open_for_write(inode);
 
 	if (!fscache_cookie_valid(cookie))
@@ -230,7 +228,7 @@ EXPORT_SYMBOL_GPL(nfs_fscache_open_file);
 void nfs_fscache_release_file(struct inode *inode, struct file *filp)
 {
 	struct nfs_fscache_inode_auxdata auxdata;
-	struct fscache_cookie *cookie = nfs_i_fscache(inode);
+	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
 	loff_t i_size = i_size_read(inode);
 
 	nfs_fscache_update_auxdata(&auxdata, inode);
@@ -243,7 +241,7 @@ void nfs_fscache_release_file(struct inode *inode, struct file *filp)
 static int fscache_fallback_read_page(struct inode *inode, struct page *page)
 {
 	struct netfs_cache_resources cres;
-	struct fscache_cookie *cookie = nfs_i_fscache(inode);
+	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
 	struct iov_iter iter;
 	struct bio_vec bvec[1];
 	int ret;
@@ -271,7 +269,7 @@ static int fscache_fallback_write_page(struct inode *inode, struct page *page,
 				       bool no_space_allocated_yet)
 {
 	struct netfs_cache_resources cres;
-	struct fscache_cookie *cookie = nfs_i_fscache(inode);
+	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
 	struct iov_iter iter;
 	struct bio_vec bvec[1];
 	loff_t start = page_offset(page);
fs/nfs/fscache.h +6 -9
index 2a37af880978..38614ed8f951 100644
@@ -54,7 +54,7 @@ static inline bool nfs_fscache_release_folio(struct folio *folio, gfp_t gfp)
 		if (current_is_kswapd() || !(gfp & __GFP_FS))
 			return false;
 		folio_wait_fscache(folio);
-		fscache_note_page_release(nfs_i_fscache(folio->mapping->host));
+		fscache_note_page_release(netfs_i_cookie(&NFS_I(folio->mapping->host)->netfs));
 		nfs_inc_fscache_stats(folio->mapping->host,
 				      NFSIOS_FSCACHE_PAGES_UNCACHED);
 	}
@@ -66,7 +66,7 @@ static inline bool nfs_fscache_release_folio(struct folio *folio, gfp_t gfp)
  */
 static inline int nfs_fscache_read_page(struct inode *inode, struct page *page)
 {
-	if (nfs_i_fscache(inode))
+	if (netfs_inode(inode)->cache)
 		return __nfs_fscache_read_page(inode, page);
 	return -ENOBUFS;
 }
@@ -78,7 +78,7 @@ static inline int nfs_fscache_read_page(struct inode *inode, struct page *page)
 static inline void nfs_fscache_write_page(struct inode *inode,
 					   struct page *page)
 {
-	if (nfs_i_fscache(inode))
+	if (netfs_inode(inode)->cache)
 		__nfs_fscache_write_page(inode, page);
 }
 
@@ -101,13 +101,10 @@ static inline void nfs_fscache_update_auxdata(struct nfs_fscache_inode_auxdata *
 static inline void nfs_fscache_invalidate(struct inode *inode, int flags)
 {
 	struct nfs_fscache_inode_auxdata auxdata;
-	struct nfs_inode *nfsi = NFS_I(inode);
+	struct fscache_cookie *cookie =  netfs_i_cookie(&NFS_I(inode)->netfs);
 
-	if (nfsi->fscache) {
-		nfs_fscache_update_auxdata(&auxdata, inode);
-		fscache_invalidate(nfsi->fscache, &auxdata,
-				   i_size_read(inode), flags);
-	}
+	nfs_fscache_update_auxdata(&auxdata, inode);
+	fscache_invalidate(cookie, &auxdata, i_size_read(inode), flags);
 }
 
 /*
fs/nfs/inode.c +3 -3
index bea7c005119c..aa2aec785ab5 100644
@@ -1411,7 +1411,7 @@ int nfs_revalidate_mapping(struct inode *inode, struct address_space *mapping)
 
 static bool nfs_file_has_writers(struct nfs_inode *nfsi)
 {
-	struct inode *inode = &nfsi->vfs_inode;
+	struct inode *inode = VFS_I(nfsi);
 
 	if (!S_ISREG(inode->i_mode))
 		return false;
@@ -2249,7 +2249,7 @@ struct inode *nfs_alloc_inode(struct super_block *sb)
 #ifdef CONFIG_NFS_V4_2
 	nfsi->xattr_cache = NULL;
 #endif
-	return &nfsi->vfs_inode;
+	return VFS_I(nfsi);
 }
 EXPORT_SYMBOL_GPL(nfs_alloc_inode);
 
@@ -2273,7 +2273,7 @@ static void init_once(void *foo)
 {
 	struct nfs_inode *nfsi = (struct nfs_inode *) foo;
 
-	inode_init_once(&nfsi->vfs_inode);
+	inode_init_once(VFS_I(nfsi));
 	INIT_LIST_HEAD(&nfsi->open_files);
 	INIT_LIST_HEAD(&nfsi->access_cache_entry_lru);
 	INIT_LIST_HEAD(&nfsi->access_cache_inode_lru);
fs/nfs/internal.h +1 -1
index 898dd95bc7a7..4d240ac4430f 100644
@@ -355,7 +355,7 @@ nfs4_label_copy(struct nfs4_label *dst, struct nfs4_label *src)
 
 static inline void nfs_zap_label_cache_locked(struct nfs_inode *nfsi)
 {
-	if (nfs_server_capable(&nfsi->vfs_inode, NFS_CAP_SECURITY_LABEL))
+	if (nfs_server_capable(VFS_I(nfsi), NFS_CAP_SECURITY_LABEL))
 		nfsi->cache_validity |= NFS_INO_INVALID_LABEL;
 }
 #else
fs/nfs/pnfs.c +6 -6
index 2613b7e36eb9..035bf2eac2cf 100644
@@ -763,19 +763,19 @@ static struct pnfs_layout_hdr *__pnfs_destroy_layout(struct nfs_inode *nfsi)
 	struct pnfs_layout_hdr *lo;
 	LIST_HEAD(tmp_list);
 
-	spin_lock(&nfsi->vfs_inode.i_lock);
+	spin_lock(&VFS_I(nfsi)->i_lock);
 	lo = nfsi->layout;
 	if (lo) {
 		pnfs_get_layout_hdr(lo);
 		pnfs_mark_layout_stateid_invalid(lo, &tmp_list);
 		pnfs_layout_clear_fail_bit(lo, NFS_LAYOUT_RO_FAILED);
 		pnfs_layout_clear_fail_bit(lo, NFS_LAYOUT_RW_FAILED);
-		spin_unlock(&nfsi->vfs_inode.i_lock);
+		spin_unlock(&VFS_I(nfsi)->i_lock);
 		pnfs_free_lseg_list(&tmp_list);
-		nfs_commit_inode(&nfsi->vfs_inode, 0);
+		nfs_commit_inode(VFS_I(nfsi), 0);
 		pnfs_put_layout_hdr(lo);
 	} else
-		spin_unlock(&nfsi->vfs_inode.i_lock);
+		spin_unlock(&VFS_I(nfsi)->i_lock);
 	return lo;
 }
 
@@ -790,9 +790,9 @@ static bool pnfs_layout_removed(struct nfs_inode *nfsi,
 {
 	bool ret;
 
-	spin_lock(&nfsi->vfs_inode.i_lock);
+	spin_lock(&VFS_I(nfsi)->i_lock);
 	ret = nfsi->layout != lo;
-	spin_unlock(&nfsi->vfs_inode.i_lock);
+	spin_unlock(&VFS_I(nfsi)->i_lock);
 	return ret;
 }
 
fs/nfs/write.c +1 -1
index f41d24b54fd1..911569022036 100644
@@ -828,7 +828,7 @@ nfs_page_search_commits_for_head_request_locked(struct nfs_inode *nfsi,
 {
 	struct nfs_page *freq, *t;
 	struct nfs_commit_info cinfo;
-	struct inode *inode = &nfsi->vfs_inode;
+	struct inode *inode = VFS_I(nfsi);
 
 	nfs_init_cinfo_from_inode(&cinfo, inode);
 
include/linux/nfs_fs.h +23 -11
index 7931fa472561..a1c402e26abf 100644
@@ -31,6 +31,10 @@
 #include <linux/sunrpc/auth.h>
 #include <linux/sunrpc/clnt.h>
 
+#ifdef CONFIG_NFS_FSCACHE
+#include <linux/netfs.h>
+#endif
+
 #include <linux/nfs.h>
 #include <linux/nfs2.h>
 #include <linux/nfs3.h>
@@ -204,9 +208,11 @@ struct nfs_inode {
 	__u64 write_io;
 	__u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-	struct fscache_cookie	*fscache;
-#endif
+	struct netfs_inode	netfs; /* netfs context and VFS inode */
+#else
 	struct inode		vfs_inode;
+#endif
+
 
 #ifdef CONFIG_NFS_V4_2
 	struct nfs4_xattr_cache *xattr_cache;
@@ -281,10 +287,25 @@ struct nfs4_copy_state {
 #define NFS_INO_LAYOUTSTATS	(11)		/* layoutstats inflight */
 #define NFS_INO_ODIRECT		(12)		/* I/O setting is O_DIRECT */
 
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+	return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+	return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+	return &nfsi->vfs_inode;
+}
 static inline struct nfs_inode *NFS_I(const struct inode *inode)
 {
 	return container_of(inode, struct nfs_inode, vfs_inode);
 }
+#endif
 
 static inline struct nfs_server *NFS_SB(const struct super_block *s)
 {
@@ -328,15 +349,6 @@ static inline int NFS_STALE(const struct inode *inode)
 	return test_bit(NFS_INO_STALE, &NFS_I(inode)->flags);
 }
 
-static inline struct fscache_cookie *nfs_i_fscache(struct inode *inode)
-{
-#ifdef CONFIG_NFS_FSCACHE
-	return NFS_I(inode)->fscache;
-#else
-	return NULL;
-#endif
-}
-
 static inline __u64 NFS_FILEID(const struct inode *inode)
 {
 	return NFS_I(inode)->fileid;
Signature · 1 line
2.31.1
#6 lore raw

PATCH v9 3/5 NFS: Convert buffered read paths to use netfs when fscache is enabled

In reply to Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs, but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled in by the network filesystem. For NFS, we only need to define 5 of the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read cannot be fulfilled locally, and must be sent to the server (either the cache is not active, or it is active but the data is not available). Once the read from the server is complete, netfs requires a call to netfs_subreq_terminated() which conveys either how many bytes were read successfully, or an error. Note that issue_read() is called with a structure, netfs_io_subrequest, which defines the IO requested, and contains a start and a length (both in bytes), and assumes the underlying netfs will return a either an error on the whole region, or the number of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined in pagelist.c. For the pgio APIs, there is no way for the caller to know how many RPCs will be sent and how the pages will be broken up into underlying RPCs, each of which will have their own completion and return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated(). Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure for each nfs_netfs_issue_read() call, store the final error and number of bytes successfully transferred in the structure, and update these values as each RPC completes. The refcount on the structure is used as a marker for the last RPC completion, is incremented in nfs_netfs_read_initiate(), and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header contains a valid pointer to the data. On the final put (which signals the final outstanding RPC is complete) in nfs_netfs_read_completion(), call netfs_subreq_terminated() with either the final error value (if one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note that when all RPCs complete successfully, the number of bytes transferred is capped to the length of the subrequest. Capping the transferred length to the subrequest length prevents "Subreq overread" warnings from netfs. This is due to the "aligned_len" in nfs_pageio_add_page(), and the corner case where NFS requests a full page at the end of the file, even when i_size reflects only a partial page (NFS overread).

---
 fs/nfs/fscache.c         | 232 +++++++++++++++++++++++----------------
 fs/nfs/fscache.h         | 102 +++++++++++------
 fs/nfs/inode.c           |   2 +
 fs/nfs/internal.h        |   9 ++
 fs/nfs/pagelist.c        |  12 ++
 fs/nfs/read.c            |  50 ++++-----
 include/linux/nfs_page.h |   3 +
 include/linux/nfs_xdr.h  |   3 +
 8 files changed, 261 insertions(+), 152 deletions(-)
fs/nfs/fscache.c +138 -94
index a6fc1c8b6644..58462f6579d6 100644
@@ -15,6 +15,9 @@
 #include <linux/seq_file.h>
 #include <linux/slab.h>
 #include <linux/iversion.h>
+#include <linux/xarray.h>
+#include <linux/fscache.h>
+#include <linux/netfs.h>
 
 #include "internal.h"
 #include "iostat.h"
@@ -184,7 +187,7 @@ void nfs_fscache_init_inode(struct inode *inode)
  */
 void nfs_fscache_clear_inode(struct inode *inode)
 {
-	fscache_relinquish_cookie(netfs_i_cookie(&NFS_I(inode)->netfs), false);
+	fscache_relinquish_cookie(netfs_i_cookie(netfs_inode(inode)), false);
 	netfs_inode(inode)->cache = NULL;
 }
 
@@ -210,7 +213,7 @@ void nfs_fscache_clear_inode(struct inode *inode)
 void nfs_fscache_open_file(struct inode *inode, struct file *filp)
 {
 	struct nfs_fscache_inode_auxdata auxdata;
-	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
+	struct fscache_cookie *cookie = netfs_i_cookie(netfs_inode(inode));
 	bool open_for_write = inode_is_open_for_write(inode);
 
 	if (!fscache_cookie_valid(cookie))
@@ -228,119 +231,160 @@ EXPORT_SYMBOL_GPL(nfs_fscache_open_file);
 void nfs_fscache_release_file(struct inode *inode, struct file *filp)
 {
 	struct nfs_fscache_inode_auxdata auxdata;
-	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
+	struct fscache_cookie *cookie = netfs_i_cookie(netfs_inode(inode));
 	loff_t i_size = i_size_read(inode);
 
 	nfs_fscache_update_auxdata(&auxdata, inode);
 	fscache_unuse_cookie(cookie, &auxdata, &i_size);
 }
 
-/*
- * Fallback page reading interface.
- */
-static int fscache_fallback_read_page(struct inode *inode, struct page *page)
+int nfs_netfs_read_folio(struct file *file, struct folio *folio)
 {
-	struct netfs_cache_resources cres;
-	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
-	struct iov_iter iter;
-	struct bio_vec bvec[1];
-	int ret;
-
-	memset(&cres, 0, sizeof(cres));
-	bvec[0].bv_page		= page;
-	bvec[0].bv_offset	= 0;
-	bvec[0].bv_len		= PAGE_SIZE;
-	iov_iter_bvec(&iter, READ, bvec, ARRAY_SIZE(bvec), PAGE_SIZE);
-
-	ret = fscache_begin_read_operation(&cres, cookie);
-	if (ret < 0)
-		return ret;
-
-	ret = fscache_read(&cres, page_offset(page), &iter, NETFS_READ_HOLE_FAIL,
-			   NULL, NULL);
-	fscache_end_operation(&cres);
-	return ret;
+	if (!netfs_inode(folio_inode(folio))->cache)
+		return -ENOBUFS;
+
+	return netfs_read_folio(file, folio);
 }
 
-/*
- * Fallback page writing interface.
- */
-static int fscache_fallback_write_page(struct inode *inode, struct page *page,
-				       bool no_space_allocated_yet)
+int nfs_netfs_readahead(struct readahead_control *ractl)
 {
-	struct netfs_cache_resources cres;
-	struct fscache_cookie *cookie = netfs_i_cookie(&NFS_I(inode)->netfs);
-	struct iov_iter iter;
-	struct bio_vec bvec[1];
-	loff_t start = page_offset(page);
-	size_t len = PAGE_SIZE;
-	int ret;
-
-	memset(&cres, 0, sizeof(cres));
-	bvec[0].bv_page		= page;
-	bvec[0].bv_offset	= 0;
-	bvec[0].bv_len		= PAGE_SIZE;
-	iov_iter_bvec(&iter, WRITE, bvec, ARRAY_SIZE(bvec), PAGE_SIZE);
-
-	ret = fscache_begin_write_operation(&cres, cookie);
-	if (ret < 0)
-		return ret;
-
-	ret = cres.ops->prepare_write(&cres, &start, &len, i_size_read(inode),
-				      no_space_allocated_yet);
-	if (ret == 0)
-		ret = fscache_write(&cres, page_offset(page), &iter, NULL, NULL);
-	fscache_end_operation(&cres);
-	return ret;
+	struct inode *inode = ractl->mapping->host;
+
+	if (!netfs_inode(inode)->cache)
+		return -ENOBUFS;
+
+	netfs_readahead(ractl);
+	return 0;
 }
 
-/*
- * Retrieve a page from fscache
- */
-int __nfs_fscache_read_page(struct inode *inode, struct page *page)
+atomic_t nfs_netfs_debug_id;
+static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *file)
 {
-	int ret;
+	rreq->netfs_priv = get_nfs_open_context(nfs_file_open_context(file));
+	rreq->debug_id = atomic_inc_return(&nfs_netfs_debug_id);
 
-	trace_nfs_fscache_read_page(inode, page);
-	if (PageChecked(page)) {
-		ClearPageChecked(page);
-		ret = 1;
-		goto out;
-	}
+	return 0;
+}
 
-	ret = fscache_fallback_read_page(inode, page);
-	if (ret < 0) {
-		nfs_inc_fscache_stats(inode, NFSIOS_FSCACHE_PAGES_READ_FAIL);
-		SetPageChecked(page);
-		goto out;
-	}
+static void nfs_netfs_free_request(struct netfs_io_request *rreq)
+{
+	put_nfs_open_context(rreq->netfs_priv);
+}
 
-	/* Read completed synchronously */
-	nfs_inc_fscache_stats(inode, NFSIOS_FSCACHE_PAGES_READ_OK);
-	SetPageUptodate(page);
-	ret = 0;
-out:
-	trace_nfs_fscache_read_page_exit(inode, page, ret);
-	return ret;
+static inline int nfs_netfs_begin_cache_operation(struct netfs_io_request *rreq)
+{
+	return fscache_begin_read_operation(&rreq->cache_resources,
+					    netfs_i_cookie(netfs_inode(rreq->inode)));
 }
 
-/*
- * Store a newly fetched page in fscache.  We can be certain there's no page
- * stored in the cache as yet otherwise we would've read it from there.
- */
-void __nfs_fscache_write_page(struct inode *inode, struct page *page)
+static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *sreq)
 {
-	int ret;
+	struct nfs_netfs_io_data *netfs;
+
+	netfs = kzalloc(sizeof(*netfs), GFP_KERNEL_ACCOUNT);
+	if (!netfs)
+		return NULL;
+	netfs->sreq = sreq;
+	refcount_set(&netfs->refcount, 1);
+	return netfs;
+}
 
-	trace_nfs_fscache_write_page(inode, page);
+static bool nfs_netfs_clamp_length(struct netfs_io_subrequest *sreq)
+{
+	size_t	rsize = NFS_SB(sreq->rreq->inode->i_sb)->rsize;
 
-	ret = fscache_fallback_write_page(inode, page, true);
+	sreq->len = min(sreq->len, rsize);
+	return true;
+}
 
-	if (ret != 0) {
-		nfs_inc_fscache_stats(inode, NFSIOS_FSCACHE_PAGES_WRITTEN_FAIL);
-		nfs_inc_fscache_stats(inode, NFSIOS_FSCACHE_PAGES_UNCACHED);
-	} else {
-		nfs_inc_fscache_stats(inode, NFSIOS_FSCACHE_PAGES_WRITTEN_OK);
+static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
+{
+	struct nfs_netfs_io_data	*netfs;
+	struct nfs_pageio_descriptor	pgio;
+	struct inode *inode = sreq->rreq->inode;
+	struct nfs_open_context *ctx = sreq->rreq->netfs_priv;
+	struct page *page;
+	int err;
+	pgoff_t start = (sreq->start + sreq->transferred) >> PAGE_SHIFT;
+	pgoff_t last = ((sreq->start + sreq->len -
+			 sreq->transferred - 1) >> PAGE_SHIFT);
+	XA_STATE(xas, &sreq->rreq->mapping->i_pages, start);
+
+	nfs_pageio_init_read(&pgio, inode, false,
+			     &nfs_async_read_completion_ops);
+
+	netfs = nfs_netfs_alloc(sreq);
+	if (!netfs)
+		return netfs_subreq_terminated(sreq, -ENOMEM, false);
+
+	pgio.pg_netfs = netfs; /* used in completion */
+
+	xas_lock(&xas);
+	xas_for_each(&xas, page, last) {
+		/* nfs_pageio_add_page() may schedule() due to pNFS layout and other RPCs  */
+		xas_pause(&xas);
+		xas_unlock(&xas);
+		err = nfs_pageio_add_page(&pgio, ctx, page);
+		if (err < 0) {
+			netfs->error = err;
+			goto out;
+		}
+		xas_lock(&xas);
 	}
-	trace_nfs_fscache_write_page_exit(inode, page, ret);
+	xas_unlock(&xas);
+out:
+	nfs_pageio_complete_read(&pgio);
+	nfs_netfs_put(netfs);
+}
+
+void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr)
+{
+	struct nfs_netfs_io_data        *netfs = hdr->netfs;
+
+	if (!netfs)
+		return;
+
+	nfs_netfs_get(netfs);
+}
+
+void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+	struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
+
+	/*
+	 * If fscache is enabled, netfs will unlock pages.
+	 */
+	if (netfs_inode(inode)->cache)
+		return;
+
+	unlock_page(req->wb_page);
 }
+
+void nfs_netfs_read_completion(struct nfs_pgio_header *hdr)
+{
+	struct nfs_netfs_io_data        *netfs = hdr->netfs;
+	struct netfs_io_subrequest      *sreq;
+
+	if (!netfs)
+		return;
+
+	sreq = netfs->sreq;
+	if (test_bit(NFS_IOHDR_EOF, &hdr->flags))
+		__set_bit(NETFS_SREQ_CLEAR_TAIL, &sreq->flags);
+
+	if (hdr->error)
+		netfs->error = hdr->error;
+	else
+		atomic64_add(hdr->res.count, &netfs->transferred);
+
+	nfs_netfs_put(netfs);
+	hdr->netfs = NULL;
+}
+
+const struct netfs_request_ops nfs_netfs_ops = {
+	.init_request		= nfs_netfs_init_request,
+	.free_request		= nfs_netfs_free_request,
+	.begin_cache_operation	= nfs_netfs_begin_cache_operation,
+	.issue_read		= nfs_netfs_issue_read,
+	.clamp_length		= nfs_netfs_clamp_length
+};
fs/nfs/fscache.h +70 -32

141 lines not on this page · raw message

fs/nfs/inode.c +2 -0
index aa2aec785ab5..b36a02b932e8 100644
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct super_block *sb)
 #ifdef CONFIG_NFS_V4_2
 	nfsi->xattr_cache = NULL;
 #endif
+	nfs_netfs_inode_init(nfsi);
+
 	return VFS_I(nfsi);
 }
 EXPORT_SYMBOL_GPL(nfs_alloc_inode);
fs/nfs/internal.h +9 -0
index 4d240ac4430f..e9c0e4abc954 100644
@@ -453,6 +453,10 @@ extern void nfs_sb_deactive(struct super_block *sb);
 extern int nfs_client_for_each_server(struct nfs_client *clp,
 				      int (*fn)(struct nfs_server *, void *),
 				      void *data);
+#ifdef CONFIG_NFS_FSCACHE
+extern const struct netfs_request_ops nfs_netfs_ops;
+#endif
+
 /* io.c */
 extern void nfs_start_io_read(struct inode *inode);
 extern void nfs_end_io_read(struct inode *inode);
@@ -482,9 +486,14 @@ extern int nfs4_get_rootfh(struct nfs_server *server, struct nfs_fh *mntfh, bool
 
 struct nfs_pgio_completion_ops;
 /* read.c */
+extern const struct nfs_pgio_completion_ops nfs_async_read_completion_ops;
 extern void nfs_pageio_init_read(struct nfs_pageio_descriptor *pgio,
 			struct inode *inode, bool force_mds,
 			const struct nfs_pgio_completion_ops *compl_ops);
+extern int nfs_pageio_add_page(struct nfs_pageio_descriptor *pgio,
+			       struct nfs_open_context *ctx,
+			       struct page *page);
+extern void nfs_pageio_complete_read(struct nfs_pageio_descriptor *pgio);
 extern void nfs_read_prepare(struct rpc_task *task, void *calldata);
 extern void nfs_pageio_reset_read_mds(struct nfs_pageio_descriptor *pgio);
 
fs/nfs/pagelist.c +12 -0
index 317cedfa52bf..e28754476d1b 100644
@@ -25,6 +25,7 @@
 #include "internal.h"
 #include "pnfs.h"
 #include "nfstrace.h"
+#include "fscache.h"
 
 #define NFSDBG_FACILITY		NFSDBG_PAGECACHE
 
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct nfs_pageio_descriptor *desc,
 	hdr->good_bytes = mirror->pg_count;
 	hdr->io_completion = desc->pg_io_completion;
 	hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+	if (desc->pg_netfs)
+		hdr->netfs = desc->pg_netfs;
+#endif
 	hdr->release = release;
 	hdr->completion_ops = desc->pg_completion_ops;
 	if (hdr->completion_ops->init_hdr)
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
 	desc->pg_lseg = NULL;
 	desc->pg_io_completion = NULL;
 	desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+	desc->pg_netfs = NULL;
+#endif
 	desc->pg_bsize = bsize;
 
 	desc->pg_mirror_count = 1;
@@ -940,6 +948,7 @@ int nfs_generic_pgio(struct nfs_pageio_descriptor *desc,
 	/* Set up the argument struct */
 	nfs_pgio_rpcsetup(hdr, mirror->pg_count, desc->pg_ioflags, &cinfo);
 	desc->pg_rpc_callops = &nfs_pgio_common_ops;
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nfs_generic_pgio);
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct nfs_pageio_descriptor *desc,
 
 	desc->pg_io_completion = hdr->io_completion;
 	desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+	desc->pg_netfs = hdr->netfs;
+#endif
 	list_splice_init(&hdr->pages, &pages);
 	while (!list_empty(&pages)) {
 		struct nfs_page *req = nfs_list_entry(pages.next);
fs/nfs/read.c +24 -26

129 lines not on this page · raw message

include/linux/nfs_page.h +3 -0
index ba7e2e4b0926..8eeb16d9bacd 100644
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
 	struct pnfs_layout_segment *pg_lseg;
 	struct nfs_io_completion *pg_io_completion;
 	struct nfs_direct_req	*pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+	void			*pg_netfs;
+#endif
 	unsigned int		pg_bsize;	/* default bsize for mirrors */
 
 	u32			pg_mirror_count;
include/linux/nfs_xdr.h +3 -0
index e86cf6642d21..e196ef595908 100644
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
 	const struct nfs_rw_ops	*rw_ops;
 	struct nfs_io_completion *io_completion;
 	struct nfs_direct_req	*dreq;
+#ifdef CONFIG_NFS_FSCACHE
+	void			*netfs;
+#endif
 
 	int			pnfs_error;
 	int			error;		/* merge with pnfs_error */
Signature · 1 line
2.31.1
maintainer
#7 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 64 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs, but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled in by the network filesystem.  For NFS, we only need to define 5 of the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read cannot be fulfilled locally, and must be sent to the server (either the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to netfs_subreq_terminated() which conveys either how many bytes were read
successfully, or an error.  Note that issue_read() is called with a structure, netfs_io_subrequest, which defines the IO requested, and contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c.  For the pgio APIs, there is no way for the caller to know how many RPCs will be sent and how the pages will be broken up into underlying RPCs, each of which will have their own completion and
return code.  In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these values
as each RPC completes.  The refcount on the structure is used as a marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data.  On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully).  Note that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest.  Capping the transferred length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the corner case where NFS requests a full page at the end of the file, even when i_size reflects only a partial page (NFS overread).

Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

Instead, it is just wrapping the NFS requests inside netfs requests. As it stands, that means it is just duplicating information, and adding unnecessary overhead to the standard I/O path (extra allocations, extra indirect calls, and extra bloat to the inode).

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#8 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 66 lines · Trond Myklebust
Quoted 64 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs, but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled in by the network filesystem. For NFS, we only need to define 5 of the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read cannot be fulfilled locally, and must be sent to the server (either the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to netfs_subreq_terminated() which conveys either how many bytes were read
successfully, or an error. Note that issue_read() is called with a structure, netfs_io_subrequest, which defines the IO requested, and contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to know how many RPCs will be sent and how the pages will be broken up into underlying RPCs, each of which will have their own completion and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these values
as each RPC completes. The refcount on the structure is used as a marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the corner case where NFS requests a full page at the end of the file, even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 5 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As it stands, that means it is just duplicating information, and adding unnecessary overhead to the standard I/O path (extra allocations, extra indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Can you be more specific as to the portions of the patch you don't like so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an
#ifdef CONFIG_NFS_FSCACHE? If so, do you require no
extra size be added to nfs_inode?

@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;

#ifdef CONFIG_NFS_FSCACHE
- struct fscache_cookie *fscache;
-#endif
+ struct netfs_inode netfs; /* netfs context and VFS inode */
+#else

        struct inode            vfs_inode;

+#endif
+

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {} +static inline void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr) {} +static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {} +static inline void nfs_netfs_readpage_release(struct nfs_page *req) +{
+ unlock_page(req->wb_page);
+}
static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {} static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct super_block *sb)

 #ifdef CONFIG_NFS_V4_2
        nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
        return VFS_I(nfsi);

}
EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct folio *folio)

        if (NFS_STALE(inode))
                goto out_unlock;

+ ret = nfs_netfs_read_folio(file, folio);
+ if (!ret)
+ goto out;
+

@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control *ractl)

        if (NFS_STALE(inode))
                goto out;

+ ret = nfs_netfs_readahead(ractl);
+ if (!ret)
+ goto out;
+

And how about these calls from different points in the read path to the earlier mentioned stub functions?

@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);

static void nfs_readpage_release(struct nfs_page *req, int error) {

-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
        struct page *page = req->wb_page;

- dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb->s_id,
- (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
- (long long)req_offset(req));
-

        if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
                SetPageError(page);

- if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
- if (PageUptodate(page))
- nfs_fscache_write_page(inode, page);
- unlock_page(page);
- }
+ if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+ nfs_netfs_readpage_release(req);
+

        nfs_release_request(req);

}

@@ -177,6 +170,8 @@ static void nfs_read_completion(struct nfs_pgio_header *hdr)

                nfs_list_remove_request(req);
                nfs_readpage_release(req, error);
        }
+       nfs_netfs_read_completion(hdr);
+
 out:
        hdr->release(hdr);
 }
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct nfs_pgio_header *hdr,
                              struct rpc_task_setup *task_setup_data, int how)
 {
        rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
        trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
        struct pnfs_layout_segment *pg_lseg;
        struct nfs_io_completion *pg_io_completion;
        struct nfs_direct_req   *pg_dreq;

+#ifdef CONFIG_NFS_FSCACHE
+ void *pg_netfs;
+#endif

@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
        const struct nfs_rw_ops *rw_ops;
        struct nfs_io_completion *io_completion;
        struct nfs_direct_req   *dreq;

+#ifdef CONFIG_NFS_FSCACHE
+ void *netfs;
+#endif

And these additions to pagelist.c?

@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct nfs_pageio_descriptor *desc,

        hdr->good_bytes = mirror->pg_count;
        hdr->io_completion = desc->pg_io_completion;
        hdr->dreq = desc->pg_dreq;

+#ifdef CONFIG_NFS_FSCACHE
+ if (desc->pg_netfs)
+ hdr->netfs = desc->pg_netfs;
+#endif

@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,

        desc->pg_lseg = NULL;
        desc->pg_io_completion = NULL;
        desc->pg_dreq = NULL;

+#ifdef CONFIG_NFS_FSCACHE
+ desc->pg_netfs = NULL;
+#endif

@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct nfs_pageio_descriptor *desc,

        desc->pg_io_completion = hdr->io_completion;
        desc->pg_dreq = hdr->dreq;

+#ifdef CONFIG_NFS_FSCACHE
+ desc->pg_netfs = hdr->netfs;
+#endif

Quoted 5 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

maintainer
#9 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem.  For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error.  Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c.  For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code.  In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes.  The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data.  On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest.  Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig.  I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure.  Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured?  Or when it is not?  Or both?  I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface.  This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

Quoted 14 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on.  I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE?  If so, do you require no extra size be added to nfs_inode?

(quoted diff) +0 -0
@@ -204,9 +208,11 @@ struct nfs_inode {
         __u64 write_io;

Review struct nfs_inode { · line 216

Quoted 10 lines
(quoted diff) +4 -2
         __u64 read_io;
  #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
         struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Quoted 39 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

 #else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

fs/nfs/fscache.h +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

 static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
 static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below?  If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

fs/nfs/inode.c +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
  #ifdef CONFIG_NFS_V4_2
         nfsi->xattr_cache = NULL;
  #endif
+       nfs_netfs_inode_init(nfsi);
+
         return VFS_I(nfsi);
  }

 EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 9 lines

Do you object to the changes in fs/nfs/read.c?  Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls?  So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request().  But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want.  Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Quoted 40 lines
fs/nfs/read.c +8 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
         if (NFS_STALE(inode))
                 goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
         if (NFS_STALE(inode))
                 goto out;
 
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 
 

And how about these calls from different points in the read path to the earlier mentioned stub functions?

fs/nfs/read.c +0 -5
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
  static void nfs_readpage_release(struct nfs_page *req, int error)
  {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
         struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 >s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-

        if (nfs_error_is_fatal_on_server(error) && error != - ETIMEDOUT)
                SetPageError(page);

Quoted 8 lines

-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) { -               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page); -               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) +               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Quoted 52 lines

        nfs_release_request(req);
 }

fs/nfs/read.c +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
                 nfs_list_remove_request(req);
                 nfs_readpage_release(req, error);
         }
+       nfs_netfs_read_completion(hdr);
+

 out:
        hdr->release(hdr);
 }

fs/nfs/read.c +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                               struct rpc_task_setup *task_setup_data,
 int how)
  {
         rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
         trace_nfs_initiate_read(hdr);

 }

Are you ok with these additions?  Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not.  That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

include/linux/nfs_page.h +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
         struct pnfs_layout_segment *pg_lseg;
         struct nfs_io_completion *pg_io_completion;
         struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
         const struct nfs_rw_ops *rw_ops;
         struct nfs_io_completion *io_completion;
         struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

fs/nfs/pagelist.c void nfs_pgheader_init(struct · line 76

Quoted 8 lines
fs/nfs/pagelist.c +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
         hdr->good_bytes = mirror->pg_count;
         hdr->io_completion = desc->pg_io_completion;
         hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Quoted 13 lines
fs/nfs/pagelist.c +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
         desc->pg_lseg = NULL;
         desc->pg_io_completion = NULL;
         desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 

fs/nfs/pagelist.c int nfs_pageio_resend(struct · line 1375

Quoted 7 lines
fs/nfs/pagelist.c +3 -0
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
         desc->pg_io_completion = hdr->io_completion;
         desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches.  Hopefully the above questions will clarify the next steps.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#10 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
        if (!netfs_inode(folio_inode(folio))->cache)
                return -ENOBUFS;

        return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
        struct inode *inode = ractl->mapping->host;

        if (!netfs_inode(inode)->cache)
                return -ENOBUFS;

        netfs_readahead(ractl);
        return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

fs/nfs/internal.h extern void nfs_sb_deactive(struct super_block *sb); · line 459

Quoted 3 lines
fs/nfs/internal.h +2 -0
        struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;

#ifdef CONFIG_NFS_FSCACHE
- struct fscache_cookie *fscache;
-#endif
+ struct netfs_inode *netfs; /* netfs context and VFS inode */
+#else

        struct inode            vfs_inode;

+#endif
+

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi) +{
+ return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode) +{
+ return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi) +{
+ return &nfsi->vfs_inode;
+}
static inline struct nfs_inode *NFS_I(const struct inode *inode)

 {
        return container_of(inode, struct nfs_inode, vfs_inode);

}
+#endif

Quoted 39 lines
Quoted 39 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

fs/nfs/fscache.h +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}
 static inline void nfs_fscache_release_super_cookie(struct

super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

fs/nfs/inode.c +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
        nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
        return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
fs/nfs/read.c +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
        if (NFS_STALE(inode))
                goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
        if (NFS_STALE(inode))
                goto out;
 
Quoted 5 lines

fs/nfs/read.c void nfs_readahead(struct readahead_control *ractl)

Quoted 5 lines
fs/nfs/read.c +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

fs/nfs/read.c +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
        struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 >s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
        if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
                SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

fs/nfs/read.c EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 119

Quoted 5 lines
fs/nfs/read.c +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
        struct page *page = req->wb_page;

        if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
                SetPageError(page);
        if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
                unlock_page(req->wb_page);
#else
                nfs_netfs_readpage_release(req);
#endif
        nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req)

{
    struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

    /*
     * If fscache is enabled, netfs will unlock pages.
     */
    if (netfs_inode(inode)->cache)
        return;

    unlock_page(req->wb_page);
}
Quoted 51 lines
Quoted 51 lines
fs/nfs/read.c +0 -0
        nfs_release_request(req);

}

fs/nfs/read.c +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
                nfs_list_remove_request(req);
                nfs_readpage_release(req, error);
        }
+       nfs_netfs_read_completion(hdr);
+
 out:
        hdr->release(hdr);

}

fs/nfs/read.c +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                              struct rpc_task_setup *task_setup_data,
 int how)
 {
        rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
        trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

include/linux/nfs_page.h +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
        struct pnfs_layout_segment *pg_lseg;
        struct nfs_io_completion *pg_io_completion;
        struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
        const struct nfs_rw_ops *rw_ops;
        struct nfs_io_completion *io_completion;
        struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

fs/nfs/pagelist.c void nfs_pgheader_init(struct · line 76

Quoted 9 lines
fs/nfs/pagelist.c +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
        hdr->good_bytes = mirror->pg_count;
        hdr->io_completion = desc->pg_io_completion;
        hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
fs/nfs/pagelist.c +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
        desc->pg_lseg = NULL;
        desc->pg_io_completion = NULL;
        desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

fs/nfs/pagelist.c int nfs_pageio_resend(struct nfs_pageio_descriptor *desc, · line 1374

Quoted 5 lines
fs/nfs/pagelist.c +3 -0
        desc->pg_io_completion = hdr->io_completion;
        desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 19 lines
Quoted 8 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

Quoted 4 lines

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

#11 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 249 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
        if (!netfs_inode(folio_inode(folio))->cache)
                return -ENOBUFS;

        return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
        struct inode *inode = ractl->mapping->host;

        if (!netfs_inode(inode)->cache)
                return -ENOBUFS;

        netfs_readahead(ractl);
        return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
        struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
        struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +9 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}

Review

Quoted 10 lines
(quoted diff) +6 -0
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}
 static inline struct nfs_inode *NFS_I(const struct inode *inode)
 {
        return container_of(inode, struct nfs_inode, vfs_inode);
 }
+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +9 -5
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                      bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
        const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
        struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
        loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
  * Resources required to do operations on a cache.
  */
@@ -312,10 +316,12 @@ static inline struct netfs_inode

*netfs_inode(struct inode *inode)
static inline void netfs_inode_init(struct netfs_inode *ctx,
const struct netfs_request_ops *ops)
{

-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);

#if IS_ENABLED(CONFIG_FSCACHE)
- ctx->cache = NULL;
+ ctx->netfs->cache = NULL;
#endif
}

Quoted 269 lines
Quoted 40 lines
Quoted 40 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}
 static inline void nfs_fscache_release_super_cookie(struct

super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
        nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
        return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
        if (NFS_STALE(inode))
                goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
        if (NFS_STALE(inode))
                goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
        struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 >s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
        if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
                SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
        struct page *page = req->wb_page;

        if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
                SetPageError(page);
        if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
                unlock_page(req->wb_page);
#else
                nfs_netfs_readpage_release(req);
#endif
        nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req)

{
    struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

    /*
     * If fscache is enabled, netfs will unlock pages.
     */
    if (netfs_inode(inode)->cache)
        return;

    unlock_page(req->wb_page);
}
Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
        nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
                nfs_list_remove_request(req);
                nfs_readpage_release(req, error);
        }
+       nfs_netfs_read_completion(hdr);
+
 out:
        hdr->release(hdr);

}

(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                              struct rpc_task_setup *task_setup_data,
 int how)
 {
        rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
        trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
        struct pnfs_layout_segment *pg_lseg;
        struct nfs_io_completion *pg_io_completion;
        struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
        const struct nfs_rw_ops *rw_ops;
        struct nfs_io_completion *io_completion;
        struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
        hdr->good_bytes = mirror->pg_count;
        hdr->io_completion = desc->pg_io_completion;
        hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
        desc->pg_lseg = NULL;
        desc->pg_io_completion = NULL;
        desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
        desc->pg_io_completion = hdr->io_completion;
        desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 19 lines
Quoted 8 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

Quoted 4 lines

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

#12 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Just wanted to add that I am really keen to see some form of these patches merged.

I am using FS-Cache for the NFS re-exporting use-case.

I (and a fair few others) are impacted heavily by the IO performance bottleneck when using FS-Cache, so much so that we are currently stuck on v5.16 of the kernel.

--
Ben

Quoted 595 lines · David Wysochanski
Quoted 249 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
        if (!netfs_inode(folio_inode(folio))->cache)
                return -ENOBUFS;

        return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
        struct inode *inode = ractl->mapping->host;

        if (!netfs_inode(inode)->cache)
                return -ENOBUFS;

        netfs_readahead(ractl);
        return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
        struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
        __u64 write_io;
        __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
        struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +9 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}

Review

Quoted 10 lines
(quoted diff) +6 -0
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}
 static inline struct nfs_inode *NFS_I(const struct inode *inode)
 {
        return container_of(inode, struct nfs_inode, vfs_inode);
 }
+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                      bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
        const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
        struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
        loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
  * Resources required to do operations on a cache.
  */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                    const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 272 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}
 static inline void nfs_fscache_release_super_cookie(struct

super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
        nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
        return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
        if (NFS_STALE(inode))
                goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
        if (NFS_STALE(inode))
                goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
        struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 >s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
        if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
                SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
        struct page *page = req->wb_page;

        if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
                SetPageError(page);
        if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
                unlock_page(req->wb_page);
#else
                nfs_netfs_readpage_release(req);
#endif
        nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req)

{
    struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

    /*
     * If fscache is enabled, netfs will unlock pages.
     */
    if (netfs_inode(inode)->cache)
        return;

    unlock_page(req->wb_page);
}
Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
        nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
                nfs_list_remove_request(req);
                nfs_readpage_release(req, error);
        }
+       nfs_netfs_read_completion(hdr);
+
 out:
        hdr->release(hdr);

}

(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                              struct rpc_task_setup *task_setup_data,
 int how)
 {
        rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
        trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
        struct pnfs_layout_segment *pg_lseg;
        struct nfs_io_completion *pg_io_completion;
        struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
        const struct nfs_rw_ops *rw_ops;
        struct nfs_io_completion *io_completion;
        struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
        hdr->good_bytes = mirror->pg_count;
        hdr->io_completion = desc->pg_io_completion;
        hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
        desc->pg_lseg = NULL;
        desc->pg_io_completion = NULL;
        desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
        desc->pg_io_completion = hdr->io_completion;
        desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 19 lines
Quoted 8 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

Quoted 4 lines

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#15 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Hi Ben,

Thanks for testing these patches. More below.

Quoted 27 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

Quoted 5 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

The above issue will be fixed with the following patch which has not been merged yet:
https://www.mail-archive.com/[email protected]/msg03043.html

Do you have time to do another test to verify that is the case? If so, I can re-post that patch on top of the first 5 patches in this series, as well as a second patch that allows NFS to use it.

Quoted 654 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#16 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Thanks Dave for getting back to me so quickly.

Quoted 3 lines

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

Apologies, I completely missed the known issues in the original opening message of the series. Just to clarify, I was only ever dropping the caches on the "NFS Client" in the below relationship:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I never dropped the caches on the Re-Export Server (the server running FS-Cache) at any point.

However my rsize was lower than my readahead value. I've since corrected that:

benmaynard@demo-cluster-1-26hm:~$ cat /proc/mounts | grep nfs
10.0.0.49:/files /srv/nfs/files nfs
rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49
0 0
benmaynard@demo-cluster-1-26hm:~$ findmnt -rnu -t nfs,nfs4 -o MAJ:MIN,TARGET
0:52 /srv/nfs/files
benmaynard@demo-cluster-1-26hm:~$ cat /sys/class/bdi/0\:52/read_ahead_kb
512

With this configuration I see the same issue, FS-Cache never reads from /var/cache/fscache, and copying the same file always leads to heavy writes to /var/cache/fscache (the cache is overwriting itself).

I have also tried this copy without clearing the caches on any server in the chain, and the same happens.

Would you expect this behaviour even though rsize > read ahead? Would you expect the referenced patch to fix this?

I tried to apply the patch you suggested
(https://www.mail-archive.com/[email protected]/msg03043.html)
but it did not apply cleanly, and I ran out of time to troubleshoot. I should get some more time on Wednesday and I can re-try.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 706 lines · David Wysochanski

Hi Ben,

Thanks for testing these patches. More below.

Quoted 27 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

Quoted 5 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

The above issue will be fixed with the following patch which has not been merged yet:
https://www.mail-archive.com/[email protected]/msg03043.html

Do you have time to do another test to verify that is the case? If so, I can re-post that patch on top of the first 5 patches in this series, as well as a second patch that allows NFS to use it.

Quoted 654 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#17 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 6 lines · Benjamin Maynard

Thanks Dave for getting back to me so quickly.

Quoted 3 lines

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

Apologies, I completely missed the known issues in the original opening message of the series. Just to clarify, I was only ever dropping the caches on the "NFS Client" in the below relationship:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

What sort of server is the Source NFS server here? If it's also Linux, then what sort of filesystem is being exported?

Quoted 755 lines

I never dropped the caches on the Re-Export Server (the server running FS-Cache) at any point.

However my rsize was lower than my readahead value. I've since corrected that:

benmaynard@demo-cluster-1-26hm:~$ cat /proc/mounts | grep nfs
10.0.0.49:/files /srv/nfs/files nfs
rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49
0 0
benmaynard@demo-cluster-1-26hm:~$ findmnt -rnu -t nfs,nfs4 -o MAJ:MIN,TARGET
0:52 /srv/nfs/files
benmaynard@demo-cluster-1-26hm:~$ cat /sys/class/bdi/0\:52/read_ahead_kb
512

With this configuration I see the same issue, FS-Cache never reads from /var/cache/fscache, and copying the same file always leads to heavy writes to /var/cache/fscache (the cache is overwriting itself).

I have also tried this copy without clearing the caches on any server in the chain, and the same happens.

Would you expect this behaviour even though rsize > read ahead? Would you expect the referenced patch to fix this?

I tried to apply the patch you suggested
(https://www.mail-archive.com/[email protected]/msg03043.html)
but it did not apply cleanly, and I ran out of time to troubleshoot. I should get some more time on Wednesday and I can re-try.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 706 lines · David Wysochanski

Hi Ben,

Thanks for testing these patches. More below.

Quoted 27 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

Quoted 5 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

The above issue will be fixed with the following patch which has not been merged yet:
https://www.mail-archive.com/[email protected]/msg03043.html

Do you have time to do another test to verify that is the case? If so, I can re-post that patch on top of the first 5 patches in this series, as well as a second patch that allows NFS to use it.

Quoted 654 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

Signature · 1 line
Jeff Layton <[email protected]>
#18 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

The source server is Linux, exporting an ext4 filesystem.

benmaynard@bjmtesting-source:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
benmaynard@bjmtesting-source:~$ uname -r
5.15.0-1021-gcp
benmaynard@bjmtesting-source:~$ df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      194G  2.5G  192G   2% /
/dev/sdb1      ext4      2.0T  501G  1.4T  27% /files
benmaynard@bjmtesting-source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 776 lines · Jeff Layton
Quoted 6 lines · Benjamin Maynard

Thanks Dave for getting back to me so quickly.

Quoted 3 lines

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

Apologies, I completely missed the known issues in the original opening message of the series. Just to clarify, I was only ever dropping the caches on the "NFS Client" in the below relationship:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

What sort of server is the Source NFS server here? If it's also Linux, then what sort of filesystem is being exported?

Quoted 41 lines

I never dropped the caches on the Re-Export Server (the server running FS-Cache) at any point.

However my rsize was lower than my readahead value. I've since corrected that:

benmaynard@demo-cluster-1-26hm:~$ cat /proc/mounts | grep nfs
10.0.0.49:/files /srv/nfs/files nfs
rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49
0 0
benmaynard@demo-cluster-1-26hm:~$ findmnt -rnu -t nfs,nfs4 -o MAJ:MIN,TARGET
0:52 /srv/nfs/files
benmaynard@demo-cluster-1-26hm:~$ cat /sys/class/bdi/0\:52/read_ahead_kb
512

With this configuration I see the same issue, FS-Cache never reads from /var/cache/fscache, and copying the same file always leads to heavy writes to /var/cache/fscache (the cache is overwriting itself).

I have also tried this copy without clearing the caches on any server in the chain, and the same happens.

Would you expect this behaviour even though rsize > read ahead? Would you expect the referenced patch to fix this?

I tried to apply the patch you suggested
(https://www.mail-archive.com/[email protected]/msg03043.html)
but it did not apply cleanly, and I ran out of time to troubleshoot. I should get some more time on Wednesday and I can re-try.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

Quoted 714 lines

[email protected]

Google, Inc.

Quoted 706 lines · David Wysochanski

Hi Ben,

Thanks for testing these patches. More below.

Quoted 27 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

Quoted 5 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

The above issue will be fixed with the following patch which has not been merged yet:
https://www.mail-archive.com/[email protected]/msg03043.html

Do you have time to do another test to verify that is the case? If so, I can re-post that patch on top of the first 5 patches in this series, as well as a second patch that allows NFS to use it.

Quoted 654 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

--
Jeff Layton <[email protected]>

maintainer
#19 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Ok. You might be running into the problem of ext4 bumping the i_version on atime updates. We have some patches in flight that should make v6.2 to fix that there.

fscache uses the NFS change attribute to tell whether the cached version of the data is stale or not, so a false bump due to an atime update can cause spurious cache invalidations.

You can mount the ext4 partition with "-o noatime" to work around the issue (if it's the same problem), or try using btrfs (xfs doesn't have a fix at the moment).

That said, you mentioned earlier:

Quoted 8 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

Quoted 5 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

Why would there be any data to download from the source NFS server at all if you deleted the file on the client? The reexporting server would have unlinked the file too, and would (likely) have purged it from its cache.

Quoted 814 lines · Benjamin Maynard

The source server is Linux, exporting an ext4 filesystem.

benmaynard@bjmtesting-source:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
benmaynard@bjmtesting-source:~$ uname -r
5.15.0-1021-gcp
benmaynard@bjmtesting-source:~$ df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      194G  2.5G  192G   2% /
/dev/sdb1      ext4      2.0T  501G  1.4T  27% /files
benmaynard@bjmtesting-source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 776 lines · Jeff Layton
Quoted 6 lines · Benjamin Maynard

Thanks Dave for getting back to me so quickly.

Quoted 3 lines

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

Apologies, I completely missed the known issues in the original opening message of the series. Just to clarify, I was only ever dropping the caches on the "NFS Client" in the below relationship:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

What sort of server is the Source NFS server here? If it's also Linux, then what sort of filesystem is being exported?

Quoted 41 lines

I never dropped the caches on the Re-Export Server (the server running FS-Cache) at any point.

However my rsize was lower than my readahead value. I've since corrected that:

benmaynard@demo-cluster-1-26hm:~$ cat /proc/mounts | grep nfs
10.0.0.49:/files /srv/nfs/files nfs
rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49
0 0
benmaynard@demo-cluster-1-26hm:~$ findmnt -rnu -t nfs,nfs4 -o MAJ:MIN,TARGET
0:52 /srv/nfs/files
benmaynard@demo-cluster-1-26hm:~$ cat /sys/class/bdi/0\:52/read_ahead_kb
512

With this configuration I see the same issue, FS-Cache never reads from /var/cache/fscache, and copying the same file always leads to heavy writes to /var/cache/fscache (the cache is overwriting itself).

I have also tried this copy without clearing the caches on any server in the chain, and the same happens.

Would you expect this behaviour even though rsize > read ahead? Would you expect the referenced patch to fix this?

I tried to apply the patch you suggested
(https://www.mail-archive.com/[email protected]/msg03043.html)
but it did not apply cleanly, and I ran out of time to troubleshoot. I should get some more time on Wednesday and I can re-try.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

Quoted 714 lines

[email protected]

Google, Inc.

Quoted 706 lines · David Wysochanski

Hi Ben,

Thanks for testing these patches. More below.

Quoted 27 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

Quoted 5 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

The above issue will be fixed with the following patch which has not been merged yet:
https://www.mail-archive.com/[email protected]/msg03043.html

Do you have time to do another test to verify that is the case? If so, I can re-post that patch on top of the first 5 patches in this series, as well as a second patch that allows NFS to use it.

Quoted 654 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

--
Jeff Layton <[email protected]>

Signature · 1 line
Jeff Layton <[email protected]>
#20 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

In reply to Benjamin Maynard
Quoted 7 lines · Benjamin Maynard

Thanks Dave for getting back to me so quickly.

Quoted 3 lines

Due to use of "drop_caches" this is almost certainly the known issue #1 I mentioned in the opening post of this series:
https://lore.kernel.org/all/[email protected]/

Quoted 8 lines

Apologies, I completely missed the known issues in the original opening message of the series. Just to clarify, I was only ever dropping the caches on the "NFS Client" in the below relationship:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I never dropped the caches on the Re-Export Server (the server running FS-Cache) at any point.

So I have never actually done that particular test (I will need to verify, but I think I would have noticed by now), but dropping caches on the re-export server definitely caused repeat reads from "source" NFS server and (re)writes to the fscache disk *without* David's suggested patch. With the patch, you can drop caches on the re-export server and get the repeat reads coming from the fscache disk as expected.

You can certainly try that test too (just source NFS server -> FS-cache client - read,drop cache,read).

I'm not sure about your particular test, but your re-export server must have dropped the file from memory otherwise you would see the repeat read just coming from page cache (with no fscache or disk cache interaction required)? So I'll assume the file in question is too large to fit into memory and effectively the cache has been dropped. So I think the suggested patch on the re-export server will fix that issue.

I should also add that I had this series working well (+suggested patch) and the performance to/from disk cache is an order of magnitude better than mainline (40MB/s vs 5000MB/s with NVMe), but it did expose a race condition in the fscache use/unuse cookie code (David is aware). In the NFS re-export case, we have lots of knfsd threads thrashing around the netfs/fscache functions.

Daire

#21 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

In reply to Benjamin Maynard

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 15 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

Quoted 7 lines

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 17 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

Quoted 8 lines

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)
- exported filesystem on the NFS server (filesystem in /etc/exports)

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Quoted 629 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#22 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

maintainer
#23 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Quoted 8 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

My mistake. I didn't realize you were using v3 exclusively. The change attr doesn't exist there, so this shouldn't be a factor.

Quoted 11 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Quoted 9 lines

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

Hopefully something I am doing wrong on my end, but I can't figure out what.

I don't think you're doing anything wrong. We'll probably need to dig into why netfs/fscache decided to go to the server instead of using the cache.

It might be interesting to turn up the cachefiles_prep_read tracepoint during this and see why it's not opting to read from cache. David and David may have other tracepoints they recommend turning on too.

Quoted 778 lines · David Wysochanski
Quoted 25 lines

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

Signature · 1 line
Jeff Layton <[email protected]>
#24 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

In reply to Benjamin Maynard
Quoted 9 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

Quoted 7 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

Yes, that's very helpful. Let me think about this one as I'm not sure. As Jeff says we may need tracepoints to track it down if I cannot repro it and/or nothing comes to mind.

Quoted 10 lines

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

I'm pretty sure the above you're hitting the drop_caches / "fscache read optimisation" issue #1 I mentioned.

I see dhowells just posted a v2 version of his previous patch:
https://lore.kernel.org/linux-mm/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/

I started with 6.1-rc5, added the above dhowells latest patch for that issue, and then my 5 patches on top. Then I added a small patch to utilize dhowells patch to ensure the read optimisation is removed. I ran my unit test that has been failing all along and as expected it passes with these patches. I pushed the series to github:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f

I will also email you the series of patches on top of 6.1-rc5 so you can just apply from your mailbox if you want.

Quoted 784 lines

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#25 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Thanks Dave, that did the trick!

Building the kernel from
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f
and re-running the exact same tests yielded the expected results. Data is now being served from /var/cache/fscache.

I also reverted my change to the read ahead, so that read ahead is now greater than the rsize. Still works as expected.

I am also seeing much better single file read speeds, and culling is working perfectly (not running into the issue we were seeing pre 5.17).

Thanks a lot Dave, Jeff and Daire for your help.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 836 lines · David Wysochanski
Quoted 9 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

Quoted 7 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

Yes, that's very helpful. Let me think about this one as I'm not sure. As Jeff says we may need tracepoints to track it down if I cannot repro it and/or nothing comes to mind.

Quoted 10 lines

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

I'm pretty sure the above you're hitting the drop_caches / "fscache read optimisation" issue #1 I mentioned.

I see dhowells just posted a v2 version of his previous patch:
https://lore.kernel.org/linux-mm/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/

I started with 6.1-rc5, added the above dhowells latest patch for that issue, and then my 5 patches on top. Then I added a small patch to utilize dhowells patch to ensure the read optimisation is removed. I ran my unit test that has been failing all along and as expected it passes with these patches. I pushed the series to github:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f

I will also email you the series of patches on top of 6.1-rc5 so you can just apply from your mailbox if you want.

Quoted 784 lines

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#26 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Hi,

I just wanted to take the opportunity to reiterate why these patches are important to me (and others like Benjamin).

The "new" fscache that is now in mainline has a major NFS performance regression from the previous fscache code in pre 5.17 kernels - single file reads from cache.

Even if you have the fastest local disk (nvme/ssd) for your fscache, reading back a cached file (via NFS) now tops out at around 40MB/s whereas before (old fscache) the local fscache disk speed was the only limit (e.g. 5000MB/s for NVMe).

So, in many cases, depending on what you are using fscache for, it can be faster to read the file over the (gigabit) network than from the local disk cache which somewhat negates its usefulness. As such, we mostly use pre-5.17 kernels in production and the old fscache code which maintains high cache read performance (but has other annoying issues).

Now this performance regression might not be noticed too much by desktop users looking to use fscache on their systems, but it sure does affect servers (e.g. re-export servers) that want to use fscache to achieve very high performance.

I can't really comment on these patches or the approach taken, but I do hope that we can restore/improve the fscache read performance for NFS in the mainline kernel as soon as possible (like these patches do).

Daire

Quoted 872 lines · Benjamin Maynard

Thanks Dave, that did the trick!

Building the kernel from
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f
and re-running the exact same tests yielded the expected results. Data is now being served from /var/cache/fscache.

I also reverted my change to the read ahead, so that read ahead is now greater than the rsize. Still works as expected.

I am also seeing much better single file read speeds, and culling is working perfectly (not running into the issue we were seeing pre 5.17).

Thanks a lot Dave, Jeff and Daire for your help.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 836 lines · David Wysochanski
Quoted 9 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

Quoted 7 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

Yes, that's very helpful. Let me think about this one as I'm not sure. As Jeff says we may need tracepoints to track it down if I cannot repro it and/or nothing comes to mind.

Quoted 10 lines

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

I'm pretty sure the above you're hitting the drop_caches / "fscache read optimisation" issue #1 I mentioned.

I see dhowells just posted a v2 version of his previous patch:
https://lore.kernel.org/linux-mm/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/

I started with 6.1-rc5, added the above dhowells latest patch for that issue, and then my 5 patches on top. Then I added a small patch to utilize dhowells patch to ensure the read optimisation is removed. I ran my unit test that has been failing all along and as expected it passes with these patches. I pushed the series to github:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f

I will also email you the series of patches on top of 6.1-rc5 so you can just apply from your mailbox if you want.

Quoted 784 lines

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be filled
in by the network filesystem. For NFS, we only need to define 5 of
the functions, the main one being the issue_read() function. The issue_read() function is called by the netfs layer when a read
cannot be fulfilled locally, and must be sent to the server (either
the cache is not active, or it is active but the data is not available).
Once the read from the server is complete, netfs requires a call to
netfs_subreq_terminated() which conveys either how many bytes were
read
successfully, or an error. Note that issue_read() is called with a
structure, netfs_io_subrequest, which defines the IO requested, and
contains a start and a length (both in bytes), and assumes the underlying
netfs will return a either an error on the whole region, or the number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs defined
in pagelist.c. For the pgio APIs, there is no way for the caller to
know how many RPCs will be sent and how the pages will be broken up
into underlying RPCs, each of which will have their own completion
and
return code. In contrast, netfs is subrequest based, a single subrequest may contain multiple pages, and a single subrequest is initiated with issue_read() and terminated with netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate the netfs API requirement on the single response to the whole subrequest, while also minimizing disruptive changes to the NFS pgio layer.

The approach taken with this patch is to allocate a small structure
for each nfs_netfs_issue_read() call, store the final error and number
of bytes successfully transferred in the structure, and update these
values
as each RPC completes. The refcount on the structure is used as a
marker
for the last RPC completion, is incremented in nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a nfs_pgio_header
contains a valid pointer to the data. On the final put (which signals
the final outstanding RPC is complete) in nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value (if
one or more READs complete with an error) or the number of bytes successfully transferred (if all RPCs complete successfully). Note
that when all RPCs complete successfully, the number of bytes transferred
is capped to the length of the subrequest. Capping the transferred
length
to the subrequest length prevents "Subreq overread" warnings from netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and the
corner case where NFS requests a full page at the end of the file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#27 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Hi all,

I just wanted to follow up on this set of patches. As Daire explained below, these patches are really important to a number of us using FS-Cache due to the significant performance regression introduced in 5.17 and above.

I'd love to see these patches merged, or some feedback on what changes might be needed.

Kind Regards
Benjamin Maynard

Quoted 908 lines · Daire Byrne

Hi,

I just wanted to take the opportunity to reiterate why these patches are important to me (and others like Benjamin).

The "new" fscache that is now in mainline has a major NFS performance regression from the previous fscache code in pre 5.17 kernels - single file reads from cache.

Even if you have the fastest local disk (nvme/ssd) for your fscache, reading back a cached file (via NFS) now tops out at around 40MB/s whereas before (old fscache) the local fscache disk speed was the only limit (e.g. 5000MB/s for NVMe).

So, in many cases, depending on what you are using fscache for, it can be faster to read the file over the (gigabit) network than from the local disk cache which somewhat negates its usefulness. As such, we mostly use pre-5.17 kernels in production and the old fscache code which maintains high cache read performance (but has other annoying issues).

Now this performance regression might not be noticed too much by desktop users looking to use fscache on their systems, but it sure does affect servers (e.g. re-export servers) that want to use fscache to achieve very high performance.

I can't really comment on these patches or the approach taken, but I do hope that we can restore/improve the fscache read performance for NFS in the mainline kernel as soon as possible (like these patches do).

Daire

Quoted 872 lines · Benjamin Maynard

Thanks Dave, that did the trick!

Building the kernel from
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f
and re-running the exact same tests yielded the expected results. Data is now being served from /var/cache/fscache.

I also reverted my change to the read ahead, so that read ahead is now greater than the rsize. Still works as expected.

I am also seeing much better single file read speeds, and culling is working perfectly (not running into the issue we were seeing pre 5.17).

Thanks a lot Dave, Jeff and Daire for your help.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 836 lines · David Wysochanski
Quoted 9 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

Quoted 7 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

Yes, that's very helpful. Let me think about this one as I'm not sure. As Jeff says we may need tracepoints to track it down if I cannot repro it and/or nothing comes to mind.

Quoted 10 lines

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

I'm pretty sure the above you're hitting the drop_caches / "fscache read optimisation" issue #1 I mentioned.

I see dhowells just posted a v2 version of his previous patch:
https://lore.kernel.org/linux-mm/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/

I started with 6.1-rc5, added the above dhowells latest patch for that issue, and then my 5 patches on top. Then I added a small patch to utilize dhowells patch to ensure the read optimisation is removed. I ran my unit test that has been failing all along and as expected it passes with these patches. I pushed the series to github:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f

I will also email you the series of patches on top of 6.1-rc5 so you can just apply from your mailbox if you want.

Quoted 784 lines

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 86 lines · Trond Myklebust
Quoted 84 lines · Dave Wysochanski

Convert the NFS buffered read code paths to corresponding netfs
APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be
filled
in by the network filesystem. For NFS, we only need to define 5
of
the functions, the main one being the issue_read() function.
The issue_read() function is called by the netfs layer when a
read
cannot be fulfilled locally, and must be sent to the server
(either
the cache is not active, or it is active but the data is not
available).
Once the read from the server is complete, netfs requires a call
to
netfs_subreq_terminated() which conveys either how many bytes
were
read
successfully, or an error. Note that issue_read() is called with
a
structure, netfs_io_subrequest, which defines the IO requested,
and
contains a start and a length (both in bytes), and assumes the
underlying
netfs will return a either an error on the whole region, or the
number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs
defined
in pagelist.c. For the pgio APIs, there is no way for the caller
to
know how many RPCs will be sent and how the pages will be broken
up
into underlying RPCs, each of which will have their own
completion
and
return code. In contrast, netfs is subrequest based, a single
subrequest may contain multiple pages, and a single subrequest is
initiated with issue_read() and terminated with
netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate
the netfs API requirement on the single response to the whole
subrequest, while also minimizing disruptive changes to the NFS
pgio layer.

The approach taken with this patch is to allocate a small
structure
for each nfs_netfs_issue_read() call, store the final error and
number
of bytes successfully transferred in the structure, and update
these
values
as each RPC completes. The refcount on the structure is used as
a
marker
for the last RPC completion, is incremented in
nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a
nfs_pgio_header
contains a valid pointer to the data. On the final put (which
signals
the final outstanding RPC is complete) in
nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value
(if
one or more READs complete with an error) or the number of bytes
successfully transferred (if all RPCs complete successfully).
Note
that when all RPCs complete successfully, the number of bytes
transferred
is capped to the length of the subrequest. Capping the
transferred
length
to the subrequest length prevents "Subreq overread" warnings from
netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and
the
corner case where NFS requests a full page at the end of the
file,
even when i_size reflects only a partial page (NFS overread).

Quoted 8 lines
Quoted 2 lines

Signed-off-by: Dave Wysochanski <[email protected]>
Reviewed-by: Jeff Layton <[email protected]>

This is not doing what I asked for, which was to separate out the fscache functionality, so that we can call that if and when it is available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs requests. As
it stands, that means it is just duplicating information, and adding
unnecessary overhead to the standard I/O path (extra allocations, extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal overhead, and should certainly not increase the overhead that we already have. Will this be addressed in future iterations of these patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#28 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Hi all,

Just pinging this thread for any further updates.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 923 lines · Benjamin Maynard

Hi all,

I just wanted to follow up on this set of patches. As Daire explained below, these patches are really important to a number of us using FS-Cache due to the significant performance regression introduced in 5.17 and above.

I'd love to see these patches merged, or some feedback on what changes might be needed.

Kind Regards
Benjamin Maynard

Quoted 908 lines · Daire Byrne

Hi,

I just wanted to take the opportunity to reiterate why these patches are important to me (and others like Benjamin).

The "new" fscache that is now in mainline has a major NFS performance regression from the previous fscache code in pre 5.17 kernels - single file reads from cache.

Even if you have the fastest local disk (nvme/ssd) for your fscache, reading back a cached file (via NFS) now tops out at around 40MB/s whereas before (old fscache) the local fscache disk speed was the only limit (e.g. 5000MB/s for NVMe).

So, in many cases, depending on what you are using fscache for, it can be faster to read the file over the (gigabit) network than from the local disk cache which somewhat negates its usefulness. As such, we mostly use pre-5.17 kernels in production and the old fscache code which maintains high cache read performance (but has other annoying issues).

Now this performance regression might not be noticed too much by desktop users looking to use fscache on their systems, but it sure does affect servers (e.g. re-export servers) that want to use fscache to achieve very high performance.

I can't really comment on these patches or the approach taken, but I do hope that we can restore/improve the fscache read performance for NFS in the mainline kernel as soon as possible (like these patches do).

Daire

Quoted 872 lines · Benjamin Maynard

Thanks Dave, that did the trick!

Building the kernel from
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f
and re-running the exact same tests yielded the expected results. Data is now being served from /var/cache/fscache.

I also reverted my change to the read ahead, so that read ahead is now greater than the rsize. Still works as expected.

I am also seeing much better single file read speeds, and culling is working perfectly (not running into the issue we were seeing pre 5.17).

Thanks a lot Dave, Jeff and Daire for your help.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 836 lines · David Wysochanski
Quoted 9 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

Quoted 7 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

Yes, that's very helpful. Let me think about this one as I'm not sure. As Jeff says we may need tracepoints to track it down if I cannot repro it and/or nothing comes to mind.

Quoted 10 lines

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

I'm pretty sure the above you're hitting the drop_caches / "fscache read optimisation" issue #1 I mentioned.

I see dhowells just posted a v2 version of his previous patch:
https://lore.kernel.org/linux-mm/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/

I started with 6.1-rc5, added the above dhowells latest patch for that issue, and then my 5 patches on top. Then I added a small patch to utilize dhowells patch to ensure the read optimisation is removed. I ran my unit test that has been failing all along and as expected it passes with these patches. I pushed the series to github:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f

I will also email you the series of patches on top of 6.1-rc5 so you can just apply from your mailbox if you want.

Quoted 784 lines

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 119 lines · David Wysochanski
Quoted 94 lines · Trond Myklebust

On Mon, 2022-10-17 at 06:52 -0400, Dave Wysochanski wrote:
Convert the NFS buffered read code paths to corresponding netfs
APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be
filled
in by the network filesystem. For NFS, we only need to define 5
of
the functions, the main one being the issue_read() function.
The issue_read() function is called by the netfs layer when a
read
cannot be fulfilled locally, and must be sent to the server
(either
the cache is not active, or it is active but the data is not
available).
Once the read from the server is complete, netfs requires a call
to
netfs_subreq_terminated() which conveys either how many bytes
were
read
successfully, or an error. Note that issue_read() is called with
a
structure, netfs_io_subrequest, which defines the IO requested,
and
contains a start and a length (both in bytes), and assumes the
underlying
netfs will return a either an error on the whole region, or the
number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs
defined
in pagelist.c. For the pgio APIs, there is no way for the caller
to
know how many RPCs will be sent and how the pages will be broken
up
into underlying RPCs, each of which will have their own
completion
and
return code. In contrast, netfs is subrequest based, a single
subrequest may contain multiple pages, and a single subrequest is
initiated with issue_read() and terminated with
netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate
the netfs API requirement on the single response to the whole
subrequest, while also minimizing disruptive changes to the NFS
pgio layer.

The approach taken with this patch is to allocate a small
structure
for each nfs_netfs_issue_read() call, store the final error and
number
of bytes successfully transferred in the structure, and update
these
values
as each RPC completes. The refcount on the structure is used as
a
marker
for the last RPC completion, is incremented in
nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a
nfs_pgio_header
contains a valid pointer to the data. On the final put (which
signals
the final outstanding RPC is complete) in
nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value
(if
one or more READs complete with an error) or the number of bytes
successfully transferred (if all RPCs complete successfully).
Note
that when all RPCs complete successfully, the number of bytes
transferred
is capped to the length of the subrequest. Capping the
transferred
length
to the subrequest length prevents "Subreq overread" warnings from
netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and
the
corner case where NFS requests a full page at the end of the
file,
even when i_size reflects only a partial page (NFS overread).

Signed-off-by: Dave Wysochanski <[email protected]>
Reviewed-by: Jeff Layton <[email protected]>

This is not doing what I asked for, which was to separate out the
fscache functionality, so that we can call that if and when it is
available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be an opt-in feature, and it was a comment on a previous patch to Kconfig. I was proceeding the best I knew how, but let me try to get back on track.

Quoted 8 lines

Instead, it is just wrapping the NFS requests inside netfs
requests. As
it stands, that means it is just duplicating information, and
adding
unnecessary overhead to the standard I/O path (extra allocations,
extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is configured? Or when it is not? Or both? I think you're objecting when it's configured, but not enabled (we mount without 'fsc'). Am I right?

Quoted 9 lines

Also, are you objecting to the design that to use fcache we now have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied from the cache, then NFS is called back via netfs_issue_read and we use the normal NFS read pageio interface. This requires we call netfs_subreq_terminated() when all the RPCs complete, which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure you're
ok with it though, since you mention "extra bloat to the inode". Do you object to this even though it's wrapped in an #ifdef CONFIG_NFS_FSCACHE? If so, do you require no extra size be added to nfs_inode?

(quoted diff) +2 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      netfs; /* netfs context and VFS inode
 */
+#else
Quoted 9 lines

Review struct nfs_inode { · line 216

Quoted 3 lines
(quoted diff) +2 -0
      struct inode            vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and when CONFIG_NFS_FSCACHE is not set, become either a no-op or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */

(quoted diff) +2 -0
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header

*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header *hdr) {}

(quoted diff) +4 -0
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+       unlock_page(req->wb_page);
+}

static inline void nfs_fscache_release_super_cookie(struct super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want #ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c

(quoted diff) +2 -0
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
 super_block *sb)
 #ifdef CONFIG_NFS_V4_2
      nfsi->xattr_cache = NULL;
 #endif
+       nfs_netfs_inode_init(nfsi);
+
      return VFS_I(nfsi);
 }

EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically, how about the below calls to netfs from nfs_read_folio and nfs_readahead into equivalent netfs calls? So when NFS_CONFIG_FSCACHE is set, but fscache is not enabled ('fsc' not on mount), these netfs functions do immediately call netfs_alloc_request(). But I wonder if we could simply add a check to see if fscache is enabled on the mount, and skip over to satisfy what you want. Am I understanding what you want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines
(quoted diff) +4 -0
@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
 folio *folio)
      if (NFS_STALE(inode))
              goto out_unlock;
 
+       ret = nfs_netfs_read_folio(file, folio);
+       if (!ret)
+               goto out;
+
 
@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
 *ractl)
      if (NFS_STALE(inode))
              goto out;
 
Quoted 5 lines

Review void nfs_readahead(struct readahead_control · line 407

Quoted 5 lines
(quoted diff) +4 -0
+       ret = nfs_netfs_readahead(ractl);
+       if (!ret)
+               goto out;
+
 

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read path to the earlier mentioned stub functions?

(quoted diff) +0 -8
@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);
 
 static void nfs_readpage_release(struct nfs_page *req, int error)
 {
-       struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
      struct page *page = req->wb_page;
 
-       dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
 > s_id,
-               (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
-               (long long)req_offset(req));
-
      if (nfs_error_is_fatal_on_server(error) && error != -
 ETIMEDOUT)
              SetPageError(page);
-       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
-               if (PageUptodate(page))
-                       nfs_fscache_write_page(inode, page);
Quoted 9 lines

Review EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); · line 121

Quoted 5 lines
(quoted diff) +3 -2
-               unlock_page(page);
-       }
+       if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+               nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines
(quoted diff) +0 -0
      nfs_release_request(req);

}

(quoted diff) +2 -0
@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
 nfs_pgio_header *hdr)
              nfs_list_remove_request(req);
              nfs_readpage_release(req, error);
      }
+       nfs_netfs_read_completion(hdr);
+

out:

      hdr->release(hdr);
}
(quoted diff) +1 -0
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
 nfs_pgio_header *hdr,
                            struct rpc_task_setup *task_setup_data,
 int how)
 {
      rpc_ops->read_setup(hdr, msg);
+       nfs_netfs_initiate_read(hdr);
      trace_nfs_initiate_read(hdr);

}

Are you ok with these additions? Something like this would be required in the case of fscache configured and enabled, because we could have some of the data in a read in fscache, and some not. That is the reason for the netfs design, and why we need to be able to call the normal NFS read IO path (netfs calls into issue_read, and we call back via netfs_subreq_terminated)?

(quoted diff) +6 -0
@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
      struct pnfs_layout_segment *pg_lseg;
      struct nfs_io_completion *pg_io_completion;
      struct nfs_direct_req   *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *pg_netfs;
+#endif
 
@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
      const struct nfs_rw_ops *rw_ops;
      struct nfs_io_completion *io_completion;
      struct nfs_direct_req   *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       void                    *netfs;
+#endif
 
 

And these additions to pagelist.c?

Quoted 12 lines

Review void nfs_pgheader_init(struct · line 76

Quoted 9 lines
(quoted diff) +4 -0
@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
 nfs_pageio_descriptor *desc,
      hdr->good_bytes = mirror->pg_count;
      hdr->io_completion = desc->pg_io_completion;
      hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       if (desc->pg_netfs)
+               hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines
(quoted diff) +3 -0
@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
 *desc,
      desc->pg_lseg = NULL;
      desc->pg_io_completion = NULL;
      desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = NULL;
+#endif
 
 
@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
 nfs_pageio_descriptor *desc,
 
Quoted 8 lines

Review int nfs_pageio_resend(struct · line 1375

Quoted 5 lines
(quoted diff) +3 -0
      desc->pg_io_completion = hdr->io_completion;
      desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+       desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

My expectation is that the standard I/O path should have minimal
overhead, and should certainly not increase the overhead that we
already have. Will this be addressed in future iterations of these
patches?

I will do what I can to satisfy what you want, either by fixing up this patch or follow-on patches. Hopefully the above questions will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#29 lore raw

Re: [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

Ben,

Thanks for your interest and sorry for the delay.
I just replied on the v10 patchset thread.

Quoted 948 lines · Benjamin Maynard

Hi all,

Just pinging this thread for any further updates.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 923 lines · Benjamin Maynard

Hi all,

I just wanted to follow up on this set of patches. As Daire explained below, these patches are really important to a number of us using FS-Cache due to the significant performance regression introduced in 5.17 and above.

I'd love to see these patches merged, or some feedback on what changes might be needed.

Kind Regards
Benjamin Maynard

Quoted 908 lines · Daire Byrne

Hi,

I just wanted to take the opportunity to reiterate why these patches are important to me (and others like Benjamin).

The "new" fscache that is now in mainline has a major NFS performance regression from the previous fscache code in pre 5.17 kernels - single file reads from cache.

Even if you have the fastest local disk (nvme/ssd) for your fscache, reading back a cached file (via NFS) now tops out at around 40MB/s whereas before (old fscache) the local fscache disk speed was the only limit (e.g. 5000MB/s for NVMe).

So, in many cases, depending on what you are using fscache for, it can be faster to read the file over the (gigabit) network than from the local disk cache which somewhat negates its usefulness. As such, we mostly use pre-5.17 kernels in production and the old fscache code which maintains high cache read performance (but has other annoying issues).

Now this performance regression might not be noticed too much by desktop users looking to use fscache on their systems, but it sure does affect servers (e.g. re-export servers) that want to use fscache to achieve very high performance.

I can't really comment on these patches or the approach taken, but I do hope that we can restore/improve the fscache read performance for NFS in the mainline kernel as soon as possible (like these patches do).

Daire

Quoted 872 lines · Benjamin Maynard

Thanks Dave, that did the trick!

Building the kernel from
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f
and re-running the exact same tests yielded the expected results. Data is now being served from /var/cache/fscache.

I also reverted my change to the read ahead, so that read ahead is now greater than the rsize. Still works as expected.

I am also seeing much better single file read speeds, and culling is working perfectly (not running into the issue we were seeing pre 5.17).

Thanks a lot Dave, Jeff and Daire for your help.

Kind Regards
Benjamin Maynard

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 836 lines · David Wysochanski
Quoted 9 lines · Benjamin Maynard

Hi Dave,

I've added responses to your questions inline below.

I also tried adding the noatime option to the mount on the source filer as Jeff suggested, but this has not made any difference and the issue is still persisting for me.

Quoted 7 lines

I created the following diagram that explains my setup, and the exact tests I am performing:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

Hopefully this is clearer than my explanations below (let me know if you'd prefer me to share an alternative way).

Yes, that's very helpful. Let me think about this one as I'm not sure. As Jeff says we may need tracepoints to track it down if I cannot repro it and/or nothing comes to mind.

Quoted 10 lines

In order to remove the re-exporting layer of complexity, I also performed the tests without the re-export server (architecture:
https://drive.google.com/file/d/1DQKhqo_UnQ8ul-z5Iram5LpisDmkKziQ/view?usp=share_link):

Source NFS Server <-- Client (with FS-Cache)

The same is happening, I cannot get FS-Cache to serve from cache. Heavy writes, but no reads, even when the same file is copied many times.

I'm pretty sure the above you're hitting the drop_caches / "fscache read optimisation" issue #1 I mentioned.

I see dhowells just posted a v2 version of his previous patch:
https://lore.kernel.org/linux-mm/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/

I started with 6.1-rc5, added the above dhowells latest patch for that issue, and then my 5 patches on top. Then I added a small patch to utilize dhowells patch to ensure the read optimisation is removed. I ran my unit test that has been failing all along and as expected it passes with these patches. I pushed the series to github:
https://github.com/DaveWysochanskiRH/kernel/commits/nfs-fscache-netfs
https://github.com/DaveWysochanskiRH/kernel/commit/42f58f3d36d83839022dc2617bb6c2d1b09db65f

I will also email you the series of patches on top of 6.1-rc5 so you can just apply from your mailbox if you want.

Quoted 784 lines

Hopefully something I am doing wrong on my end, but I can't figure out what.

Kind Regards
Benjamin Maynard

Quoted 25 lines · David Wysochanski

I apologize I did not read carefully enough and I missed some details in your original post.
More below.

Quoted 19 lines · Benjamin Maynard

Hi all,

I've been doing some more testing with these patches, I applied all of the patches (v10 from
https://patchwork.kernel.org/project/linux-nfs/list/?series=691729)
apart from Patch 6 (the RFC patch) to version 6.0.8 of the kernel.

I have the following setup:

Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.

I have a 500Gb file on the Source NFS Server, which I am then copying to the NFS Client via the Re-Export Server.

On the first copy, I see heavy writes to /var/cache/fscache on the re-export server, and once the file copy completes I see that /var/cache/fscache is approximately 500Gb in size. All good so far.

Quoted 5 lines
Quoted 3 lines

I then deleted that file from the NFS Client, and dropped the caches just to be safe (echo 3 > /proc/sys/vm/drop_caches on the NFS Client).

If you delete the file from the NFS client, how does that not delete the file from the original NFS server?

Sorry - to be clear, I never deleted the file from the NFS mount (which I know would in turn delete it from the re-export server and the source filer).

In order to perform the performance test, I copied the file from the NFS mount on the NFS Client, to a local directory (cp /mnt/nfs/500gb.img /tmp).

When I said "I then deleted that file from the NFS Client", I meant I deleted the local copy of that file. Not the file on the mount (rm /tmp/500gb.img).

Just to also stress, I have never dropped the caches on the Re-Export Server (the one with FS-Cache) at any point in any of these tests, so I don't think this is the problem. I have only ever dropped the caches on the NFS client that is mounting the Re-Export Server.

Quoted 7 lines
Quoted 5 lines

I then performed another copy of the 500Gb file on the NFS Client, again via the Re-Export Server. What I expected would happen is that I would see heavy reads from the /var/cache/fscache volume as the file should be served from FS-Cache.

I don't understand this. When you say you "performed another copy" of what file? Wasn't the file deleted in the above step?

As above, only the local copy was deleted.

Quoted 30 lines
Quoted 4 lines

However what I actually saw was no reads whatsoever, FS-Cache seems to be ignored and the file is pulled from the Source NFS Filer again. I also see heavy writes to /var/cache/fscache, so it appears that FS-Cache is overwriting its existing cache, and never using it.

That would happen if the file was changed or re-created.

Quoted 23 lines

I only have 104Gb of memory on the Re-Export Server (with FS-Cache) so it is not possible that the file is being served from the page cache.

We saw this behaviour before on an older set of the patches when our mount between the Re-Export Server and the Source NFS Filer was using the "sync" option, but we are now using the "async" option and the same is happening.

Mount options:

Source NFS Server <-- Re-Export Server (with FS-Cache):

10.0.0.49:/files /srv/nfs/files nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.49,mountvers=3,mountport=37485,mountproto=tcp,fsc,local_lock=none,addr=10.0.0.49

Re-Export Server (with FS-Cache) <-- NFS Client:

10.0.0.3:/files /mnt/nfs nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.3,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.3

It is also worth noting this behaviour is not unique to the re-export use case. I see FS-Cache not being used with the following setup:

Quoted 6 lines
Quoted 2 lines

Source NFS Server <-- Client (with FS-Cache).

This points at something more fundamental like something missed in the test or maybe a mount option. Can you explain what test you're doing here when you say "this behavior is not unique"?

I've created the following diagram which explains the test I am performing. I think it is a little easier to follow than explaining in text. This should be viewable without any authentication:
https://drive.google.com/file/d/12Xf-9yHCKM4eMr2YGqdSAVfGcximW4OG/view?usp=sharing.

By "this behaviour is not unique to the re-export use case" I mean that the same happens if I remove the re-export server completely, and just have the following setup:

Source NFS Server <-- Client (with FS-Cache).

Quoted 2 lines

Can you show the mount options for both:
- fscache filesystem on the re-export server (/var/cache/fscache)

root@reexport:~$ mount | grep /var/cache/fscache
/dev/md127 on /var/cache/fscache type ext4
(rw,relatime,discard,nobarrier,stripe=1024)
Quoted 1 line

- exported filesystem on the NFS server (filesystem in /etc/exports)

I have tried both:

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)

and (at Jeff's suggestion):

root@source:~$ mount | grep files
/dev/sdb1 on /files type ext4 (rw,noatime)
root@source:~$ cat /etc/exports
/files 10.0.0.0/8(rw,sync,wdelay,no_root_squash,no_all_squash,no_subtree_check,sec=sys,secure,nohide)
Quoted 8 lines

Unfortunately the problem with drop_caches makes it more difficult to know when fscache is truly working. But some other unit test I have shows fscache does work with this patchset so I'm puzzled why you're not seeing it work at all.

I pinged dhowells on the drop_caches issue so maybe we can get that one sorted out soon but I'm not sure since it's part of a series and proposes changes in mm.

Just to be clear, I have never used drop_caches on the re-export server in any of these tests. I have only ever done this on the NFS Client.

Quoted 631 lines
Quoted 630 lines

Thanks,
Ben

Kind Regards

Benjamin Maynard

Customer Engineer

[email protected]

Google, Inc.

Quoted 610 lines · Trond Myklebust
Quoted 262 lines · David Wysochanski
Quoted 256 lines · David Wysochanski
Quoted 131 lines · Trond Myklebust
Quoted 128 lines · David Wysochanski

On Thu, Oct 27, 2022 at 3:16 PM Trond Myklebust <[email protected]>
wrote:

On Mon, 2022-10-17 at 06:52 -0400, Dave Wysochanski wrote:
Convert the NFS buffered read code paths to corresponding netfs
APIs,
but only when fscache is configured and enabled.

The netfs API defines struct netfs_request_ops which must be
filled
in by the network filesystem. For NFS, we only need to define 5
of
the functions, the main one being the issue_read() function.
The issue_read() function is called by the netfs layer when a
read
cannot be fulfilled locally, and must be sent to the server
(either
the cache is not active, or it is active but the data is not
available).
Once the read from the server is complete, netfs requires a call
to
netfs_subreq_terminated() which conveys either how many bytes
were
read
successfully, or an error. Note that issue_read() is called with
a
structure, netfs_io_subrequest, which defines the IO requested,
and
contains a start and a length (both in bytes), and assumes the
underlying
netfs will return a either an error on the whole region, or the
number
of bytes successfully read.

The NFS IO path is page based and the main APIs are the pgio APIs
defined
in pagelist.c. For the pgio APIs, there is no way for the caller
to
know how many RPCs will be sent and how the pages will be broken
up
into underlying RPCs, each of which will have their own
completion
and
return code. In contrast, netfs is subrequest based, a single
subrequest may contain multiple pages, and a single subrequest is
initiated with issue_read() and terminated with
netfs_subreq_terminated().
Thus, to utilze the netfs APIs, NFS needs some way to accommodate
the netfs API requirement on the single response to the whole
subrequest, while also minimizing disruptive changes to the NFS
pgio layer.

The approach taken with this patch is to allocate a small
structure
for each nfs_netfs_issue_read() call, store the final error and
number
of bytes successfully transferred in the structure, and update
these
values
as each RPC completes. The refcount on the structure is used as
a
marker
for the last RPC completion, is incremented in
nfs_netfs_read_initiate(),
and decremented inside nfs_netfs_read_completion(), when a
nfs_pgio_header
contains a valid pointer to the data. On the final put (which
signals
the final outstanding RPC is complete) in
nfs_netfs_read_completion(),
call netfs_subreq_terminated() with either the final error value
(if
one or more READs complete with an error) or the number of bytes
successfully transferred (if all RPCs complete successfully).
Note
that when all RPCs complete successfully, the number of bytes
transferred
is capped to the length of the subrequest. Capping the
transferred
length
to the subrequest length prevents "Subreq overread" warnings from
netfs.
This is due to the "aligned_len" in nfs_pageio_add_page(), and
the
corner case where NFS requests a full page at the end of the
file,
even when i_size reflects only a partial page (NFS overread).

Signed-off-by: Dave Wysochanski <[email protected]>
Reviewed-by: Jeff Layton <[email protected]>

This is not doing what I asked for, which was to separate out the
fscache functionality, so that we can call that if and when it is
available.

I must have misunderstood then.

The last feedback I have from you was that you wanted it to be
an opt-in feature, and it was a comment on a previous patch
to Kconfig. I was proceeding the best I knew how, but
let me try to get back on track.

Instead, it is just wrapping the NFS requests inside netfs
requests. As
it stands, that means it is just duplicating information, and
adding
unnecessary overhead to the standard I/O path (extra allocations,
extra
indirect calls, and extra bloat to the inode).

I think I understand what you're saying but I'm not sure. Let me
ask some clarifying questions.

Are you objecting to the code when CONFIG_NFS_FSCACHE is
configured? Or when it is not? Or both? I think you're objecting
when it's configured, but not enabled (we mount without 'fsc').
Am I right?

Also, are you objecting to the design that to use fcache we now
have to use netfs, specifically:
- call into netfs via either netfs_read_folio or netfs_readahead
- if fscache is enabled, then the IO can be satisfied from fscache
- if fscache is not enabled, or some of the IO cannot be satisfied
from the cache, then NFS is called back via netfs_issue_read
and we use the normal NFS read pageio interface. This requires
we call netfs_subreq_terminated() when all the RPCs complete,
which is the reason for the small changes to pagelist.c

Quoted 3 lines

I'm objecting to any middle layer "solution" that adds overhead to the NFS I/O paths.

Got it.

Quoted 9 lines

I'm willing to consider solutions that are specific only to the fscache use case (i.e. when the 'fsc' mount option is specified). However when I perform a normal NFS mount, and do I/O, then I don't want to see extra memory allocations, extra indirect calls and larger inode footprints.

IOW: I want the code to optimise for the case of standard NFS, not for the case of 'NFS with cachefs additions'.

I agree completely. Are you seeing extra memory allocations happen on mounts without 'fsc' or is it more a concern or how some of the patches look? We should not be calling any netfs or fscache code if 'fsc' is not on the mount and I don't see any in my testing. So either there's a misunderstanding here, or there's a bug I'm missing.

If fscache is not configured, then nfs_netfs_read_folio() and nfs_netfs_readahead() is a wrapper that returns -ENOBUFS. If it's configured but not enabled, then the checks for netfs_inode(inode)->cache should skip over any netfs code. But maybe there's a non-obvious bug you're seeing and somehow netfs is still getting called? Because I cannot see netfs getting called if 'fsc' is not on the mount in my tests.

int nfs_netfs_read_folio(struct file *file, struct folio *folio)

{
      if (!netfs_inode(folio_inode(folio))->cache)
              return -ENOBUFS;

      return netfs_read_folio(file, folio);
}

int nfs_netfs_readahead(struct readahead_control *ractl)

{
      struct inode *inode = ractl->mapping->host;

      if (!netfs_inode(inode)->cache)
              return -ENOBUFS;

      netfs_readahead(ractl);
      return 0;
}
Quoted 21 lines
Quoted 21 lines

Can you be more specific as to the portions of the patch you don't
like
so I can move it in the right direction?

This is from patch #2 which you didn't comment on. I'm not sure
you're
ok with it though, since you mention "extra bloat to the inode".
Do you object to this even though it's wrapped in an
#ifdef CONFIG_NFS_FSCACHE? If so, do you require no
extra size be added to nfs_inode?

@@ -204,9 +208,11 @@ struct nfs_inode {
__u64 write_io;
__u64 read_io;
#ifdef CONFIG_NFS_FSCACHE
- struct fscache_cookie *fscache;
-#endif
+ struct netfs_inode netfs; /* netfs context and VFS inode
*/
+#else

Quoted 9 lines
Quoted 3 lines

struct inode vfs_inode;
+#endif
+

Ideally, I'd prefer no extra size. I can live with it up to a certain point, however for now NFS is not unconditionally opting into the netfs project. If we're to ever do that, then I want to see streamlined code for the standard I/O case.

Ok and understood about standard I/O case.

I was thinking how we might not increase the size, but I don't think I can make it work.

I thought we could change to something like the below, without an embedded struct inode:

(quoted diff) +4 -2
@@ -204,9 +208,11 @@ struct nfs_inode {
      __u64 write_io;
      __u64 read_io;
 #ifdef CONFIG_NFS_FSCACHE
-       struct fscache_cookie   *fscache;
-#endif
+       struct netfs_inode      *netfs; /* netfs context and VFS inode */
+#else
      struct inode            vfs_inode;
+#endif
+
 

Then I would need to alloc/free a netfs_inode at the time of nfs_inode initiation. Unfortunately this has the issue that the NFS_I() macro cannot work, because it requires an embedded "struct inode" due to "container_of" use:

(quoted diff) +14 -0
+#ifdef CONFIG_NFS_FSCACHE
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->netfs.inode;
+}
+static inline struct nfs_inode *NFS_I(const struct inode *inode)
+{
+       return container_of(inode, struct nfs_inode, netfs.inode);
+}
+#else
+static inline struct inode *VFS_I(struct nfs_inode *nfsi)
+{
+       return &nfsi->vfs_inode;
+}

static inline struct nfs_inode *NFS_I(const struct inode *inode)

{
      return container_of(inode, struct nfs_inode, vfs_inode);
}
Quoted 8 lines
Quoted 3 lines

+#endif

Actually Trond maybe we can achieve a "0 length increase" of nfs_inode if dhowells would take a patch to modify the definition of struct netfs_inode and netfs_inode_init(), something like the WIP patch below. What do you think?

That works for me.

Quoted 322 lines

I think maybe this could be a follow-on patch and if you/dhowells think it's an ok idea I can try to work out what is needed across the tree. I thought about it more and I kinda agree that in the case for NFS where fscache is "configured but not enabled", then even though we're only adding 24 bytes to the nfs_inode each time, it will add up so it is worth at least a discussion.

include/linux/netfs.h +14 -8
index f2402ddeafbf..195714f1c355 100644
@@ -118,11 +118,7 @@ enum netfs_io_source {
 typedef void (*netfs_io_terminated_t)(void *priv, ssize_t transferred_or_error,
                                    bool was_async);
 
-/*
- * Per-inode context.  This wraps the VFS inode.
- */
-struct netfs_inode {
-       struct inode            inode;          /* The VFS inode */
+struct netfs_info {
      const struct netfs_request_ops *ops;
 #if IS_ENABLED(CONFIG_FSCACHE)
      struct fscache_cookie   *cache;
@@ -130,6 +126,14 @@ struct netfs_inode {
      loff_t                  remote_i_size;  /* Size of the remote file */
 };
 
+/*
+ * Per-inode context.  This wraps the VFS inode.
+ */
+struct netfs_inode {
+       struct inode            inode;          /* The VFS inode */
+       struct netfs_info       *netfs;         /* Rest of netfs data */
+};
+
 /*
 * Resources required to do operations on a cache.
 */
@@ -312,10 +316,12 @@ static inline struct netfs_inode
 *netfs_inode(struct inode *inode)
 static inline void netfs_inode_init(struct netfs_inode *ctx,
                                  const struct netfs_request_ops *ops)
 {
-       ctx->ops = ops;
-       ctx->remote_i_size = i_size_read(&ctx->inode);
+       ctx->netfs = kzalloc(sizeof(struct netfs_info)), GFP_KERNEL);
+       /* FIXME: Check for NULL */
+       ctx->netfs->ops = ops;
+       ctx->netfs->remote_i_size = i_size_read(&ctx->inode);
 #if IS_ENABLED(CONFIG_FSCACHE)
-       ctx->cache = NULL;
+       ctx->netfs->cache = NULL;
 #endif
 }
Quoted 266 lines
Quoted 43 lines
Quoted 43 lines

Are you ok with the stub functions which are placed in fscache.h, and
when CONFIG_NFS_FSCACHE is not set, become either a no-op
or a 1-liner (nfs_netfs_readpage_release)?

#else /* CONFIG_NFS_FSCACHE */
+static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi) {}
+static inline void nfs_netfs_initiate_read(struct nfs_pgio_header
*hdr) {}
+static inline void nfs_netfs_read_completion(struct nfs_pgio_header
*hdr) {}
+static inline void nfs_netfs_readpage_release(struct nfs_page *req)
+{
+ unlock_page(req->wb_page);
+}
static inline void nfs_fscache_release_super_cookie(struct
super_block *sb) {}
static inline void nfs_fscache_init_inode(struct inode *inode) {}

Do you object to the below? If so, then do you want
#ifdef CONFIG_NFS_FSCACHE here?

-- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2249,6 +2249,8 @@ struct inode *nfs_alloc_inode(struct
super_block *sb)
#ifdef CONFIG_NFS_V4_2
nfsi->xattr_cache = NULL;
#endif
+ nfs_netfs_inode_init(nfsi);
+
return VFS_I(nfsi);
}
EXPORT_SYMBOL_GPL(nfs_alloc_i
node);

Quoted 16 lines
Quoted 9 lines

Do you object to the changes in fs/nfs/read.c? Specifically,
how about the below calls to netfs from nfs_read_folio and
nfs_readahead into equivalent netfs calls? So when
NFS_CONFIG_FSCACHE is set, but fscache is not enabled
('fsc' not on mount), these netfs functions do immediately call
netfs_alloc_request(). But I wonder if we could simply add a
check to see if fscache is enabled on the mount, and skip
over to satisfy what you want. Am I understanding what you
want?

Quite frankly, I'd prefer that we just split out the functionality that is needed from the netfs code so that it can be optimised. However I'm not interested enough in the cachefs functionality to work on that myself. ...and as I indicated above, I might be OK with opting into the netfs project, once the overhead can be made to disappear.

Understood.

If you think it makes more sense, I can move some of the nfs_netfs_* functions into a netfs.c file as a starting point. Or that can maybe be done in a future patchset?

For now I was equating netfs and fscache together so we can move on from the much older and single-page limiting fscache interface that is likely to go away soon.

Quoted 16 lines
Quoted 16 lines

@@ -355,6 +343,10 @@ int nfs_read_folio(struct file *file, struct
folio *folio)
if (NFS_STALE(inode))
goto out_unlock;

+ ret = nfs_netfs_read_folio(file, folio);
+ if (!ret)
+ goto out;
+

@@ -405,6 +399,10 @@ void nfs_readahead(struct readahead_control
*ractl)
if (NFS_STALE(inode))
goto out;

Quoted 5 lines
Quoted 5 lines

+ ret = nfs_netfs_readahead(ractl);
+ if (!ret)
+ goto out;
+

The above wrappers should prevent any additional overhead when fscache is not enabled. As far as I know these work to avoid calling netfs when 'fsc' is not on the mount.

Quoted 22 lines
Quoted 22 lines

And how about these calls from different points in the read
path to the earlier mentioned stub functions?

@@ -110,20 +110,13 @@ EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds);

static void nfs_readpage_release(struct nfs_page *req, int error)
{
- struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
struct page *page = req->wb_page;

- dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb-
s_id,
- (unsigned long long)NFS_FILEID(inode), req->wb_bytes,
- (long long)req_offset(req));
-
if (nfs_error_is_fatal_on_server(error) && error != -
ETIMEDOUT)
SetPageError(page);
- if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) {
- if (PageUptodate(page))
- nfs_fscache_write_page(inode, page);

Quoted 9 lines
Quoted 5 lines

- unlock_page(page);
- }
+ if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
+ nfs_netfs_readpage_release(req);
+

I'm not seeing the value of wrapping unlock_page(), no... That code is going to need to change when we move it to use folios natively anyway.

Ok, how about I make it conditional on whether fscache is configured and enabled then, similar to the nfs_netfs_read_folio() and nfs_netfs_readahead()? Below is what that would look like. I could inline the code in nfs_netfs_readpage_release() if you think it would be clearer.

static void nfs_readpage_release(struct nfs_page *req, int error)

{
      struct page *page = req->wb_page;

      if (nfs_error_is_fatal_on_server(error) && error != -ETIMEDOUT)
              SetPageError(page);
      if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE))
#ifndef CONFIG_NFS_FSCACHE
              unlock_page(req->wb_page);
#else
              nfs_netfs_readpage_release(req);
#endif
      nfs_release_request(req);
}

void nfs_netfs_readpage_release(struct nfs_page *req) {
struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);

/*
* If fscache is enabled, netfs will unlock pages. */
if (netfs_inode(inode)->cache)

      return;

unlock_page(req->wb_page);
}

Quoted 51 lines
Quoted 51 lines

nfs_release_request(req);
}

@@ -177,6 +170,8 @@ static void nfs_read_completion(struct
nfs_pgio_header *hdr)
nfs_list_remove_request(req);
nfs_readpage_release(req, error);
}
+ nfs_netfs_read_completion(hdr);
+
out:
hdr->release(hdr);
}
@@ -187,6 +182,7 @@ static void nfs_initiate_read(struct
nfs_pgio_header *hdr,
struct rpc_task_setup *task_setup_data,
int how)
{
rpc_ops->read_setup(hdr, msg);
+ nfs_netfs_initiate_read(hdr);
trace_nfs_initiate_read(hdr);
}

Are you ok with these additions? Something like this would
be required in the case of fscache configured and enabled,
because we could have some of the data in a read in
fscache, and some not. That is the reason for the netfs
design, and why we need to be able to call the normal
NFS read IO path (netfs calls into issue_read, and we call
back via netfs_subreq_terminated)?

@@ -101,6 +101,9 @@ struct nfs_pageio_descriptor {
struct pnfs_layout_segment *pg_lseg;
struct nfs_io_completion *pg_io_completion;
struct nfs_direct_req *pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+ void *pg_netfs;
+#endif

@@ -1619,6 +1619,9 @@ struct nfs_pgio_header {
const struct nfs_rw_ops *rw_ops;
struct nfs_io_completion *io_completion;
struct nfs_direct_req *dreq;
+#ifdef CONFIG_NFS_FSCACHE
+ void *netfs;
+#endif

And these additions to pagelist.c?

Quoted 12 lines
Quoted 9 lines

@@ -68,6 +69,10 @@ void nfs_pgheader_init(struct
nfs_pageio_descriptor *desc,
hdr->good_bytes = mirror->pg_count;
hdr->io_completion = desc->pg_io_completion;
hdr->dreq = desc->pg_dreq;
+#ifdef CONFIG_NFS_FSCACHE
+ if (desc->pg_netfs)
+ hdr->netfs = desc->pg_netfs;
+#endif

Why the conditional?

Not really needed and I was thinking of removing it, so I'll do that.

Quoted 15 lines
Quoted 15 lines

@@ -846,6 +851,9 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
*desc,
desc->pg_lseg = NULL;
desc->pg_io_completion = NULL;
desc->pg_dreq = NULL;
+#ifdef CONFIG_NFS_FSCACHE
+ desc->pg_netfs = NULL;
+#endif

@@ -1360,6 +1369,9 @@ int nfs_pageio_resend(struct
nfs_pageio_descriptor *desc,

Quoted 8 lines
Quoted 5 lines

desc->pg_io_completion = hdr->io_completion;
desc->pg_dreq = hdr->dreq;
+#ifdef CONFIG_NFS_FSCACHE
+ desc->pg_netfs = hdr->netfs;
+#endif

Those all need wrapper functions instead of embedding #ifdefs.

Ok.

Quoted 13 lines
Quoted 12 lines

My expectation is that the standard I/O path should have minimal
overhead, and should certainly not increase the overhead that we
already have. Will this be addressed in future iterations of these
patches?

I will do what I can to satisfy what you want, either by fixing up
this patch or follow-on patches. Hopefully the above questions
will clarify the next steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Trond Myklebust
CTO, Hammerspace Inc
1900 S Norfolk St, Suite 350 - #45
San Mateo, CA 94403

www.hammer.space

#30 lore raw

PATCH v9 4/5 NFS: Remove all NFSIOS_FSCACHE counters due to conversion to netfs API

In reply to Dave Wysochanski

The old NFSIOS_FSCACHE counters are no longer accurate or useful with the conversion to the new netfs API. The new API does not have a page based interface, and so the counters in nfs_stat_fscachecounters are no longer obtainable. The new netfs the API has extensive statistics inside /proc/fs/fscache/stats so we no longer need NFS specific fscache stats.

Note this also removes the 'fsc:' line from /proc/self/mountstats so it will be a user-visible change.

---
 fs/nfs/iostat.h            | 17 -----------------
 fs/nfs/super.c             | 11 -----------
 include/linux/nfs_iostat.h | 12 ------------
 3 files changed, 40 deletions(-)
fs/nfs/iostat.h +0 -17
index 2ddaab1ac653..5aa776b5a3e7 100644
@@ -17,9 +17,6 @@
 
 struct nfs_iostats {
 	unsigned long long	bytes[__NFSIOS_BYTESMAX];
-#ifdef CONFIG_NFS_FSCACHE
-	unsigned long long	fscache[__NFSIOS_FSCACHEMAX];
-#endif
 	unsigned long		events[__NFSIOS_COUNTSMAX];
 } ____cacheline_aligned;
 
@@ -49,20 +46,6 @@ static inline void nfs_add_stats(const struct inode *inode,
 	nfs_add_server_stats(NFS_SERVER(inode), stat, addend);
 }
 
-#ifdef CONFIG_NFS_FSCACHE
-static inline void nfs_add_fscache_stats(struct inode *inode,
-					 enum nfs_stat_fscachecounters stat,
-					 long addend)
-{
-	this_cpu_add(NFS_SERVER(inode)->io_stats->fscache[stat], addend);
-}
-static inline void nfs_inc_fscache_stats(struct inode *inode,
-					 enum nfs_stat_fscachecounters stat)
-{
-	this_cpu_inc(NFS_SERVER(inode)->io_stats->fscache[stat]);
-}
-#endif
-
 static inline struct nfs_iostats __percpu *nfs_alloc_iostats(void)
 {
 	return alloc_percpu(struct nfs_iostats);
fs/nfs/super.c +0 -11
index ee66ffdb985e..302148258ff1 100644
@@ -692,10 +692,6 @@ int nfs_show_stats(struct seq_file *m, struct dentry *root)
 			totals.events[i] += stats->events[i];
 		for (i = 0; i < __NFSIOS_BYTESMAX; i++)
 			totals.bytes[i] += stats->bytes[i];
-#ifdef CONFIG_NFS_FSCACHE
-		for (i = 0; i < __NFSIOS_FSCACHEMAX; i++)
-			totals.fscache[i] += stats->fscache[i];
-#endif
 
 		preempt_enable();
 	}
@@ -706,13 +702,6 @@ int nfs_show_stats(struct seq_file *m, struct dentry *root)
 	seq_puts(m, "\n\tbytes:\t");
 	for (i = 0; i < __NFSIOS_BYTESMAX; i++)
 		seq_printf(m, "%Lu ", totals.bytes[i]);
-#ifdef CONFIG_NFS_FSCACHE
-	if (nfss->options & NFS_OPTION_FSCACHE) {
-		seq_puts(m, "\n\tfsc:\t");
-		for (i = 0; i < __NFSIOS_FSCACHEMAX; i++)
-			seq_printf(m, "%Lu ", totals.fscache[i]);
-	}
-#endif
 	seq_putc(m, '\n');
 
 	rpc_clnt_show_stats(m, nfss->client);
include/linux/nfs_iostat.h +0 -12
index 027874c36c88..8d946089d151 100644
@@ -119,16 +119,4 @@ enum nfs_stat_eventcounters {
 	__NFSIOS_COUNTSMAX,
 };
 
-/*
- * NFS local caching servicing counters
- */
-enum nfs_stat_fscachecounters {
-	NFSIOS_FSCACHE_PAGES_READ_OK,
-	NFSIOS_FSCACHE_PAGES_READ_FAIL,
-	NFSIOS_FSCACHE_PAGES_WRITTEN_OK,
-	NFSIOS_FSCACHE_PAGES_WRITTEN_FAIL,
-	NFSIOS_FSCACHE_PAGES_UNCACHED,
-	__NFSIOS_FSCACHEMAX,
-};
-
 #endif	/* _LINUX_NFS_IOSTAT */
Signature · 1 line
2.31.1
#31 lore raw

PATCH v9 5/5 NFS: Remove fscache specific trace points and NFS_INO_FSCACHE bit

In reply to Dave Wysochanski

The NFS specific trace points are no longer needed as tracing is well covered by netfs and fscache.

---
 fs/nfs/nfstrace.h      | 91 ------------------------------------------
 include/linux/nfs_fs.h |  1 -
 2 files changed, 92 deletions(-)
fs/nfs/nfstrace.h +0 -91
index 8c6cc58679ff..6b56abe49ec2 100644
@@ -39,7 +39,6 @@
 			{ BIT(NFS_INO_STALE), "STALE" }, \
 			{ BIT(NFS_INO_ACL_LRU_SET), "ACL_LRU_SET" }, \
 			{ BIT(NFS_INO_INVALIDATING), "INVALIDATING" }, \
-			{ BIT(NFS_INO_FSCACHE), "FSCACHE" }, \
 			{ BIT(NFS_INO_LAYOUTCOMMIT), "NEED_LAYOUTCOMMIT" }, \
 			{ BIT(NFS_INO_LAYOUTCOMMITTING), "LAYOUTCOMMIT" }, \
 			{ BIT(NFS_INO_LAYOUTSTATS), "LAYOUTSTATS" }, \
@@ -1213,96 +1212,6 @@ TRACE_EVENT(nfs_readpage_short,
 		)
 );
 
-DECLARE_EVENT_CLASS(nfs_fscache_page_event,
-		TP_PROTO(
-			const struct inode *inode,
-			struct page *page
-		),
-
-		TP_ARGS(inode, page),
-
-		TP_STRUCT__entry(
-			__field(dev_t, dev)
-			__field(u32, fhandle)
-			__field(u64, fileid)
-			__field(loff_t, offset)
-		),
-
-		TP_fast_assign(
-			const struct nfs_inode *nfsi = NFS_I(inode);
-			const struct nfs_fh *fh = &nfsi->fh;
-
-			__entry->offset = page_index(page) << PAGE_SHIFT;
-			__entry->dev = inode->i_sb->s_dev;
-			__entry->fileid = nfsi->fileid;
-			__entry->fhandle = nfs_fhandle_hash(fh);
-		),
-
-		TP_printk(
-			"fileid=%02x:%02x:%llu fhandle=0x%08x "
-			"offset=%lld",
-			MAJOR(__entry->dev), MINOR(__entry->dev),
-			(unsigned long long)__entry->fileid,
-			__entry->fhandle,
-			(long long)__entry->offset
-		)
-);
-DECLARE_EVENT_CLASS(nfs_fscache_page_event_done,
-		TP_PROTO(
-			const struct inode *inode,
-			struct page *page,
-			int error
-		),
-
-		TP_ARGS(inode, page, error),
-
-		TP_STRUCT__entry(
-			__field(int, error)
-			__field(dev_t, dev)
-			__field(u32, fhandle)
-			__field(u64, fileid)
-			__field(loff_t, offset)
-		),
-
-		TP_fast_assign(
-			const struct nfs_inode *nfsi = NFS_I(inode);
-			const struct nfs_fh *fh = &nfsi->fh;
-
-			__entry->offset = page_index(page) << PAGE_SHIFT;
-			__entry->dev = inode->i_sb->s_dev;
-			__entry->fileid = nfsi->fileid;
-			__entry->fhandle = nfs_fhandle_hash(fh);
-			__entry->error = error;
-		),
-
-		TP_printk(
-			"fileid=%02x:%02x:%llu fhandle=0x%08x "
-			"offset=%lld error=%d",
-			MAJOR(__entry->dev), MINOR(__entry->dev),
-			(unsigned long long)__entry->fileid,
-			__entry->fhandle,
-			(long long)__entry->offset, __entry->error
-		)
-);
-#define DEFINE_NFS_FSCACHE_PAGE_EVENT(name) \
-	DEFINE_EVENT(nfs_fscache_page_event, name, \
-			TP_PROTO( \
-				const struct inode *inode, \
-				struct page *page \
-			), \
-			TP_ARGS(inode, page))
-#define DEFINE_NFS_FSCACHE_PAGE_EVENT_DONE(name) \
-	DEFINE_EVENT(nfs_fscache_page_event_done, name, \
-			TP_PROTO( \
-				const struct inode *inode, \
-				struct page *page, \
-				int error \
-			), \
-			TP_ARGS(inode, page, error))
-DEFINE_NFS_FSCACHE_PAGE_EVENT(nfs_fscache_read_page);
-DEFINE_NFS_FSCACHE_PAGE_EVENT_DONE(nfs_fscache_read_page_exit);
-DEFINE_NFS_FSCACHE_PAGE_EVENT(nfs_fscache_write_page);
-DEFINE_NFS_FSCACHE_PAGE_EVENT_DONE(nfs_fscache_write_page_exit);
 
 TRACE_EVENT(nfs_pgio_error,
 	TP_PROTO(
include/linux/nfs_fs.h +0 -1
index a1c402e26abf..0150a5673419 100644
@@ -281,7 +281,6 @@ struct nfs4_copy_state {
 #define NFS_INO_ACL_LRU_SET	(2)		/* Inode is on the LRU list */
 #define NFS_INO_INVALIDATING	(3)		/* inode is being invalidated */
 #define NFS_INO_PRESERVE_UNLINKED (4)		/* preserve file if removed while open */
-#define NFS_INO_FSCACHE		(5)		/* inode can be cached by FS-Cache */
 #define NFS_INO_LAYOUTCOMMIT	(9)		/* layoutcommit required */
 #define NFS_INO_LAYOUTCOMMITTING (10)		/* layoutcommit inflight */
 #define NFS_INO_LAYOUTSTATS	(11)		/* layoutstats inflight */
Signature · 1 line
2.31.1