tmpfs: fix regressions from wider use of ZERO_PAGE
Subsystems (6)
Hugh Dickins
maintainerPATCH 1/1 tmpfs: fix regressions from wider use of ZERO_PAGE
Chuck Lever reported fsx-based xfstests generic 075 091 112 127 failing when 5.18-rc1 NFS server exports tmpfs: bisected to recent tmpfs change.
Whilst nfsd_splice_action() does contain some questionable handling of repeated pages, and Chuck was able to work around there, history from Mark Hemment makes clear that there might be similar dangers elsewhere:
it was not a good idea for me to pass ZERO_PAGE down to unknown actors.
Revert shmem_file_read_iter() to using ZERO_PAGE for holes only when iter_is_iovec(); in other cases, use the more natural iov_iter_zero() instead of copy_page_to_iter(). We would use iov_iter_zero() throughout, but the x86 clear_user() is not nearly so well optimized as copy to user (dd of 1T sparse tmpfs file takes 57 seconds rather than 44 seconds).
And now pagecache_init() does not need to SetPageUptodate(ZERO_PAGE(0)):
which had caused boot failure on arm noMMU STM32F7 and STM32H7 boards
---
mm/filemap.c | 6 ------ mm/shmem.c | 31 ++++++++++++++++++++----------- 2 files changed, 20 insertions(+), 17 deletions(-)
mm/filemap.c +0 -6
@@ -1063,12 +1063,6 @@ void __init pagecache_init(void)init_waitqueue_head (&folio_wait_table[i]);page_writeback_init (); - -/* -* tmpfs uses the ZERO_PAGE for reading holes: it is up-to-date, -* and splice's page_cache_pipe_buf_confirm() needs to see that. -*/ -SetPageUptodate (ZERO_PAGE (0 )); }/*
mm/shmem.c +20 -11
@@ -2513,7 +2513,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)pgoff_t end_index;unsigned long nr, ret;loff_t i_size =i_size_read (inode); -bool got_page; end_index = i_size >>PAGE_SHIFT ;if (index > end_index)
@@ -2570,24 +2569,34 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)*/ if (!offset)mark_page_accessed (page); - got_page = true; +/* +* Ok, we have the page, and it's up-to-date, so +* now we can copy it to user space... +*/ + ret =copy_page_to_iter (page, offset, nr, to); +put_page (page); + + }else if (iter_is_iovec (to)) { +/* +* Copy to user tends to be so well optimized, but +* clear_user() not so much, that it is noticeably +* faster to copy the zero page instead of clearing. +*/ + ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); }else { - page =ZERO_PAGE (0 ); - got_page = false; +/* +* But submitting the same page twice in a row to +* splice() - or others? - can result in confusion: +* so don't attempt that optimization on pipes etc. +*/ + ret =iov_iter_zero (nr, to); } -/* -* Ok, we have the page, and it's up-to-date, so -* now we can copy it to user space... -*/ - ret =copy_page_to_iter (page, offset, nr, to); retval += ret; offset += ret; index += offset >>PAGE_SHIFT ; offset &= ~PAGE_MASK ; -if (got_page) -put_page (page);if (!iov_iter_count (to))break ;if (ret < nr) {
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 2585
Quoted 7 lines · Hugh Dickins
mm/shmem.c +7
-0+ }else if (iter_is_iovec (to)) { +/* +* Copy to user tends to be so well optimized, but +* clear_user() not so much, that it is noticeably +* faster to copy the zero page instead of clearing. +*/ + ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to);
Is the offset and length guaranteed to be less than PAGE_SIZE here?
Either way I'd rather do this optimization in iov_iter_zero rather than hiding it in tmpfs.
Hugh Dickins
maintainerQuoted 10 lines · Christoph Hellwig
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 2585
Quoted 7 lines · Hugh Dickins
mm/shmem.c +7
-0+ }else if (iter_is_iovec (to)) { +/* +* Copy to user tends to be so well optimized, but +* clear_user() not so much, that it is noticeably +* faster to copy the zero page instead of clearing. +*/ + ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to);Is the offset and length guaranteed to be less than
PAGE_SIZEhere?
Almost :) The offset is guaranteed to be less than PAGE_SIZE here, and the length is guaranteed to be less than or equal to PAGE_SIZE - offset.
Quoted 3 lines
Either way I'd rather do this optimization in
iov_iter_zerorather than hiding it in tmpfs.
Let's see what others say. I think we would all prefer clear_user() to be enhanced, and hack around it neither here in tmpfs nor in iov_iter_zero(). But that careful work won't get done by magic, nor by me.
And iov_iter_zero() has to deal with a wider range of possibilities, when pulling in cache lines of ZERO_PAGE(0) will be less advantageous, than in tmpfs doing a large dd - the case I'm aiming not to regress here (tmpfs has been copying ZERO_PAGE(0) like this for years).
Hugh
making x86 clear_user not suck, was Re: [PATCH] tmpfs: fix regressions from wider use of ZERO_PAGE
Quoted 7 lines · Hugh Dickins
Quoted 3 lines
Either way I'd rather do this optimization in
iov_iter_zerorather than hiding it in tmpfs.Let's see what others say. I think we would all prefer
clear_user()to be enhanced, and hack around it neither here in tmpfs nor iniov_iter_zero(). But that careful work won't get done by magic, nor by me.
I agree with that.
Quoted 4 lines
And
iov_iter_zero()has to deal with a wider range of possibilities, when pulling in cache lines ofZERO_PAGE(0)will be less advantageous, than in tmpfs doing a large dd - the case I'm aiming not to regress here (tmpfs has been copyingZERO_PAGE(0)like this for years).
Maybe. OTOH I'd hate to have iov_iter_zero not used much because it sucks too much.
So how can we entice someone with the right knowledge to implement a decent clear_user for x86?
Matthew Wilcox
maintainerRe: making x86 clear_user not suck, was Re: [PATCH] tmpfs: fix regressions from wider use of ZERO_PAGE
Quoted 16 lines · Christoph Hellwig
Quoted 7 lines · Hugh Dickins
Quoted 3 lines
Either way I'd rather do this optimization in
iov_iter_zerorather than hiding it in tmpfs.Let's see what others say. I think we would all prefer
clear_user()to be enhanced, and hack around it neither here in tmpfs nor iniov_iter_zero(). But that careful work won't get done by magic, nor by me.I agree with that.
Quoted 4 lines
And
iov_iter_zero()has to deal with a wider range of possibilities, when pulling in cache lines ofZERO_PAGE(0)will be less advantageous, than in tmpfs doing a large dd - the case I'm aiming not to regress here (tmpfs has been copyingZERO_PAGE(0)like this for years).
Quoted 5 lines
Maybe. OTOH I'd hate to have
iov_iter_zeronot used much because it sucks too much.So how can we entice someone with the right knowledge to implement a decent
clear_userfor x86?
Apparently that already happened, but it needs finishing up:
https://lore.kernel.org/lkml/[email protected]/
Andrew Morton
maintainerQuoted 4 lines · On Fri, 8 Apr 2022 23:08:29 -0700 (PDT) Hugh Dickins <[email protected]> wrote:
Quoted 3 lines
Either way I'd rather do this optimization in
iov_iter_zerorather than hiding it in tmpfs.
Quoted 8 lines
Let's see what others say. I think we would all prefer
clear_user()to be enhanced, and hack around it neither here in tmpfs nor iniov_iter_zero(). But that careful work won't get done by magic, nor by me.And
iov_iter_zero()has to deal with a wider range of possibilities, when pulling in cache lines ofZERO_PAGE(0)will be less advantageous, than in tmpfs doing a large dd - the case I'm aiming not to regress here (tmpfs has been copyingZERO_PAGE(0)like this for years).
We do need something to get 5.18 fixed. Christoph, do you think we should proceed with this patch for 5.18?
Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Fri, 8 Apr 2022 23:08:29 -0700 (PDT) Hugh Dickins <[email protected]> wrote:
Quoted 3 lines
Either way I'd rather do this optimization in
iov_iter_zerorather than hiding it in tmpfs.Let's see what others say. I think we would all prefer
clear_user()to be enhanced, and hack around it neither here in tmpfs nor iniov_iter_zero(). But that careful work won't get done by magic, nor by me.
Quoted 7 lines
Quoted 4 lines
And
iov_iter_zero()has to deal with a wider range of possibilities, when pulling in cache lines ofZERO_PAGE(0)will be less advantageous, than in tmpfs doing a large dd - the case I'm aiming not to regress here (tmpfs has been copyingZERO_PAGE(0)like this for years).We do need something to get 5.18 fixed. Christoph, do you think we should proceed with this patch for 5.18?
Well, let's queue it up then.