Skip to contentSkip to conversation

Large pages in the page cache

PATCH Superseded Matthew Wilcox 3 participants 68 messages Feb 12, 2020 – Feb 21, 2020

Subsystems (16)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Matthew Wilcox

author maintainer
#0 lore raw

v2 cover Large pages in the page cache

From: "Matthew Wilcox (Oracle)" <[email protected]>

This patch set does not pass xfstests. Test at your own risk. It is based on the readahead patchset which I posted yesterday.

The principal idea here is that a large part of the overhead in dealing with individual pages is that there's just so darned many of them. We would be better off dealing with fewer, larger pages, even if they don't get to be the size necessary for the CPU to use a larger TLB entry.

The first five patches are more or less random cleanups which I came across while working on this patchset ... Andrew, if you want to just take those into your tree, it'd probably be a good thing.

hpage_nr_pages() is adapted to handle arbitrary order pages. I also add thp_order() and thp_size() for legibility. Then the patches tear through the page cache fixing the places which assume pages are either PMD_SIZE or PAGE_SIZE. After that, I tackle the iomap buffered I/O path, removing the assumptions of PAGE_SIZE there.

Finally, we get to actually allocating large pages in the readahead code. We gradually grow the page size that is allocated, so we don't just jump straight from order-0 to order-9 pages, but gradually get there through order-2, order-4, order-6, order-8 and order-9 (on x86; other architectures will have a different PMD_ORDER).

In some testing, I've seen the code go as far as order-6. Right now it falls over on an earlier xfstest when it discovers a delayed allocation extent in an inode which is being removed at unmount.

Matthew Wilcox (Oracle) (24):
mm: Use vm_fault error code directly
mm: Optimise find_subpage for !THP
mm: Use VM_BUG_ON_PAGE in clear_page_dirty_for_io
mm: Unexport find_get_entry
mm: Fix documentation of FGP flags
mm: Allow hpages to be arbitrary order
mm: Introduce thp_size
mm: Introduce thp_order
fs: Add a filesystem flag for large pages
fs: Introduce i_blocks_per_page
fs: Make page_mkwrite_check_truncate thp-aware
mm: Add file_offset_of_ helpers
fs: Add zero_user_large
iomap: Support arbitrarily many blocks per page
iomap: Support large pages in iomap_adjust_read_range
iomap: Support large pages in read paths
iomap: Support large pages in write paths
iomap: Inline data shouldn't see large pages
xfs: Support large pages
mm: Make prep_transhuge_page return its argument
mm: Add __page_cache_alloc_order
mm: Allow large pages to be added to the page cache
mm: Allow large pages to be removed from the page cache
mm: Add large page readahead

William Kucharski (1):
mm: Align THP mappings for non-DAX

 drivers/net/ethernet/ibm/ibmveth.c |   2 -
 drivers/nvdimm/btt.c               |   4 +-
 drivers/nvdimm/pmem.c              |   3 +-
 fs/iomap/buffered-io.c             | 111 ++++++++++++++++-------------
 fs/jfs/jfs_metapage.c              |   2 +-
 fs/xfs/xfs_aops.c                  |   4 +-
 fs/xfs/xfs_super.c                 |   2 +-
 include/linux/fs.h                 |   1 +
 include/linux/highmem.h            |  22 ++++++
 include/linux/huge_mm.h            |  21 +++---
 include/linux/mm.h                 |   2 +
 include/linux/pagemap.h            |  78 ++++++++++++++++----
 mm/filemap.c                       |  67 +++++++++++------
 mm/huge_memory.c                   |  14 ++--
 mm/internal.h                      |   2 +-
 mm/page-writeback.c                |   2 +-
 mm/page_io.c                       |   2 +-
 mm/page_vma_mapped.c               |   4 +-
 mm/readahead.c                     |  98 +++++++++++++++++++++++--
 19 files changed, 322 insertions(+), 119 deletions(-)
Signature · 1 line
2.25.0
author maintainer
#1 lore raw

PATCH v2 1/25 mm: Use vm_fault error code directly

From: "Matthew Wilcox (Oracle)" <[email protected]>

Use VM_FAULT_OOM instead of indirecting through vmf_error(-ENOMEM).

---
 mm/filemap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/filemap.c +1 -1
index 1784478270e1..1beb7716276b 100644
@@ -2491,7 +2491,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 		if (!page) {
 			if (fpin)
 				goto out_retry;
-			return vmf_error(-ENOMEM);
+			return VM_FAULT_OOM;
 		}
 	}
 
Signature · 1 line
2.25.0
author maintainer
#3 lore raw

PATCH v2 2/25 mm: Optimise find_subpage for !THP

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If THP is disabled, find_subpage can become a no-op.

---
 include/linux/pagemap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/pagemap.h +1 -1
index 75bdfec49710..0842622cca90 100644
@@ -340,7 +340,7 @@ static inline struct page *find_subpage(struct page *page, pgoff_t offset)
 
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
-	return page + (offset & (compound_nr(page) - 1));
+	return page + (offset & (hpage_nr_pages(page) - 1));
 }
 
 struct page *find_get_entry(struct address_space *mapping, pgoff_t offset);
Signature · 1 line
2.25.0
#4 lore raw

Re: [PATCH v2 02/25] mm: Optimise find_subpage for !THP

Quoted 14 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If THP is disabled, find_subpage can become a no-op.

---
 include/linux/pagemap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/pagemap.h +0 -0
index 75bdfec49710..0842622cca90 100644

include/linux/pagemap.h static inline struct page *find_subpage(struct page *page, pgoff_t offset) · line 344

Quoted 6 lines
include/linux/pagemap.h +1 -1
@@ -340,7 +340,7 @@ static inline struct page *find_subpage(struct page *page, pgoff_t offset)
 
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
-	return page + (offset & (compound_nr(page) - 1));
+	return page + (offset & (hpage_nr_pages(page) - 1));
 }

So just above the quoted code is a

	if (PageHuge(page))
		return page;

So how can we get into a compound page that is not a huge page, but only if THP is enabled? (Yes, maybe I'm a little rusty on VM internals).

Can you add comments describing the use case of this function and why it does all these checks? It looks like black magic to me.

author maintainer
#5 lore raw

Re: [PATCH v2 02/25] mm: Optimise find_subpage for !THP

Quoted 24 lines · Christoph Hellwig
Quoted 14 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If THP is disabled, find_subpage can become a no-op.

---
 include/linux/pagemap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/pagemap.h +0 -0
index 75bdfec49710..0842622cca90 100644

include/linux/pagemap.h static inline struct page *find_subpage(struct page *page, pgoff_t offset) · line 344

Quoted 6 lines
include/linux/pagemap.h +1 -1
@@ -340,7 +340,7 @@ static inline struct page *find_subpage(struct page *page, pgoff_t offset)
 
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
-	return page + (offset & (compound_nr(page) - 1));
+	return page + (offset & (hpage_nr_pages(page) - 1));
 }

So just above the quoted code is a

Quoted 6 lines
	if (PageHuge(page))
		return page;

So how can we get into a compound page that is not a huge page, but only if THP is enabled? (Yes, maybe I'm a little rusty on VM internals).

That's for hugetlbfs:

        if (!PageCompound(page))
                return 0;

        page = compound_head(page);
        return page[1].compound_dtor == HUGETLB_PAGE_DTOR;
Quoted 2 lines

Can you add comments describing the use case of this function and why it does all these checks? It looks like black magic to me.

Would this help?

-static inline struct page *find_subpage(struct page *page, pgoff_t offset) +/*
+ * Given the page we found in the page cache, return the page corresponding
+ * to this offset in the file
+ */
+static inline struct page *find_subpage(struct page *head, pgoff_t offset)
{
- if (PageHuge(page))
- return page;
+ /* HugeTLBfs wants the head page regardless */
+ if (PageHuge(head))
+ return head;

- VM_BUG_ON_PAGE(PageTail(page), page);
+ VM_BUG_ON_PAGE(PageTail(head), head);

- return page + (offset & (hpage_nr_pages(page) - 1));
+ return head + (offset & (hpage_nr_pages(head) - 1));
}

#6 lore raw

Re: [PATCH v2 02/25] mm: Optimise find_subpage for !THP

Quoted 13 lines · Matthew Wilcox
Quoted 2 lines

Can you add comments describing the use case of this function and why it does all these checks? It looks like black magic to me.

Would this help?

(quoted diff) +5 -2
-static inline struct page *find_subpage(struct page *page, pgoff_t offset)
+/*
+ * Given the page we found in the page cache, return the page corresponding
+ * to this offset in the file
+ */
+static inline struct page *find_subpage(struct page *head, pgoff_t offset)
 {
-       if (PageHuge(page))

Review

Quoted 10 lines
(quoted diff) +5 -3
-               return page;
+       /* HugeTLBfs wants the head page regardless */
+       if (PageHuge(head))
+               return head;
 
-       VM_BUG_ON_PAGE(PageTail(page), page);
+       VM_BUG_ON_PAGE(PageTail(head), head);
 
-       return page + (offset & (hpage_nr_pages(page) - 1));
+       return head + (offset & (hpage_nr_pages(head) - 1));

Much better.

author maintainer
#8 lore raw

PATCH v2 3/25 mm: Use VM_BUG_ON_PAGE in clear_page_dirty_for_io

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Dumping the page information in this circumstance helps for debugging.

---
 mm/page-writeback.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page-writeback.c +1 -1
index 2caf780a42e7..9173c25cf8e6 100644
@@ -2655,7 +2655,7 @@ int clear_page_dirty_for_io(struct page *page)
 	struct address_space *mapping = page_mapping(page);
 	int ret = 0;
 
-	BUG_ON(!PageLocked(page));
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
 
 	if (mapping && mapping_cap_account_dirty(mapping)) {
 		struct inode *inode = mapping->host;
Signature · 1 line
2.25.0
author maintainer
#11 lore raw

PATCH v2 4/25 mm: Unexport find_get_entry

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

No in-tree users (proc, madvise, memcg, mincore) can be built as a module.

---
 mm/filemap.c | 1 -
 1 file changed, 1 deletion(-)
mm/filemap.c +0 -1
index 1beb7716276b..83ce9ce0bee1 100644
@@ -1536,7 +1536,6 @@ struct page *find_get_entry(struct address_space *mapping, pgoff_t offset)
 
 	return page;
 }
-EXPORT_SYMBOL(find_get_entry);
 
 /**
  * find_lock_entry - locate, pin and lock a page cache entry
Signature · 1 line
2.25.0
author maintainer
#14 lore raw

PATCH v2 5/25 mm: Fix documentation of FGP flags

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We never had PCG flags; they've been called FGP flags since their introduction in 2014.

---
 mm/filemap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +2 -2
index 83ce9ce0bee1..3204293f9b58 100644
@@ -1577,12 +1577,12 @@ EXPORT_SYMBOL(find_lock_entry);
  * pagecache_get_page - find and get a page reference
  * @mapping: the address_space to search
  * @offset: the page index
- * @fgp_flags: PCG flags
+ * @fgp_flags: FGP flags
  * @gfp_mask: gfp mask to use for the page cache data page allocation
  *
  * Looks up the page cache slot at @mapping & @offset.
  *
- * PCG flags modify how the page is returned.
+ * FGP flags modify how the page is returned.
  *
  * @fgp_flags can be:
  *
Signature · 1 line
2.25.0
#15 lore raw

Re: [PATCH v2 05/25] mm: Fix documentation of FGP flags

Quoted 8 lines · Matthew Wilcox

- * @fgp_flags: PCG flags
+ * @fgp_flags: FGP flags
* @gfp_mask: gfp mask to use for the page cache data page allocation *
* Looks up the page cache slot at @mapping & @offset.
*
- * PCG flags modify how the page is returned.
+ * FGP flags modify how the page is returned.

This still looks weird. Why not just a single line:

* @fgp_flags: FGP_* flags that control how the page is returned.

author maintainer
#16 lore raw

Re: [PATCH v2 05/25] mm: Fix documentation of FGP flags

Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Matthew Wilcox

- * @fgp_flags: PCG flags
+ * @fgp_flags: FGP flags
* @gfp_mask: gfp mask to use for the page cache data page allocation *
* Looks up the page cache slot at @mapping & @offset.
*
- * PCG flags modify how the page is returned.
+ * FGP flags modify how the page is returned.

Quoted 3 lines

This still looks weird. Why not just a single line:

* @fgp_flags: FGP_* flags that control how the page is returned.

Well, now you got me reading the entire comment for this function, and looking at the html output, so I ended up rewriting it entirely.

+++ b/mm/filemap.c
@@ -1574,37 +1574,34 @@ struct page *find_lock_entry(struct address_space *mapping, pgoff_t offset)
EXPORT_SYMBOL(find_lock_entry);

/**
- * pagecache_get_page - find and get a page reference
- * @mapping: the address_space to search
- * @offset: the page index
- * @fgp_flags: FGP flags
- * @gfp_mask: gfp mask to use for the page cache data page allocation
- *
- * Looks up the page cache slot at @mapping & @offset.
+ * pagecache_get_page - Find and get a reference to a page.
+ * @mapping: The address_space to search.
+ * @offset: The page index.
+ * @fgp_flags: %FGP flags modify how the page is returned.
+ * @gfp_mask: Memory allocation flags to use if %FGP_CREAT is specified. *
- * FGP flags modify how the page is returned.
+ * Looks up the page cache entry at @mapping & @offset.
*
- * @fgp_flags can be:
+ * @fgp_flags can be zero or more of these flags:
*
- * - FGP_ACCESSED: the page will be marked accessed
- * - FGP_LOCK: Page is return locked
- * - FGP_CREAT: If page is not present then a new page is allocated using

- *   @gfp_mask and added to the page cache and the VM's LRU
- *   list. The page is returned locked and with an increased
- *   refcount.

- * - FGP_FOR_MMAP: Similar to FGP_CREAT, only we want to allow the caller to do
- * its own locking dance if the page is already in cache, or unlock the page
- * before returning if we had to add the page to pagecache.
+ * * %FGP_ACCESSED - The page will be marked accessed.
+ * * %FGP_LOCK - The page is returned locked.
+ * * %FGP_CREAT - If no page is present then a new page is allocated using
+ * @gfp_mask and added to the page cache and the VM's LRU list.
+ * The page is returned locked and with an increased refcount.
+ * * %FGP_FOR_MMAP - The caller wants to do its own locking dance if the
+ * page is already in cache. If the page was allocated, unlock it before
+ * returning so the caller can do the same dance.
*
- * If FGP_LOCK or FGP_CREAT are specified then the function may sleep even
- * if the GFP flags specified for FGP_CREAT are atomic.
+ * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even
+ * if the %GFP flags specified for %FGP_CREAT are atomic.
*
* If there is a page cache page, it is returned with an increased refcount. *
- * Return: the found page or %NULL otherwise.
+ * Return: The found page or %NULL otherwise.
*/
struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
- int fgp_flags, gfp_t gfp_mask)
+ int fgp_flags, gfp_t gfp_mask)
{

#17 lore raw

Re: [PATCH v2 05/25] mm: Fix documentation of FGP flags

Quoted 64 lines · Matthew Wilcox
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Matthew Wilcox

- * @fgp_flags: PCG flags
+ * @fgp_flags: FGP flags
* @gfp_mask: gfp mask to use for the page cache data page allocation *
* Looks up the page cache slot at @mapping & @offset.
*
- * PCG flags modify how the page is returned.
+ * FGP flags modify how the page is returned.

Quoted 3 lines

This still looks weird. Why not just a single line:

* @fgp_flags: FGP_* flags that control how the page is returned.

Well, now you got me reading the entire comment for this function, and looking at the html output, so I ended up rewriting it entirely.

+++ b/mm/filemap.c

(quoted diff) +17 -20
@@ -1574,37 +1574,34 @@ struct page *find_lock_entry(struct address_space *mapping, pgoff_t offset)
 EXPORT_SYMBOL(find_lock_entry);
 
 /**
- * pagecache_get_page - find and get a page reference
- * @mapping: the address_space to search
- * @offset: the page index
- * @fgp_flags: FGP flags
- * @gfp_mask: gfp mask to use for the page cache data page allocation
- *
- * Looks up the page cache slot at @mapping & @offset.
+ * pagecache_get_page - Find and get a reference to a page.
+ * @mapping: The address_space to search.
+ * @offset: The page index.
+ * @fgp_flags: %FGP flags modify how the page is returned.
+ * @gfp_mask: Memory allocation flags to use if %FGP_CREAT is specified.
  *
- * FGP flags modify how the page is returned.
+ * Looks up the page cache entry at @mapping & @offset.
  *
- * @fgp_flags can be:
+ * @fgp_flags can be zero or more of these flags:
  *
- * - FGP_ACCESSED: the page will be marked accessed
- * - FGP_LOCK: Page is return locked
- * - FGP_CREAT: If page is not present then a new page is allocated using
- *   @gfp_mask and added to the page cache and the VM's LRU
- *   list. The page is returned locked and with an increased
- *   refcount.
- * - FGP_FOR_MMAP: Similar to FGP_CREAT, only we want to allow the caller to do
- *   its own locking dance if the page is already in cache, or unlock the page
- *   before returning if we had to add the page to pagecache.
+ * * %FGP_ACCESSED - The page will be marked accessed.
+ * * %FGP_LOCK - The page is returned locked.
+ * * %FGP_CREAT - If no page is present then a new page is allocated using
+ *   @gfp_mask and added to the page cache and the VM's LRU list.
+ *   The page is returned locked and with an increased refcount.
+ * * %FGP_FOR_MMAP - The caller wants to do its own locking dance if the
+ *   page is already in cache.  If the page was allocated, unlock it before
+ *   returning so the caller can do the same dance.
  *
- * If FGP_LOCK or FGP_CREAT are specified then the function may sleep even
- * if the GFP flags specified for FGP_CREAT are atomic.
+ * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even
+ * if the %GFP flags specified for %FGP_CREAT are atomic.

Review struct page *find_lock_entry(struct address_space *mapping, pgoff_t offset) · line 1605

Quoted 10 lines
(quoted diff) +2 -2
  *
  * If there is a page cache page, it is returned with an increased refcount.
  *
- * Return: the found page or %NULL otherwise.
+ * Return: The found page or %NULL otherwise.
  */
 struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
-       int fgp_flags, gfp_t gfp_mask)
+               int fgp_flags, gfp_t gfp_mask)
 {

LGTM:

Signature · 1 line
 Kirill A. Shutemov
#18 lore raw

Re: [PATCH v2 05/25] mm: Fix documentation of FGP flags

In reply to Matthew Wilcox
Quoted 3 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We never had PCG flags

We actually had :P But it's totally different story.

See git log for include/linux/page_cgroup.h.

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#19 lore raw

Re: [PATCH v2 05/25] mm: Fix documentation of FGP flags

Quoted 8 lines · Kirill A. Shutemov
Quoted 3 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We never had PCG flags

We actually had :P But it's totally different story.

See git log for include/linux/page_cgroup.h.

Thanks, wording updated.

author maintainer
#20 lore raw

PATCH v2 6/25 mm: Allow hpages to be arbitrary order

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Remove the assumption in hpage_nr_pages() that compound pages are necessarily PMD sized. The return type needs to be signed as we need to use the negative value, eg when calling update_lru_size().

---
 include/linux/huge_mm.h | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
include/linux/huge_mm.h +3 -7
index 5aca3d1bdb32..16367e2f771e 100644
@@ -230,12 +230,8 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,
 	else
 		return NULL;
 }
-static inline int hpage_nr_pages(struct page *page)
-{
-	if (unlikely(PageTransHuge(page)))
-		return HPAGE_PMD_NR;
-	return 1;
-}
+
+#define hpage_nr_pages(page)	(long)compound_nr(page)
 
 struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, int flags, struct dev_pagemap **pgmap);
@@ -289,7 +285,7 @@ static inline struct list_head *page_deferred_list(struct page *page)
 #define HPAGE_PUD_MASK ({ BUILD_BUG(); 0; })
 #define HPAGE_PUD_SIZE ({ BUILD_BUG(); 0; })
 
-#define hpage_nr_pages(x) 1
+#define hpage_nr_pages(x) 1L
 
 static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma)
 {
Signature · 1 line
2.25.0
#21 lore raw

Re: [PATCH v2 06/25] mm: Allow hpages to be arbitrary order

Quoted 5 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Remove the assumption in hpage_nr_pages() that compound pages are necessarily PMD sized. The return type needs to be signed as we need to use the negative value, eg when calling update_lru_size().

But should it be long?
Any reason to use macros instead of inline function?

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#22 lore raw

Re: [PATCH v2 06/25] mm: Allow hpages to be arbitrary order

Quoted 9 lines · Kirill A. Shutemov
Quoted 5 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Remove the assumption in hpage_nr_pages() that compound pages are necessarily PMD sized. The return type needs to be signed as we need to use the negative value, eg when calling update_lru_size().

But should it be long?
Any reason to use macros instead of inline function?

Huh, that does look like a bit of a weird change now you point it out. I'll change it back:

static inline int hpage_nr_pages(struct page *page) {
- if (unlikely(PageTransHuge(page)))
- return HPAGE_PMD_NR;
- return 1;
+ return compound_nr(page);
}

author maintainer
#23 lore raw

PATCH v2 7/25 mm: Introduce thp_size

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

This is like page_size(), but compiles down to just PAGE_SIZE if THP are disabled. Convert the users of hpage_nr_pages() which would prefer this interface.

---
 drivers/nvdimm/btt.c    | 4 +---
 drivers/nvdimm/pmem.c   | 3 +--
 include/linux/huge_mm.h | 2 ++
 mm/internal.h           | 2 +-
 mm/page_io.c            | 2 +-
 mm/page_vma_mapped.c    | 4 ++--
 6 files changed, 8 insertions(+), 9 deletions(-)
drivers/nvdimm/btt.c +1 -3
index 0d04ea3d9fd7..5d6a2a22f5a0 100644
@@ -1488,10 +1488,8 @@ static int btt_rw_page(struct block_device *bdev, sector_t sector,
 {
 	struct btt *btt = bdev->bd_disk->private_data;
 	int rc;
-	unsigned int len;
 
-	len = hpage_nr_pages(page) * PAGE_SIZE;
-	rc = btt_do_bvec(btt, NULL, page, len, 0, op, sector);
+	rc = btt_do_bvec(btt, NULL, page, thp_size(page), 0, op, sector);
 	if (rc == 0)
 		page_endio(page, op_is_write(op), 0);
 
drivers/nvdimm/pmem.c +1 -2
index 4eae441f86c9..9c71c81f310f 100644
@@ -223,8 +223,7 @@ static int pmem_rw_page(struct block_device *bdev, sector_t sector,
 	struct pmem_device *pmem = bdev->bd_queue->queuedata;
 	blk_status_t rc;
 
-	rc = pmem_do_bvec(pmem, page, hpage_nr_pages(page) * PAGE_SIZE,
-			  0, op, sector);
+	rc = pmem_do_bvec(pmem, page, thp_size(page), 0, op, sector);
 
 	/*
 	 * The ->rw_page interface is subtle and tricky.  The core
include/linux/huge_mm.h +2 -0
index 16367e2f771e..3680ae2d9019 100644
@@ -232,6 +232,7 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,
 }
 
 #define hpage_nr_pages(page)	(long)compound_nr(page)
+#define thp_size(page)		page_size(page)
 
 struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, int flags, struct dev_pagemap **pgmap);
@@ -286,6 +287,7 @@ static inline struct list_head *page_deferred_list(struct page *page)
 #define HPAGE_PUD_SIZE ({ BUILD_BUG(); 0; })
 
 #define hpage_nr_pages(x) 1L
+#define thp_size(x)		PAGE_SIZE
 
 static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma)
 {
mm/internal.h +1 -1
index 41b93c4b3ab7..390d81d8b85f 100644
@@ -358,7 +358,7 @@ vma_address(struct page *page, struct vm_area_struct *vma)
 	unsigned long start, end;
 
 	start = __vma_address(page, vma);
-	end = start + PAGE_SIZE * (hpage_nr_pages(page) - 1);
+	end = start + thp_size(page) - PAGE_SIZE;
 
 	/* page should be within @vma mapping range */
 	VM_BUG_ON_VMA(end < vma->vm_start || start >= vma->vm_end, vma);
mm/page_io.c +1 -1
index 76965be1d40e..dd935129e3cb 100644
@@ -41,7 +41,7 @@ static struct bio *get_swap_bio(gfp_t gfp_flags,
 		bio->bi_iter.bi_sector <<= PAGE_SHIFT - 9;
 		bio->bi_end_io = end_io;
 
-		bio_add_page(bio, page, PAGE_SIZE * hpage_nr_pages(page), 0);
+		bio_add_page(bio, page, thp_size(page), 0);
 	}
 	return bio;
 }
mm/page_vma_mapped.c +2 -2
index 719c35246cfa..e65629c056e8 100644
@@ -227,7 +227,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
 			if (pvmw->address >= pvmw->vma->vm_end ||
 			    pvmw->address >=
 					__vma_address(pvmw->page, pvmw->vma) +
-					hpage_nr_pages(pvmw->page) * PAGE_SIZE)
+					thp_size(pvmw->page))
 				return not_found(pvmw);
 			/* Did we cross page table boundary? */
 			if (pvmw->address % PMD_SIZE == 0) {
@@ -268,7 +268,7 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma)
 	unsigned long start, end;
 
 	start = __vma_address(page, vma);
-	end = start + PAGE_SIZE * (hpage_nr_pages(page) - 1);
+	end = start + thp_size(page) - PAGE_SIZE;
 
 	if (unlikely(end < vma->vm_start || start >= vma->vm_end))
 		return 0;
Signature · 1 line
2.25.0
#24 lore raw

Re: [PATCH v2 07/25] mm: Introduce thp_size

Quoted 7 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

This is like page_size(), but compiles down to just PAGE_SIZE if THP are disabled. Convert the users of hpage_nr_pages() which would prefer this interface.

I would prefer the new helper to be inline function rather than macro. Otherwise looks good.

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#25 lore raw

PATCH v2 8/25 mm: Introduce thp_order

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Like compound_order() except 0 when THP is disabled.

---
 include/linux/huge_mm.h | 2 ++
 1 file changed, 2 insertions(+)
include/linux/huge_mm.h +2 -0
index 3680ae2d9019..3de788ee25bd 100644
@@ -233,6 +233,7 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,
 
 #define hpage_nr_pages(page)	(long)compound_nr(page)
 #define thp_size(page)		page_size(page)
+#define thp_order(page)		compound_order(page)
 
 struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, int flags, struct dev_pagemap **pgmap);
@@ -288,6 +289,7 @@ static inline struct list_head *page_deferred_list(struct page *page)
 
 #define hpage_nr_pages(x) 1L
 #define thp_size(x)		PAGE_SIZE
+#define thp_order(x)		0U
 
 static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma)
 {
Signature · 1 line
2.25.0
#26 lore raw

Re: [PATCH v2 08/25] mm: Introduce thp_order

Quoted 3 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Like compound_order() except 0 when THP is disabled.

Again, functions are preferred if an option.

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#27 lore raw

PATCH v2 9/25 fs: Add a filesystem flag for large pages

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

The page cache needs to know whether the filesystem supports pages > PAGE_SIZE.

---
 include/linux/fs.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/fs.h +1 -0
index d4e2d2964346..24e720723afb 100644
@@ -2235,6 +2235,7 @@ struct file_system_type {
 #define FS_HAS_SUBTYPE		4
 #define FS_USERNS_MOUNT		8	/* Can be mounted by userns root */
 #define FS_DISALLOW_NOTIFY_PERM	16	/* Disable fanotify permission events */
+#define FS_LARGE_PAGES		8192	/* Remove once all fs converted */
 #define FS_RENAME_DOES_D_MOVE	32768	/* FS will handle d_move() during rename() internally. */
 	int (*init_fs_context)(struct fs_context *);
 	const struct fs_parameter_spec *parameters;
Signature · 1 line
2.25.0
#28 lore raw

Re: [PATCH v2 09/25] fs: Add a filesystem flag for large pages

Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

The page cache needs to know whether the filesystem supports pages > PAGE_SIZE.

Does it make sense to set this flag on the file_system_type, which is rather broad scope, or a specific superblock or even inode?

For some file systems we might require on-disk flags that aren't set for all instances.

author maintainer
#29 lore raw

Re: [PATCH v2 09/25] fs: Add a filesystem flag for large pages

Quoted 11 lines · Christoph Hellwig
Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

The page cache needs to know whether the filesystem supports pages > PAGE_SIZE.

Does it make sense to set this flag on the file_system_type, which is rather broad scope, or a specific superblock or even inode?

For some file systems we might require on-disk flags that aren't set for all instances.

I don't see why we'd need on-disk flags or need to control this on a per-inode or per-sb basis. My intent for this flag is to represent whether the filesystem understands large pages; how the file is cached should make no difference to the on-disk layout.

author maintainer
#30 lore raw

PATCH v2 10/25 fs: Introduce i_blocks_per_page

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

This helper is useful for both large pages in the page cache and for supporting block size larger than page size. Convert some example users (we have a few different ways of writing this idiom).

---
 fs/iomap/buffered-io.c  |  8 ++++----
 fs/jfs/jfs_metapage.c   |  2 +-
 fs/xfs/xfs_aops.c       |  2 +-
 include/linux/pagemap.h | 16 ++++++++++++++++
 4 files changed, 22 insertions(+), 6 deletions(-)
fs/iomap/buffered-io.c +4 -4
index e40eb45230fa..c551a48e2a81 100644
@@ -46,7 +46,7 @@ iomap_page_create(struct inode *inode, struct page *page)
 {
 	struct iomap_page *iop = to_iomap_page(page);
 
-	if (iop || i_blocksize(inode) == PAGE_SIZE)
+	if (iop || i_blocks_per_page(inode, page) <= 1)
 		return iop;
 
 	iop = kmalloc(sizeof(*iop), GFP_NOFS | __GFP_NOFAIL);
@@ -152,7 +152,7 @@ iomap_iop_set_range_uptodate(struct page *page, unsigned off, unsigned len)
 	unsigned int i;
 
 	spin_lock_irqsave(&iop->uptodate_lock, flags);
-	for (i = 0; i < PAGE_SIZE / i_blocksize(inode); i++) {
+	for (i = 0; i < i_blocks_per_page(inode, page); i++) {
 		if (i >= first && i <= last)
 			set_bit(i, iop->uptodate);
 		else if (!test_bit(i, iop->uptodate))
@@ -1073,7 +1073,7 @@ iomap_finish_page_writeback(struct inode *inode, struct page *page,
 		mapping_set_error(inode->i_mapping, -EIO);
 	}
 
-	WARN_ON_ONCE(i_blocksize(inode) < PAGE_SIZE && !iop);
+	WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop);
 	WARN_ON_ONCE(iop && atomic_read(&iop->write_count) <= 0);
 
 	if (!iop || atomic_dec_and_test(&iop->write_count))
@@ -1369,7 +1369,7 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc,
 	int error = 0, count = 0, i;
 	LIST_HEAD(submit_list);
 
-	WARN_ON_ONCE(i_blocksize(inode) < PAGE_SIZE && !iop);
+	WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop);
 	WARN_ON_ONCE(iop && atomic_read(&iop->write_count) != 0);
 
 	/*
fs/jfs/jfs_metapage.c +1 -1
index a2f5338a5ea1..176580f54af9 100644
@@ -473,7 +473,7 @@ static int metapage_readpage(struct file *fp, struct page *page)
 	struct inode *inode = page->mapping->host;
 	struct bio *bio = NULL;
 	int block_offset;
-	int blocks_per_page = PAGE_SIZE >> inode->i_blkbits;
+	int blocks_per_page = i_blocks_per_page(inode, page);
 	sector_t page_start;	/* address of page in fs blocks */
 	sector_t pblock;
 	int xlen;
fs/xfs/xfs_aops.c +1 -1
index 0897dd71c622..5573bf2957dd 100644
@@ -544,7 +544,7 @@ xfs_discard_page(
 			page, ip->i_ino, offset);
 
 	error = xfs_bmap_punch_delalloc_range(ip, start_fsb,
-			PAGE_SIZE / i_blocksize(inode));
+			i_blocks_per_page(inode, page));
 	if (error && !XFS_FORCED_SHUTDOWN(mp))
 		xfs_alert(mp, "page discard unable to remove delalloc mapping.");
 out_invalidate:
include/linux/pagemap.h +16 -0
index 0842622cca90..aa925295347c 100644
@@ -748,4 +748,20 @@ static inline int page_mkwrite_check_truncate(struct page *page,
 	return offset;
 }
 
+/**
+ * i_blocks_per_page - How many blocks fit in this page.
+ * @inode: The inode which contains the blocks.
+ * @page: The (potentially large) page.
+ *
+ * If the block size is larger than the size of this page, will return
+ * zero,
+ *
+ * Context: Any context.
+ * Return: The number of filesystem blocks covered by this page.
+ */
+static inline
+unsigned int i_blocks_per_page(struct inode *inode, struct page *page)
+{
+	return thp_size(page) >> inode->i_blkbits;
+}
 #endif /* _LINUX_PAGEMAP_H */
Signature · 1 line
2.25.0
Reviewed-by on v2 10/25
#31 lore raw

Re: [PATCH v2 10/25] fs: Introduce i_blocks_per_page

Looks good modulo some nitpicks below:

Quoted 1 line

+ * Context: Any context.

Does this add any value for a trivial helper like this?

include/linux/pagemap.h static inline int page_mkwrite_check_truncate(struct page *page, · line 763

Quoted 4 lines

+ * Return: The number of filesystem blocks covered by this page.
+ */

include/linux/pagemap.h +2 -0
+static inline
+unsigned int i_blocks_per_page(struct inode *inode, struct page *page)

static inline unisnged int i_blocks_per_page(struct inode *inode, struct page *page)

author maintainer
#32 lore raw

Re: [PATCH v2 10/25] fs: Introduce i_blocks_per_page

Quoted 7 lines · Christoph Hellwig

Looks good modulo some nitpicks below:

Quoted 1 line

+ * Context: Any context.

Does this add any value for a trivial helper like this?

I think it's good to put them in to remind people they should be putting them in for more complex functions. Just like the Return: section.

Quoted 7 lines

include/linux/pagemap.h static inline int page_mkwrite_check_truncate(struct page *page, · line 763

Quoted 4 lines

+ * Return: The number of filesystem blocks covered by this page.
+ */

include/linux/pagemap.h +2 -0
+static inline
+unsigned int i_blocks_per_page(struct inode *inode, struct page *page)

static inline unisnged int i_blocks_per_page(struct inode *inode, struct page *page)

That's XFS coding style. Linus has specifically forbidden that:

https://lore.kernel.org/lkml/[email protected]/

#33 lore raw

Re: [PATCH v2 10/25] fs: Introduce i_blocks_per_page

Quoted 11 lines · Matthew Wilcox
Quoted 7 lines

include/linux/pagemap.h static inline int page_mkwrite_check_truncate(struct page *page, · line 763

Quoted 4 lines

+ * Return: The number of filesystem blocks covered by this page.
+ */

include/linux/pagemap.h +2 -0
+static inline
+unsigned int i_blocks_per_page(struct inode *inode, struct page *page)

static inline unisnged int i_blocks_per_page(struct inode *inode, struct page *page)

That's XFS coding style. Linus has specifically forbidden that:

https://lore.kernel.org/lkml/[email protected]/

Not just xfs but lots of places. But if you don't like that follow the real Linus style we use elsewhere:

static inline unsigned int i_blocks_per_page(struct inode *inode,
struct page *page)

#34 lore raw

Re: [PATCH v2 10/25] fs: Introduce i_blocks_per_page

In reply to Matthew Wilcox
Quoted 5 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

This helper is useful for both large pages in the page cache and for supporting block size larger than page size. Convert some example users (we have a few different ways of writing this idiom).

Maybe we should list what was converted and what wasn't. Like it's important to know that fs/buffer.c is not covered.

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#35 lore raw

Re: [PATCH v2 10/25] fs: Introduce i_blocks_per_page

Quoted 9 lines · Kirill A. Shutemov
Quoted 5 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

This helper is useful for both large pages in the page cache and for supporting block size larger than page size. Convert some example users (we have a few different ways of writing this idiom).

Maybe we should list what was converted and what wasn't. Like it's important to know that fs/buffer.c is not covered.

I don't know what could have been converted and wasn't ... I just went looking for a few places that use idioms like this. Happy to add some more examples.

author maintainer
#36 lore raw

PATCH v2 11/25 fs: Make page_mkwrite_check_truncate thp-aware

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If the page is compound, check the appropriate indices and return the appropriate sizes.

---
 include/linux/pagemap.h | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
include/linux/pagemap.h +6 -5
index aa925295347c..2ec33aabdbf6 100644
@@ -732,17 +732,18 @@ static inline int page_mkwrite_check_truncate(struct page *page,
 					      struct inode *inode)
 {
 	loff_t size = i_size_read(inode);
-	pgoff_t index = size >> PAGE_SHIFT;
-	int offset = offset_in_page(size);
+	pgoff_t first_index = size >> PAGE_SHIFT;
+	pgoff_t last_index = first_index + hpage_nr_pages(page) - 1;
+	unsigned long offset = offset_in_this_page(page, size);
 
 	if (page->mapping != inode->i_mapping)
 		return -EFAULT;
 
 	/* page is wholly inside EOF */
-	if (page->index < index)
-		return PAGE_SIZE;
+	if (page->index < first_index)
+		return thp_size(page);
 	/* page is wholly past EOF */
-	if (page->index > index || !offset)
+	if (page->index > last_index || !offset)
 		return -EFAULT;
 	/* page is partially inside EOF */
 	return offset;
Signature · 1 line
2.25.0
#37 lore raw

Re: [PATCH v2 11/25] fs: Make page_mkwrite_check_truncate thp-aware

Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If the page is compound, check the appropriate indices and return the appropriate sizes.

Is it guarnteed that the page is never called on tail page?

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#38 lore raw

Re: [PATCH v2 11/25] fs: Make page_mkwrite_check_truncate thp-aware

Quoted 7 lines · Kirill A. Shutemov
Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If the page is compound, check the appropriate indices and return the appropriate sizes.

Is it guarnteed that the page is never called on tail page?

I think so. page_mkwrite_check_truncate() is only called on pages which belong to a particular filesystem. Only filesystems which have the FS_LARGE_PAGES flag set will have compound pages allocated in the page cache for their files. As filesystems are converted, they will only see large head pages.

I'll happily put in a VM_BUG_ON(PageTail(page), page); to ensure we don't screw that up.

author maintainer
#39 lore raw

PATCH v2 12/25 mm: Add file_offset_of_ helpers

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

The page_offset function is badly named for people reading the functions which call it. The natural meaning of a function with this name would be 'offset within a page', not 'page offset in bytes within a file'. Dave Chinner suggests file_offset_of_page() as a replacement function name and I'm also adding file_offset_of_next_page() as a helper for the large page work. Also add kernel-doc for these functions so they show up in the kernel API book.

page_offset() is retained as a compatibility define for now.

---
 drivers/net/ethernet/ibm/ibmveth.c |  2 --
 include/linux/pagemap.h            | 25 ++++++++++++++++++++++---
 2 files changed, 22 insertions(+), 5 deletions(-)
drivers/net/ethernet/ibm/ibmveth.c +0 -2
index 84121aab7ff1..4cad94ac9bc9 100644
@@ -978,8 +978,6 @@ static int ibmveth_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
 	return -EOPNOTSUPP;
 }
 
-#define page_offset(v) ((unsigned long)(v) & ((1 << 12) - 1))
-
 static int ibmveth_send(struct ibmveth_adapter *adapter,
 			union ibmveth_buf_desc *descs, unsigned long mss)
 {
include/linux/pagemap.h +22 -3
index 2ec33aabdbf6..497197315b73 100644
@@ -432,14 +432,33 @@ static inline pgoff_t page_to_pgoff(struct page *page)
 	return page_to_index(page);
 }
 
-/*
- * Return byte-offset into filesystem object for page.
+/**
+ * file_offset_of_page - File offset of this page.
+ * @page: Page cache page.
+ *
+ * Context: Any context.
+ * Return: The offset of the first byte of this page.
  */
-static inline loff_t page_offset(struct page *page)
+static inline loff_t file_offset_of_page(struct page *page)
 {
 	return ((loff_t)page->index) << PAGE_SHIFT;
 }
 
+/* Legacy; please convert callers */
+#define page_offset(page)	file_offset_of_page(page)
+
+/**
+ * file_offset_of_next_page - File offset of the next page.
+ * @page: Page cache page.
+ *
+ * Context: Any context.
+ * Return: The offset of the first byte after this page.
+ */
+static inline loff_t file_offset_of_next_page(struct page *page)
+{
+	return file_offset_of_page(page) + thp_size(page);
+}
+
 static inline loff_t page_file_offset(struct page *page)
 {
 	return ((loff_t)page_index(page)) << PAGE_SHIFT;
Signature · 1 line
2.25.0
#40 lore raw

Re: [PATCH v2 12/25] mm: Add file_offset_of_ helpers

drivers/net/ethernet/ibm/ibmveth.c static int ibmveth_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd) · line 981

Quoted 7 lines
drivers/net/ethernet/ibm/ibmveth.c +0 -1
@@ -978,8 +978,6 @@ static int ibmveth_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
 	return -EOPNOTSUPP;
 }
 
-#define page_offset(v) ((unsigned long)(v) & ((1 << 12) - 1))

This one realy should be killed off in a separate patch, it has nothing to do with the kernel-wide page_offset.

drivers/net/ethernet/ibm/ibmveth.c static int ibmveth_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd) · line 982

Quoted 2 lines
drivers/net/ethernet/ibm/ibmveth.c +2 -0
+/* Legacy; please convert callers */
+#define page_offset(page)	file_offset_of_page(page)

I'd say send a script to Linus to conver it as soon as the change is in.

author maintainer
#41 lore raw

PATCH v2 13/25 fs: Add zero_user_large

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We can't kmap() a THP, so add a wrapper around zero_user() for large pages.

---
 include/linux/highmem.h | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
include/linux/highmem.h +22 -0
index ea5cdbd8c2c3..4465b8784353 100644
@@ -245,6 +245,28 @@ static inline void zero_user(struct page *page,
 	zero_user_segments(page, start, start + size, 0, 0);
 }
 
+static inline void zero_user_large(struct page *page,
+		unsigned start, unsigned size)
+{
+	unsigned int i;
+
+	for (i = 0; i < thp_order(page); i++) {
+		if (start > PAGE_SIZE) {
+			start -= PAGE_SIZE;
+		} else {
+			unsigned this_size = size;
+
+			if (size > (PAGE_SIZE - start))
+				this_size = PAGE_SIZE - start;
+			zero_user(page + i, start, this_size);
+			start = 0;
+			size -= this_size;
+			if (!size)
+				break;
+		}
+	}
+}
+
 #ifndef __HAVE_ARCH_COPY_USER_HIGHPAGE
 
 static inline void copy_user_highpage(struct page *to, struct page *from,
Signature · 1 line
2.25.0
#42 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We can't kmap() a THP, so add a wrapper around zero_user() for large pages.

I would rather address it closer to the root: make zero_user_segments() handle compound pages.

Quoted 11 lines
---
 include/linux/highmem.h | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
include/linux/highmem.h +0 -0
index ea5cdbd8c2c3..4465b8784353 100644

include/linux/highmem.h static inline void zero_user(struct page *page, · line 254

Quoted 10 lines
include/linux/highmem.h +7 -0
@@ -245,6 +245,28 @@ static inline void zero_user(struct page *page,
 	zero_user_segments(page, start, start + size, 0, 0);
 }
 
+static inline void zero_user_large(struct page *page,
+		unsigned start, unsigned size)
+{
+	unsigned int i;
+
+	for (i = 0; i < thp_order(page); i++) {
+		if (start > PAGE_SIZE) {

Off-by-one? >= ?

Quoted 22 lines
include/linux/highmem.h +15 -0
+			start -= PAGE_SIZE;
+		} else {
+			unsigned this_size = size;
+
+			if (size > (PAGE_SIZE - start))
+				this_size = PAGE_SIZE - start;
+			zero_user(page + i, start, this_size);
+			start = 0;
+			size -= this_size;
+			if (!size)
+				break;
+		}
+	}
+}
+
 #ifndef __HAVE_ARCH_COPY_USER_HIGHPAGE
 
 static inline void copy_user_highpage(struct page *to, struct page *from,
Signature · 1 line
2.25.0
Signature · 1 line
 Kirill A. Shutemov
author maintainer
#43 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 8 lines · Kirill A. Shutemov
Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We can't kmap() a THP, so add a wrapper around zero_user() for large pages.

I would rather address it closer to the root: make zero_user_segments() handle compound pages.

Hah. I ended up doing that, but hadn't sent it out. I don't like how ugly it is:

@@ -219,18 +219,57 @@ static inline void zero_user_segments(struct page *page,
unsigned start1, unsigned end1,
unsigned start2, unsigned end2)
{
- void *kaddr = kmap_atomic(page);
-
- BUG_ON(end1 > PAGE_SIZE || end2 > PAGE_SIZE);
-
- if (end1 > start1)
- memset(kaddr + start1, 0, end1 - start1);
-
- if (end2 > start2)
- memset(kaddr + start2, 0, end2 - start2);
-

-       kunmap_atomic(kaddr);
-       flush_dcache_page(page);
+       unsigned int i;

+
+ BUG_ON(end1 > thp_size(page) || end2 > thp_size(page));
+
+ for (i = 0; i < hpage_nr_pages(page); i++) {
+ void *kaddr;
+ unsigned this_end;
+
+ if (end1 == 0 && start2 >= PAGE_SIZE) {

+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;
+                       continue;
+               }
+
+               if (start1 >= PAGE_SIZE) {
+                       start1 -= PAGE_SIZE;
+                       end1 -= PAGE_SIZE;
+                       if (start2) {

+ start2 -= PAGE_SIZE;
+ end2 -= PAGE_SIZE;
+ }
+ continue;
+ }
+
+ kaddr = kmap_atomic(page + i);
+
+ this_end = min_t(unsigned, end1, PAGE_SIZE);
+ if (end1 > start1)
+ memset(kaddr + start1, 0, this_end - start1);
+ end1 -= this_end;
+ start1 = 0;
+
+ if (start2 >= PAGE_SIZE) {
+ start2 -= PAGE_SIZE;
+ end2 -= PAGE_SIZE;
+ } else {
+ this_end = min_t(unsigned, end2, PAGE_SIZE);
+ if (end2 > start2)
+ memset(kaddr + start2, 0, this_end - start2);
+ end2 -= this_end;
+ start2 = 0;
+ }
+
+ kunmap_atomic(kaddr);
+ flush_dcache_page(page + i);
+
+ if (!end1 && !end2)
+ break;
+ }
+
+ BUG_ON((start1 | start2 | end1 | end2) != 0);
}

I think at this point it has to move out-of-line too.

Quoted 9 lines

include/linux/highmem.h static inline void zero_user(struct page *page, · line 254

Quoted 7 lines
include/linux/highmem.h +7 -0
+static inline void zero_user_large(struct page *page,
+		unsigned start, unsigned size)
+{
+	unsigned int i;
+
+	for (i = 0; i < thp_order(page); i++) {
+		if (start > PAGE_SIZE) {

Off-by-one? >= ?

Good catch; I'd also noticed that when I came to redo the zero_user_segments().

#44 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 40 lines · Matthew Wilcox
Quoted 8 lines · Kirill A. Shutemov
Quoted 4 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We can't kmap() a THP, so add a wrapper around zero_user() for large pages.

I would rather address it closer to the root: make zero_user_segments() handle compound pages.

Hah. I ended up doing that, but hadn't sent it out. I don't like how ugly it is:

(quoted diff) +11 -12
@@ -219,18 +219,57 @@ static inline void zero_user_segments(struct page *page,
        unsigned start1, unsigned end1,
        unsigned start2, unsigned end2)
 {
-       void *kaddr = kmap_atomic(page);
-
-       BUG_ON(end1 > PAGE_SIZE || end2 > PAGE_SIZE);
-
-       if (end1 > start1)
-               memset(kaddr + start1, 0, end1 - start1);
-
-       if (end2 > start2)
-               memset(kaddr + start2, 0, end2 - start2);
-
-       kunmap_atomic(kaddr);
-       flush_dcache_page(page);
+       unsigned int i;
+
+       BUG_ON(end1 > thp_size(page) || end2 > thp_size(page));
+
+       for (i = 0; i < hpage_nr_pages(page); i++) {
+               void *kaddr;
+               unsigned this_end;
+
+               if (end1 == 0 && start2 >= PAGE_SIZE) {
+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;

Review static inline void zero_user_segments(struct page *page, · line 242

Quoted 10 lines
(quoted diff) +10 -0
+                       continue;
+               }
+
+               if (start1 >= PAGE_SIZE) {
+                       start1 -= PAGE_SIZE;
+                       end1 -= PAGE_SIZE;
+                       if (start2) {
+                               start2 -= PAGE_SIZE;
+                               end2 -= PAGE_SIZE;
+                       }

You assume start2/end2 is always after start1/end1 in the page. Is it always true? If so, I would add BUG_ON() for it.

Otherwise, looks good.

Quoted 46 lines
(quoted diff) +30 -0
+                       continue;
+               }
+
+               kaddr = kmap_atomic(page + i);
+
+               this_end = min_t(unsigned, end1, PAGE_SIZE);
+               if (end1 > start1)
+                       memset(kaddr + start1, 0, this_end - start1);
+               end1 -= this_end;
+               start1 = 0;
+
+               if (start2 >= PAGE_SIZE) {
+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;
+               } else {
+                       this_end = min_t(unsigned, end2, PAGE_SIZE);
+                       if (end2 > start2)
+                               memset(kaddr + start2, 0, this_end - start2);
+                       end2 -= this_end;
+                       start2 = 0;
+               }
+
+               kunmap_atomic(kaddr);
+               flush_dcache_page(page + i);
+
+               if (!end1 && !end2)
+                       break;
+       }
+
+       BUG_ON((start1 | start2 | end1 | end2) != 0);
 }
 

I think at this point it has to move out-of-line too.

Quoted 9 lines

include/linux/highmem.h static inline void zero_user(struct page *page, · line 254

Quoted 7 lines
include/linux/highmem.h +7 -0
+static inline void zero_user_large(struct page *page,
+		unsigned start, unsigned size)
+{
+	unsigned int i;
+
+	for (i = 0; i < thp_order(page); i++) {
+		if (start > PAGE_SIZE) {

Off-by-one? >= ?

Good catch; I'd also noticed that when I came to redo the zero_user_segments().

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#45 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 10 lines · Kirill A. Shutemov

Review

Quoted 7 lines
(quoted diff) +7 -0
+               if (start1 >= PAGE_SIZE) {
+                       start1 -= PAGE_SIZE;
+                       end1 -= PAGE_SIZE;
+                       if (start2) {
+                               start2 -= PAGE_SIZE;
+                               end2 -= PAGE_SIZE;
+                       }

You assume start2/end2 is always after start1/end1 in the page. Is it always true? If so, I would add BUG_ON() for it.

after or zero. Yes, I should add a BUG_ON to check for that.

Quoted 1 line

Otherwise, looks good.

Here's what I currently have (I'll add the BUG_ON later):

commit 7fabe16755365cdc6e80343ef994843ecebde60a

Author: Matthew Wilcox (Oracle) <[email protected]>
Date:   Sat Feb 1 03:38:49 2020 -0500

fs: Support THPs in zero_user_segments

We can only kmap() one subpage of a THP at a time, so loop over all relevant subpages, skipping ones which don't need to be zeroed. This is too large to inline when THPs are enabled and we actually need highmem, so put it in highmem.c.

include/linux/highmem.h +11 -4
index ea5cdbd8c2c3..74614903619d 100644
@@ -215,13 +215,18 @@ static inline void clear_highpage(struct page *page)
        kunmap_atomic(kaddr);
 }
 
+#if defined(CONFIG_HIGHMEM) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
+void zero_user_segments(struct page *page, unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2);
+#else /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */
 static inline void zero_user_segments(struct page *page,
-       unsigned start1, unsigned end1,
-       unsigned start2, unsigned end2)
+               unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2)
 {
+       unsigned long i;
        void *kaddr = kmap_atomic(page);
 
-       BUG_ON(end1 > PAGE_SIZE || end2 > PAGE_SIZE);
+       BUG_ON(end1 > thp_size(page) || end2 > thp_size(page));
 
        if (end1 > start1)
                memset(kaddr + start1, 0, end1 - start1);
@@ -230,8 +235,10 @@ static inline void zero_user_segments(struct page *page,
                memset(kaddr + start2, 0, end2 - start2);
 
        kunmap_atomic(kaddr);
-       flush_dcache_page(page);
+       for (i = 0; i < hpage_nr_pages(page); i++)
+               flush_dcache_page(page + i);
 }
+#endif /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */
 
 static inline void zero_user_segment(struct page *page,
        unsigned start, unsigned end)
mm/highmem.c +60 -2

69 lines not on this page · raw message

Quoted 49 lines
Quoted 46 lines
(quoted diff) +30 -0
+                       continue;
+               }
+
+               kaddr = kmap_atomic(page + i);
+
+               this_end = min_t(unsigned, end1, PAGE_SIZE);
+               if (end1 > start1)
+                       memset(kaddr + start1, 0, this_end - start1);
+               end1 -= this_end;
+               start1 = 0;
+
+               if (start2 >= PAGE_SIZE) {
+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;
+               } else {
+                       this_end = min_t(unsigned, end2, PAGE_SIZE);
+                       if (end2 > start2)
+                               memset(kaddr + start2, 0, this_end - start2);
+                       end2 -= this_end;
+                       start2 = 0;
+               }
+
+               kunmap_atomic(kaddr);
+               flush_dcache_page(page + i);
+
+               if (!end1 && !end2)
+                       break;
+       }
+
+       BUG_ON((start1 | start2 | end1 | end2) != 0);
 }

I think at this point it has to move out-of-line too.

Quoted 9 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+static inline void zero_user_large(struct page *page,
+		unsigned start, unsigned size)
+{
+	unsigned int i;
+
+	for (i = 0; i < thp_order(page); i++) {
+		if (start > PAGE_SIZE) {

Off-by-one? >= ?

Good catch; I'd also noticed that when I came to redo the zero_user_segments().

Signature · 1 line
 Kirill A. Shutemov
#46 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 22 lines · Matthew Wilcox
Quoted 10 lines · Kirill A. Shutemov

mm/highmem.c void kunmap_high(struct page *page) · line 396

Quoted 7 lines
mm/highmem.c +7 -0
+               if (start1 >= PAGE_SIZE) {
+                       start1 -= PAGE_SIZE;
+                       end1 -= PAGE_SIZE;
+                       if (start2) {
+                               start2 -= PAGE_SIZE;
+                               end2 -= PAGE_SIZE;
+                       }

You assume start2/end2 is always after start1/end1 in the page. Is it always true? If so, I would add BUG_ON() for it.

after or zero. Yes, I should add a BUG_ON to check for that.

Quoted 1 line

Otherwise, looks good.

Here's what I currently have (I'll add the BUG_ON later):

commit 7fabe16755365cdc6e80343ef994843ecebde60a

Author: Matthew Wilcox (Oracle) <[email protected]>
Date:   Sat Feb 1 03:38:49 2020 -0500
Quoted 8 lines

fs: Support THPs in zero_user_segments

We can only kmap() one subpage of a THP at a time, so loop over all relevant subpages, skipping ones which don't need to be zeroed. This is too large to inline when THPs are enabled and we actually need highmem, so put it in highmem.c.

include/linux/highmem.h static inline void clear_highpage(struct page *page) · line 221

Quoted 13 lines
include/linux/highmem.h +4 -0
index ea5cdbd8c2c3..74614903619d 100644
@@ -215,13 +215,18 @@ static inline void clear_highpage(struct page *page)
        kunmap_atomic(kaddr);
 }
 
+#if defined(CONFIG_HIGHMEM) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
+void zero_user_segments(struct page *page, unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2);
+#else /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */

This is a neat trick. I like it.

Although, it means non-inlined version will never get tested :/

Quoted 152 lines
include/linux/highmem.h +7 -4
 static inline void zero_user_segments(struct page *page,
-       unsigned start1, unsigned end1,
-       unsigned start2, unsigned end2)
+               unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2)
 {
+       unsigned long i;
        void *kaddr = kmap_atomic(page);
 
-       BUG_ON(end1 > PAGE_SIZE || end2 > PAGE_SIZE);
+       BUG_ON(end1 > thp_size(page) || end2 > thp_size(page));
 
        if (end1 > start1)
                memset(kaddr + start1, 0, end1 - start1);
@@ -230,8 +235,10 @@ static inline void zero_user_segments(struct page *page,
                memset(kaddr + start2, 0, end2 - start2);
 
        kunmap_atomic(kaddr);
-       flush_dcache_page(page);
+       for (i = 0; i < hpage_nr_pages(page); i++)
+               flush_dcache_page(page + i);
 }
+#endif /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */
 
 static inline void zero_user_segment(struct page *page,
        unsigned start, unsigned end)
mm/highmem.c +60 -2
index 64d8dea47dd1..3a85c66ef532 100644
@@ -367,9 +367,67 @@ void kunmap_high(struct page *page)
        if (need_wakeup)
                wake_up(pkmap_map_wait);
 }
-
 EXPORT_SYMBOL(kunmap_high);
-#endif
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+void zero_user_segments(struct page *page, unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2)
+{
+       unsigned int i;
+
+       BUG_ON(end1 > thp_size(page) || end2 > thp_size(page));
+
+       for (i = 0; i < hpage_nr_pages(page); i++) {
+               void *kaddr;
+               unsigned this_end;
+
+               if (end1 == 0 && start2 >= PAGE_SIZE) {
+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;
+                       continue;
+               }
+
+               if (start1 >= PAGE_SIZE) {
+                       start1 -= PAGE_SIZE;
+                       end1 -= PAGE_SIZE;
+                       if (start2) {
+                               start2 -= PAGE_SIZE;
+                               end2 -= PAGE_SIZE;
+                       }
+                       continue;
+               }
+
+               kaddr = kmap_atomic(page + i);
+
+               this_end = min_t(unsigned, end1, PAGE_SIZE);
+               if (end1 > start1)
+                       memset(kaddr + start1, 0, this_end - start1);
+               end1 -= this_end;
+               start1 = 0;
+
+               if (start2 >= PAGE_SIZE) {
+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;
+               } else {
+                       this_end = min_t(unsigned, end2, PAGE_SIZE);
+                       if (end2 > start2)
+                               memset(kaddr + start2, 0, this_end - start2);
+                       end2 -= this_end;
+                       start2 = 0;
+               }
+
+               kunmap_atomic(kaddr);
+               flush_dcache_page(page + i);
+
+               if (!end1 && !end2)
+                       break;
+       }
+
+       BUG_ON((start1 | start2 | end1 | end2) != 0);
+}
+EXPORT_SYMBOL(zero_user_segments);
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+#endif /* CONFIG_HIGHMEM */
 
 #if defined(HASHED_PAGE_VIRTUAL)
 
Quoted 52 lines
Quoted 49 lines
mm/highmem.c +30 -0
+                       continue;
+               }
+
+               kaddr = kmap_atomic(page + i);
+
+               this_end = min_t(unsigned, end1, PAGE_SIZE);
+               if (end1 > start1)
+                       memset(kaddr + start1, 0, this_end - start1);
+               end1 -= this_end;
+               start1 = 0;
+
+               if (start2 >= PAGE_SIZE) {
+                       start2 -= PAGE_SIZE;
+                       end2 -= PAGE_SIZE;
+               } else {
+                       this_end = min_t(unsigned, end2, PAGE_SIZE);
+                       if (end2 > start2)
+                               memset(kaddr + start2, 0, this_end - start2);
+                       end2 -= this_end;
+                       start2 = 0;
+               }
+
+               kunmap_atomic(kaddr);
+               flush_dcache_page(page + i);
+
+               if (!end1 && !end2)
+                       break;
+       }
+
+       BUG_ON((start1 | start2 | end1 | end2) != 0);
 }

I think at this point it has to move out-of-line too.

Quoted 9 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+static inline void zero_user_large(struct page *page,
+		unsigned start, unsigned size)
+{
+	unsigned int i;
+
+	for (i = 0; i < thp_order(page); i++) {
+		if (start > PAGE_SIZE) {

Off-by-one? >= ?

Good catch; I'd also noticed that when I came to redo the zero_user_segments().

Signature · 1 line
 Kirill A. Shutemov
Signature · 1 line
 Kirill A. Shutemov
author maintainer
#47 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 1 line · Kirill A. Shutemov

Thanks

Quoted 8 lines

include/linux/highmem.h static inline void clear_highpage(struct page *page) · line 221

Quoted 4 lines
include/linux/highmem.h +4 -0
+#if defined(CONFIG_HIGHMEM) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
+void zero_user_segments(struct page *page, unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2);
+#else /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */

This is a neat trick. I like it.

Although, it means non-inlined version will never get tested :/

I worry about that too, but I don't really want to incur the overhead on platforms people actually use.

#48 lore raw

Re: [PATCH v2 13/25] fs: Add zero_user_large

Quoted 12 lines · Matthew Wilcox
Quoted 1 line · Kirill A. Shutemov

Thanks

Quoted 7 lines

include/linux/highmem.h static inline void clear_highpage(struct page *page) · line 221

Quoted 4 lines
include/linux/highmem.h +4 -0
+#if defined(CONFIG_HIGHMEM) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
+void zero_user_segments(struct page *page, unsigned start1, unsigned end1,
+               unsigned start2, unsigned end2);
+#else /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */

This is a neat trick. I like it.

Quoted 4 lines
Quoted 1 line

Although, it means non-inlined version will never get tested :/

I worry about that too, but I don't really want to incur the overhead on platforms people actually use.

I'm also worried about latency: kmap_atomic() disables preemption even if system has no highmem. Some archs have way too large THP to clear them with preemption disabled.

I *think* there's no real need in preemption disabling in this situation and we can wrap kmap_atomic()/kunmap_atomic() into CONFIG_HIGHMEM.

Signature · 1 line
 Kirill A. Shutemov
author maintainer
#49 lore raw

PATCH v2 14/25 iomap: Support arbitrarily many blocks per page

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Size the uptodate array dynamically. Now that this array is protected by a spinlock, we can use bitmap functions to set the bits in this array instead of a loop around set_bit().

---
 fs/iomap/buffered-io.c | 27 +++++++++------------------
 1 file changed, 9 insertions(+), 18 deletions(-)
fs/iomap/buffered-io.c +9 -18
index c551a48e2a81..5e5a6b038fc3 100644
@@ -22,14 +22,14 @@
 #include "../internal.h"
 
 /*
- * Structure allocated for each page when block size < PAGE_SIZE to track
+ * Structure allocated for each page when block size < page size to track
  * sub-page uptodate status and I/O completions.
  */
 struct iomap_page {
 	atomic_t		read_count;
 	atomic_t		write_count;
 	spinlock_t		uptodate_lock;
-	DECLARE_BITMAP(uptodate, PAGE_SIZE / 512);
+	unsigned long		uptodate[];
 };
 
 static inline struct iomap_page *to_iomap_page(struct page *page)
@@ -45,15 +45,14 @@ static struct iomap_page *
 iomap_page_create(struct inode *inode, struct page *page)
 {
 	struct iomap_page *iop = to_iomap_page(page);
+	unsigned int n, nr_blocks = i_blocks_per_page(inode, page);
 
-	if (iop || i_blocks_per_page(inode, page) <= 1)
+	if (iop || nr_blocks <= 1)
 		return iop;
 
-	iop = kmalloc(sizeof(*iop), GFP_NOFS | __GFP_NOFAIL);
-	atomic_set(&iop->read_count, 0);
-	atomic_set(&iop->write_count, 0);
+	n = BITS_TO_LONGS(nr_blocks);
+	iop = kzalloc(struct_size(iop, uptodate, n), GFP_NOFS | __GFP_NOFAIL);
 	spin_lock_init(&iop->uptodate_lock);
-	bitmap_zero(iop->uptodate, PAGE_SIZE / SECTOR_SIZE);
 
 	/*
 	 * migrate_page_move_mapping() assumes that pages with private data have
@@ -146,20 +145,12 @@ iomap_iop_set_range_uptodate(struct page *page, unsigned off, unsigned len)
 	struct iomap_page *iop = to_iomap_page(page);
 	struct inode *inode = page->mapping->host;
 	unsigned first = off >> inode->i_blkbits;
-	unsigned last = (off + len - 1) >> inode->i_blkbits;
-	bool uptodate = true;
+	unsigned count = len >> inode->i_blkbits;
 	unsigned long flags;
-	unsigned int i;
 
 	spin_lock_irqsave(&iop->uptodate_lock, flags);
-	for (i = 0; i < i_blocks_per_page(inode, page); i++) {
-		if (i >= first && i <= last)
-			set_bit(i, iop->uptodate);
-		else if (!test_bit(i, iop->uptodate))
-			uptodate = false;
-	}
-
-	if (uptodate)
+	bitmap_set(iop->uptodate, first, count);
+	if (bitmap_full(iop->uptodate, i_blocks_per_page(inode, page)))
 		SetPageUptodate(page);
 	spin_unlock_irqrestore(&iop->uptodate_lock, flags);
 }
Signature · 1 line
2.25.0
#50 lore raw

Re: [PATCH v2 14/25] iomap: Support arbitrarily many blocks per page

Quoted 37 lines · Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Size the uptodate array dynamically. Now that this array is protected by a spinlock, we can use bitmap functions to set the bits in this array instead of a loop around set_bit().

---
 fs/iomap/buffered-io.c | 27 +++++++++------------------
 1 file changed, 9 insertions(+), 18 deletions(-)
fs/iomap/buffered-io.c +3 -2
index c551a48e2a81..5e5a6b038fc3 100644
@@ -22,14 +22,14 @@
 #include "../internal.h"
 
 /*
- * Structure allocated for each page when block size < PAGE_SIZE to track
+ * Structure allocated for each page when block size < page size to track
  * sub-page uptodate status and I/O completions.
  */
 struct iomap_page {
 	atomic_t		read_count;
 	atomic_t		write_count;
 	spinlock_t		uptodate_lock;
-	DECLARE_BITMAP(uptodate, PAGE_SIZE / 512);
+	unsigned long		uptodate[];
 };
 
 static inline struct iomap_page *to_iomap_page(struct page *page)
@@ -45,15 +45,14 @@ static struct iomap_page *
 iomap_page_create(struct inode *inode, struct page *page)
 {
 	struct iomap_page *iop = to_iomap_page(page);
+	unsigned int n, nr_blocks = i_blocks_per_page(inode, page);

fs/iomap/buffered-io.c static struct iomap_page * · line 54

Quoted 10 lines
fs/iomap/buffered-io.c +3 -4
 
-	if (iop || i_blocks_per_page(inode, page) <= 1)
+	if (iop || nr_blocks <= 1)
 		return iop;
 
-	iop = kmalloc(sizeof(*iop), GFP_NOFS | __GFP_NOFAIL);
-	atomic_set(&iop->read_count, 0);
-	atomic_set(&iop->write_count, 0);
+	n = BITS_TO_LONGS(nr_blocks);
+	iop = kzalloc(struct_size(iop, uptodate, n), GFP_NOFS | __GFP_NOFAIL);

Nit: I don't really hink we need the n variable here, we can just opencode the BITS_TO_LONGS in the struct_size call.

Quoted 10 lines
fs/iomap/buffered-io.c +1 -4
 	struct inode *inode = page->mapping->host;
 	unsigned first = off >> inode->i_blkbits;
-	unsigned last = (off + len - 1) >> inode->i_blkbits;
-	bool uptodate = true;
+	unsigned count = len >> inode->i_blkbits;
 	unsigned long flags;
-	unsigned int i;
 
 	spin_lock_irqsave(&iop->uptodate_lock, flags);
-	for (i = 0; i < i_blocks_per_page(inode, page); i++) {

fs/iomap/buffered-io.c iomap_iop_set_range_uptodate(struct page *page, unsigned off, unsigned len) · line 154

Quoted 10 lines
fs/iomap/buffered-io.c +2 -7
-		if (i >= first && i <= last)
-			set_bit(i, iop->uptodate);
-		else if (!test_bit(i, iop->uptodate))
-			uptodate = false;
-	}
-
-	if (uptodate)
+	bitmap_set(iop->uptodate, first, count);
+	if (bitmap_full(iop->uptodate, i_blocks_per_page(inode, page)))
 		SetPageUptodate(page);

Switching to the bitmap helpers might be worthwhile prep patch that can go in directly now that we're having the uptodate_lock.

author maintainer
#51 lore raw

PATCH v2 15/25 iomap: Support large pages in iomap_adjust_read_range

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Pass the struct page instead of the iomap_page so we can determine the size of the page. Introduce offset_in_this_page() and use thp_size() instead of PAGE_SIZE.

---
 fs/iomap/buffered-io.c | 16 +++++++++-------
 include/linux/mm.h     |  2 ++
 2 files changed, 11 insertions(+), 7 deletions(-)
fs/iomap/buffered-io.c +9 -7
index 5e5a6b038fc3..e522039f627f 100644
@@ -83,15 +83,16 @@ iomap_page_release(struct page *page)
  * Calculate the range inside the page that we actually need to read.
  */
 static void
-iomap_adjust_read_range(struct inode *inode, struct iomap_page *iop,
+iomap_adjust_read_range(struct inode *inode, struct page *page,
 		loff_t *pos, loff_t length, unsigned *offp, unsigned *lenp)
 {
+	struct iomap_page *iop = to_iomap_page(page);
 	loff_t orig_pos = *pos;
 	loff_t isize = i_size_read(inode);
 	unsigned block_bits = inode->i_blkbits;
 	unsigned block_size = (1 << block_bits);
-	unsigned poff = offset_in_page(*pos);
-	unsigned plen = min_t(loff_t, PAGE_SIZE - poff, length);
+	unsigned poff = offset_in_this_page(page, *pos);
+	unsigned plen = min_t(loff_t, thp_size(page) - poff, length);
 	unsigned first = poff >> block_bits;
 	unsigned last = (poff + plen - 1) >> block_bits;
 
@@ -129,7 +130,8 @@ iomap_adjust_read_range(struct inode *inode, struct iomap_page *iop,
 	 * page cache for blocks that are entirely outside of i_size.
 	 */
 	if (orig_pos <= isize && orig_pos + length > isize) {
-		unsigned end = offset_in_page(isize - 1) >> block_bits;
+		unsigned end = offset_in_this_page(page, isize - 1) >>
+				block_bits;
 
 		if (first <= end && last > end)
 			plen -= (last - end) * block_size;
@@ -256,7 +258,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 	}
 
 	/* zero post-eof blocks as the page may be mapped */
-	iomap_adjust_read_range(inode, iop, &pos, length, &poff, &plen);
+	iomap_adjust_read_range(inode, page, &pos, length, &poff, &plen);
 	if (plen == 0)
 		goto done;
 
@@ -547,7 +549,6 @@ static int
 __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags,
 		struct page *page, struct iomap *srcmap)
 {
-	struct iomap_page *iop = iomap_page_create(inode, page);
 	loff_t block_size = i_blocksize(inode);
 	loff_t block_start = pos & ~(block_size - 1);
 	loff_t block_end = (pos + len + block_size - 1) & ~(block_size - 1);
@@ -556,9 +557,10 @@ __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags,
 
 	if (PageUptodate(page))
 		return 0;
+	iomap_page_create(inode, page);
 
 	do {
-		iomap_adjust_read_range(inode, iop, &block_start,
+		iomap_adjust_read_range(inode, page, &block_start,
 				block_end - block_start, &poff, &plen);
 		if (plen == 0)
 			break;
include/linux/mm.h +2 -0
index 52269e56c514..b4bf86590096 100644
@@ -1387,6 +1387,8 @@ static inline void clear_page_pfmemalloc(struct page *page)
 extern void pagefault_out_of_memory(void);
 
 #define offset_in_page(p)	((unsigned long)(p) & ~PAGE_MASK)
+#define offset_in_this_page(page, p)	\
+	((unsigned long)(p) & (thp_size(page) - 1))
 
 /*
  * Flags passed to show_mem() and show_free_areas() to suppress output in
Signature · 1 line
2.25.0
#52 lore raw

Re: [PATCH v2 15/25] iomap: Support large pages in iomap_adjust_read_range

fs/iomap/buffered-io.c __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags, · line 560

Quoted 12 lines

__iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags,

fs/iomap/buffered-io.c +1 -1
 		struct page *page, struct iomap *srcmap)
 {
-	struct iomap_page *iop = iomap_page_create(inode, page);
 	loff_t block_size = i_blocksize(inode);
 	loff_t block_start = pos & ~(block_size - 1);
 	loff_t block_end = (pos + len + block_size - 1) & ~(block_size - 1);
@@ -556,9 +557,10 @@ __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags,
 
 	if (PageUptodate(page))
 		return 0;
+	iomap_page_create(inode, page);

FYI, I have a similar change in a pending series that only creates the iomap_page if a page isn't actually mapped by a contiguous extent. Lets see which series goes in first, but the conflicts shouldn't be too bad.

include/linux/mm.h static inline void clear_page_pfmemalloc(struct page *page) · line 1391

Quoted 10 lines
include/linux/mm.h +2 -0
index 52269e56c514..b4bf86590096 100644
@@ -1387,6 +1387,8 @@ static inline void clear_page_pfmemalloc(struct page *page)
 extern void pagefault_out_of_memory(void);
 
 #define offset_in_page(p)	((unsigned long)(p) & ~PAGE_MASK)
+#define offset_in_this_page(page, p)	\
+	((unsigned long)(p) & (thp_size(page) - 1))

I think this should go int oa separate patch.

author maintainer
#53 lore raw

PATCH v2 16/25 iomap: Support large pages in read paths

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Use thp_size() instead of PAGE_SIZE, use offset_in_this_page() and abstract away how to access the list of readahead pages.

---
 fs/iomap/buffered-io.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
fs/iomap/buffered-io.c +20 -8
index e522039f627f..68f8903ecd6d 100644
@@ -179,14 +179,16 @@ iomap_read_finish(struct iomap_page *iop, struct page *page)
 static void
 iomap_read_page_end_io(struct bio_vec *bvec, int error)
 {
-	struct page *page = bvec->bv_page;
+	struct page *page = compound_head(bvec->bv_page);
 	struct iomap_page *iop = to_iomap_page(page);
+	unsigned offset = bvec->bv_offset +
+				PAGE_SIZE * (bvec->bv_page - page);
 
 	if (unlikely(error)) {
 		ClearPageUptodate(page);
 		SetPageError(page);
 	} else {
-		iomap_set_range_uptodate(page, bvec->bv_offset, bvec->bv_len);
+		iomap_set_range_uptodate(page, offset, bvec->bv_len);
 	}
 
 	iomap_read_finish(iop, page);
@@ -239,6 +241,16 @@ static inline bool iomap_block_needs_zeroing(struct inode *inode,
 		pos >= i_size_read(inode);
 }
 
+/*
+ * Estimate the number of vectors we need based on the current page size;
+ * if we're wrong we'll end up doing an overly large allocation or needing
+ * to do a second allocation, neither of which is a big deal.
+ */
+static unsigned int iomap_nr_vecs(struct page *page, loff_t length)
+{
+	return (length + thp_size(page) - 1) >> page_shift(page);
+}
+
 static loff_t
 iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 		struct iomap *iomap, struct iomap *srcmap)
@@ -263,7 +275,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 		goto done;
 
 	if (iomap_block_needs_zeroing(inode, iomap, pos)) {
-		zero_user(page, poff, plen);
+		zero_user_large(page, poff, plen);
 		iomap_set_range_uptodate(page, poff, plen);
 		goto done;
 	}
@@ -294,7 +306,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 
 	if (!ctx->bio || !is_contig || bio_full(ctx->bio, plen)) {
 		gfp_t gfp = mapping_gfp_constraint(page->mapping, GFP_KERNEL);
-		int nr_vecs = (length + PAGE_SIZE - 1) >> PAGE_SHIFT;
+		int nr_vecs = iomap_nr_vecs(page, length);
 
 		if (ctx->bio)
 			submit_bio(ctx->bio);
@@ -331,9 +343,9 @@ iomap_readpage(struct page *page, const struct iomap_ops *ops)
 
 	trace_iomap_readpage(page->mapping->host, 1);
 
-	for (poff = 0; poff < PAGE_SIZE; poff += ret) {
-		ret = iomap_apply(inode, page_offset(page) + poff,
-				PAGE_SIZE - poff, 0, ops, &ctx,
+	for (poff = 0; poff < thp_size(page); poff += ret) {
+		ret = iomap_apply(inode, file_offset_of_page(page) + poff,
+				thp_size(page) - poff, 0, ops, &ctx,
 				iomap_readpage_actor);
 		if (ret <= 0) {
 			WARN_ON_ONCE(ret == 0);
@@ -376,7 +388,7 @@ iomap_readahead_actor(struct inode *inode, loff_t pos, loff_t length,
 		if (WARN_ON(ret == 0))
 			break;
 		done += ret;
-		if (offset_in_page(pos + done) == 0) {
+		if (offset_in_this_page(ctx->cur_page, pos + done) == 0) {
 			ctx->rac->nr_pages -= ctx->rac->batch_count;
 			if (!ctx->cur_page_in_bio)
 				unlock_page(ctx->cur_page);
Signature · 1 line
2.25.0
#54 lore raw

Re: [PATCH v2 16/25] iomap: Support large pages in read paths

fs/iomap/buffered-io.c static inline bool iomap_block_needs_zeroing(struct inode *inode, · line 252

Quoted 9 lines
fs/iomap/buffered-io.c +9 -0
+/*
+ * Estimate the number of vectors we need based on the current page size;
+ * if we're wrong we'll end up doing an overly large allocation or needing
+ * to do a second allocation, neither of which is a big deal.
+ */
+static unsigned int iomap_nr_vecs(struct page *page, loff_t length)
+{
+	return (length + thp_size(page) - 1) >> page_shift(page);
+}

With the multipage bvecs a huge page (or any physically contigous piece of memory) will always use one or less (if merged into the previous) bio_vec. So this can be simplified and optimized.

author maintainer
#55 lore raw

Re: [PATCH v2 16/25] iomap: Support large pages in read paths

Quoted 5 lines · Christoph Hellwig
Quoted 5 lines

+/*
+ * Estimate the number of vectors we need based on the current page size;
+ * if we're wrong we'll end up doing an overly large allocation or needing
+ * to do a second allocation, neither of which is a big deal.
+ */

Quoted 8 lines

fs/iomap/buffered-io.c static inline bool iomap_block_needs_zeroing(struct inode *inode, · line 252

Quoted 4 lines
fs/iomap/buffered-io.c +4 -0
+static unsigned int iomap_nr_vecs(struct page *page, loff_t length)
+{
+	return (length + thp_size(page) - 1) >> page_shift(page);
+}

With the multipage bvecs a huge page (or any physically contigous piece of memory) will always use one or less (if merged into the previous) bio_vec. So this can be simplified and optimized.

Oh, hm, right. So what you really want here is for me to pass in the number of thp pages allocated by the readahead operation. rac->nr_pages is the number of PAGE_SIZE pages, but we could have an rac->nr_segs element as well.

author maintainer
#56 lore raw

PATCH v2 17/25 iomap: Support large pages in write paths

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Use thp_size() instead of PAGE_SIZE, offset_in_this_page(). Also simplify the logic in iomap_do_writepage() for determining end of file.

---
 fs/iomap/buffered-io.c | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)
fs/iomap/buffered-io.c +21 -15
index 68f8903ecd6d..af1f56408fcd 100644
@@ -445,7 +445,7 @@ iomap_is_partially_uptodate(struct page *page, unsigned long from,
 	unsigned i;
 
 	/* Limit range to one page */
-	len = min_t(unsigned, PAGE_SIZE - from, count);
+	len = min_t(unsigned, thp_size(page) - from, count);
 
 	/* First and last blocks in range within page */
 	first = from >> inode->i_blkbits;
@@ -488,7 +488,7 @@ iomap_invalidatepage(struct page *page, unsigned int offset, unsigned int len)
 	 * If we are invalidating the entire page, clear the dirty state from it
 	 * and release it to avoid unnecessary buildup of the LRU.
 	 */
-	if (offset == 0 && len == PAGE_SIZE) {
+	if (offset == 0 && len == thp_size(page)) {
 		WARN_ON_ONCE(PageWriteback(page));
 		cancel_dirty_page(page);
 		iomap_page_release(page);
@@ -564,7 +564,9 @@ __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags,
 	loff_t block_size = i_blocksize(inode);
 	loff_t block_start = pos & ~(block_size - 1);
 	loff_t block_end = (pos + len + block_size - 1) & ~(block_size - 1);
-	unsigned from = offset_in_page(pos), to = from + len, poff, plen;
+	unsigned from = offset_in_this_page(page, pos);
+	unsigned to = from + len;
+	unsigned poff, plen;
 	int status;
 
 	if (PageUptodate(page))
@@ -696,7 +698,7 @@ __iomap_write_end(struct inode *inode, loff_t pos, unsigned len,
 	 */
 	if (unlikely(copied < len && !PageUptodate(page)))
 		return 0;
-	iomap_set_range_uptodate(page, offset_in_page(pos), len);
+	iomap_set_range_uptodate(page, offset_in_this_page(page, pos), len);
 	iomap_set_page_dirty(page);
 	return copied;
 }
@@ -771,6 +773,10 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 		unsigned long bytes;	/* Bytes to write to page */
 		size_t copied;		/* Bytes copied from user */
 
+		/*
+		 * XXX: We don't know what size page we'll find in the
+		 * page cache, so only copy up to a regular page boundary.
+		 */
 		offset = offset_in_page(pos);
 		bytes = min_t(unsigned long, PAGE_SIZE - offset,
 						iov_iter_count(i));
@@ -1320,7 +1326,7 @@ iomap_add_to_ioend(struct inode *inode, loff_t offset, struct page *page,
 {
 	sector_t sector = iomap_sector(&wpc->iomap, offset);
 	unsigned len = i_blocksize(inode);
-	unsigned poff = offset & (PAGE_SIZE - 1);
+	unsigned poff = offset & (thp_size(page) - 1);
 	bool merged, same_page = false;
 
 	if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, offset, sector)) {
@@ -1372,9 +1378,10 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc,
 	unsigned len = i_blocksize(inode);
 	u64 file_offset; /* file offset of page */
 	int error = 0, count = 0, i;
+	int nr_blocks = i_blocks_per_page(inode, page);
 	LIST_HEAD(submit_list);
 
-	WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop);
+	WARN_ON_ONCE(nr_blocks > 1 && !iop);
 	WARN_ON_ONCE(iop && atomic_read(&iop->write_count) != 0);
 
 	/*
@@ -1382,8 +1389,8 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc,
 	 * end of the current map or find the current map invalid, grab a new
 	 * one.
 	 */
-	for (i = 0, file_offset = page_offset(page);
-	     i < (PAGE_SIZE >> inode->i_blkbits) && file_offset < end_offset;
+	for (i = 0, file_offset = file_offset_of_page(page);
+	     i < nr_blocks && file_offset < end_offset;
 	     i++, file_offset += len) {
 		if (iop && !test_bit(i, iop->uptodate))
 			continue;
@@ -1477,7 +1484,6 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data)
 {
 	struct iomap_writepage_ctx *wpc = data;
 	struct inode *inode = page->mapping->host;
-	pgoff_t end_index;
 	u64 end_offset;
 	loff_t offset;
 
@@ -1518,10 +1524,9 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data)
 	 * ---------------------------------^------------------|
 	 */
 	offset = i_size_read(inode);
-	end_index = offset >> PAGE_SHIFT;
-	if (page->index < end_index)
-		end_offset = (loff_t)(page->index + 1) << PAGE_SHIFT;
-	else {
+	end_offset = file_offset_of_next_page(page);
+
+	if (end_offset > offset) {
 		/*
 		 * Check whether the page to write out is beyond or straddles
 		 * i_size or not.
@@ -1533,7 +1538,8 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data)
 		 * |				    |      Straddles     |
 		 * ---------------------------------^-----------|--------|
 		 */
-		unsigned offset_into_page = offset & (PAGE_SIZE - 1);
+		unsigned offset_into_page = offset_in_this_page(page, offset);
+		pgoff_t end_index = offset >> PAGE_SHIFT;
 
 		/*
 		 * Skip the page if it is fully outside i_size, e.g. due to a
@@ -1564,7 +1570,7 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data)
 		 * memory is zeroed when mapped, and writes to that region are
 		 * not written out to the file."
 		 */
-		zero_user_segment(page, offset_into_page, PAGE_SIZE);
+		zero_user_segment(page, offset_into_page, thp_size(page));
 
 		/* Adjust the end_offset to the end of file */
 		end_offset = offset;
Signature · 1 line
2.25.0
#57 lore raw

Re: [PATCH v2 17/25] iomap: Support large pages in write paths

fs/iomap/buffered-io.c iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data) · line 1529

Quoted 13 lines · Matthew Wilcox

Also simplify the logic in iomap_do_writepage() for determining end of file.

fs/iomap/buffered-io.c +3 -4
 	 * ---------------------------------^------------------|
 	 */
 	offset = i_size_read(inode);
-	end_index = offset >> PAGE_SHIFT;
-	if (page->index < end_index)
-		end_offset = (loff_t)(page->index + 1) << PAGE_SHIFT;
-	else {
+	end_offset = file_offset_of_next_page(page);
+
+	if (end_offset > offset) {

Nit: can you drop the empty line here? Maybe it is just a pet peeve of mine, but I hate empty line between a variable assignment, and a use that is directly related to it.

Also it might be worth to preload this as a separate patch as it is a very nice but not completely obvious cleanup that deserves to stand out.

author maintainer
#58 lore raw

PATCH v2 18/25 iomap: Inline data shouldn't see large pages

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

Assert that we're not seeing large pages in functions that read/write inline data, rather than zeroing out the tail.

---
 fs/iomap/buffered-io.c | 2 ++
 1 file changed, 2 insertions(+)
fs/iomap/buffered-io.c +2 -0
index af1f56408fcd..a7a21b99b3a0 100644
@@ -224,6 +224,7 @@ iomap_read_inline_data(struct inode *inode, struct page *page,
 		return;
 
 	BUG_ON(page->index);
+	BUG_ON(PageCompound(page));
 	BUG_ON(size > PAGE_SIZE - offset_in_page(iomap->inline_data));
 
 	addr = kmap_atomic(page);
@@ -710,6 +711,7 @@ iomap_write_end_inline(struct inode *inode, struct page *page,
 	void *addr;
 
 	WARN_ON_ONCE(!PageUptodate(page));
+	BUG_ON(PageCompound(page));
 	BUG_ON(pos + copied > PAGE_SIZE - offset_in_page(iomap->inline_data));
 
 	addr = kmap_atomic(page);
Signature · 1 line
2.25.0
author maintainer
#60 lore raw

PATCH v2 19/25 xfs: Support large pages

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

There is one place which assumes the size of a page; fix it.

---
 fs/xfs/xfs_aops.c  | 2 +-
 fs/xfs/xfs_super.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
fs/xfs/xfs_aops.c +1 -1
index 5573bf2957dd..0c10fd799f8c 100644
@@ -548,7 +548,7 @@ xfs_discard_page(
 	if (error && !XFS_FORCED_SHUTDOWN(mp))
 		xfs_alert(mp, "page discard unable to remove delalloc mapping.");
 out_invalidate:
-	iomap_invalidatepage(page, 0, PAGE_SIZE);
+	iomap_invalidatepage(page, 0, thp_size(page));
 }
 
 static const struct iomap_writeback_ops xfs_writeback_ops = {
fs/xfs/xfs_super.c +1 -1
index 2094386af8ac..a02efa1f72af 100644
@@ -1779,7 +1779,7 @@ static struct file_system_type xfs_fs_type = {
 	.init_fs_context	= xfs_init_fs_context,
 	.parameters		= xfs_fs_parameters,
 	.kill_sb		= kill_block_super,
-	.fs_flags		= FS_REQUIRES_DEV,
+	.fs_flags		= FS_REQUIRES_DEV | FS_LARGE_PAGES,
 };
 MODULE_ALIAS_FS("xfs");
 
Signature · 1 line
2.25.0
author maintainer
#61 lore raw

PATCH v2 20/25 mm: Make prep_transhuge_page return its argument

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

By permitting NULL or order-0 pages as an argument, and returning the argument, callers can write:

	return prep_transhuge_page(alloc_pages(...));

instead of assigning the result to a temporary variable and conditionally passing that to prep_transhuge_page().

---
 include/linux/huge_mm.h | 7 +++++--
 mm/huge_memory.c        | 9 +++++++--
 2 files changed, 12 insertions(+), 4 deletions(-)
include/linux/huge_mm.h +5 -2
index 3de788ee25bd..865b9c16c99c 100644
@@ -158,7 +158,7 @@ extern unsigned long thp_get_unmapped_area(struct file *filp,
 		unsigned long addr, unsigned long len, unsigned long pgoff,
 		unsigned long flags);
 
-extern void prep_transhuge_page(struct page *page);
+extern struct page *prep_transhuge_page(struct page *page);
 extern void free_transhuge_page(struct page *page);
 bool is_transparent_hugepage(struct page *page);
 
@@ -307,7 +307,10 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
 	return false;
 }
 
-static inline void prep_transhuge_page(struct page *page) {}
+static inline struct page *prep_transhuge_page(struct page *page)
+{
+	return page;
+}
 
 static inline bool is_transparent_hugepage(struct page *page)
 {
mm/huge_memory.c +7 -2
index b08b199f9a11..b52e007f0856 100644
@@ -508,15 +508,20 @@ static inline struct deferred_split *get_deferred_split_queue(struct page *page)
 }
 #endif
 
-void prep_transhuge_page(struct page *page)
+struct page *prep_transhuge_page(struct page *page)
 {
+	if (!page || compound_order(page) == 0)
+		return page;
 	/*
-	 * we use page->mapping and page->indexlru in second tail page
+	 * we use page->mapping and page->index in second tail page
 	 * as list_head: assuming THP order >= 2
 	 */
+	BUG_ON(compound_order(page) == 1);
 
 	INIT_LIST_HEAD(page_deferred_list(page));
 	set_compound_page_dtor(page, TRANSHUGE_PAGE_DTOR);
+
+	return page;
 }
 
 bool is_transparent_hugepage(struct page *page)
Signature · 1 line
2.25.0
author maintainer
#62 lore raw

PATCH v2 21/25 mm: Add __page_cache_alloc_order

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

This new function allows page cache pages to be allocated that are larger than an order-0 page.

---
 include/linux/pagemap.h | 24 +++++++++++++++++++++---
 mm/filemap.c            | 12 ++++++++----
 2 files changed, 29 insertions(+), 7 deletions(-)
include/linux/pagemap.h +21 -3
index 497197315b73..64a3cf79611f 100644
@@ -207,15 +207,33 @@ static inline int page_cache_add_speculative(struct page *page, int count)
 	return __page_cache_add_speculative(page, count);
 }
 
+static inline gfp_t thp_gfpmask(gfp_t gfp)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	/* We'd rather allocate smaller pages than stall a page fault */
+	gfp |= GFP_TRANSHUGE_LIGHT;
+	gfp &= ~__GFP_DIRECT_RECLAIM;
+#endif
+	return gfp;
+}
+
 #ifdef CONFIG_NUMA
-extern struct page *__page_cache_alloc(gfp_t gfp);
+extern struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order);
 #else
-static inline struct page *__page_cache_alloc(gfp_t gfp)
+static inline
+struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order)
 {
-	return alloc_pages(gfp, 0);
+	if (order == 0)
+		return alloc_pages(gfp, 0);
+	return prep_transhuge_page(alloc_pages(thp_gfpmask(gfp), order));
 }
 #endif
 
+static inline struct page *__page_cache_alloc(gfp_t gfp)
+{
+	return __page_cache_alloc_order(gfp, 0);
+}
+
 static inline struct page *page_cache_alloc(struct address_space *x)
 {
 	return __page_cache_alloc(mapping_gfp_mask(x));
mm/filemap.c +8 -4
index 3204293f9b58..1061463a169e 100644
@@ -941,24 +941,28 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
 EXPORT_SYMBOL_GPL(add_to_page_cache_lru);
 
 #ifdef CONFIG_NUMA
-struct page *__page_cache_alloc(gfp_t gfp)
+struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order)
 {
 	int n;
 	struct page *page;
 
+	if (order > 0)
+		gfp = thp_gfpmask(gfp);
+
 	if (cpuset_do_page_mem_spread()) {
 		unsigned int cpuset_mems_cookie;
 		do {
 			cpuset_mems_cookie = read_mems_allowed_begin();
 			n = cpuset_mem_spread_node();
-			page = __alloc_pages_node(n, gfp, 0);
+			page = __alloc_pages_node(n, gfp, order);
+			prep_transhuge_page(page);
 		} while (!page && read_mems_allowed_retry(cpuset_mems_cookie));
 
 		return page;
 	}
-	return alloc_pages(gfp, 0);
+	return prep_transhuge_page(alloc_pages(gfp, order));
 }
-EXPORT_SYMBOL(__page_cache_alloc);
+EXPORT_SYMBOL(__page_cache_alloc_order);
 #endif
 
 /*
Signature · 1 line
2.25.0
author maintainer
#63 lore raw

PATCH v2 22/25 mm: Allow large pages to be added to the page cache

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

We return -EEXIST if there are any non-shadow entries in the page cache in the range covered by the large page. If there are multiple shadow entries in the range, we set *shadowp to one of them (currently the one at the highest index). If that turns out to be the wrong answer, we can implement something more complex. This is mostly modelled after the equivalent function in the shmem code.

---
 mm/filemap.c | 46 +++++++++++++++++++++++++++++++++-------------
 1 file changed, 33 insertions(+), 13 deletions(-)
mm/filemap.c +33 -13
index 1061463a169e..08b5cd4ce47b 100644
@@ -834,6 +834,7 @@ static int __add_to_page_cache_locked(struct page *page,
 	int huge = PageHuge(page);
 	struct mem_cgroup *memcg;
 	int error;
+	unsigned int nr = 1;
 	void *old;
 
 	VM_BUG_ON_PAGE(!PageLocked(page), page);
@@ -845,31 +846,50 @@ static int __add_to_page_cache_locked(struct page *page,
 					      gfp_mask, &memcg, false);
 		if (error)
 			return error;
+		xas_set_order(&xas, offset, thp_order(page));
+		nr = hpage_nr_pages(page);
 	}
 
-	get_page(page);
+	page_ref_add(page, nr);
 	page->mapping = mapping;
 	page->index = offset;
 
 	do {
+		unsigned long exceptional = 0;
+		unsigned int i = 0;
+
 		xas_lock_irq(&xas);
-		old = xas_load(&xas);
-		if (old && !xa_is_value(old))
-			xas_set_err(&xas, -EEXIST);
-		xas_store(&xas, page);
+		xas_for_each_conflict(&xas, old) {
+			if (!xa_is_value(old)) {
+				xas_set_err(&xas, -EEXIST);
+				break;
+			}
+			exceptional++;
+			if (shadowp)
+				*shadowp = old;
+		}
+		xas_create_range(&xas);
 		if (xas_error(&xas))
 			goto unlock;
 
-		if (xa_is_value(old)) {
-			mapping->nrexceptional--;
-			if (shadowp)
-				*shadowp = old;
+next:
+		xas_store(&xas, page);
+		if (++i < nr) {
+			xas_next(&xas);
+			goto next;
 		}
-		mapping->nrpages++;
+		mapping->nrexceptional -= exceptional;
+		mapping->nrpages += nr;
 
 		/* hugetlb pages do not participate in page cache accounting */
-		if (!huge)
-			__inc_node_page_state(page, NR_FILE_PAGES);
+		if (!huge) {
+			__mod_node_page_state(page_pgdat(page), NR_FILE_PAGES,
+						nr);
+			if (nr > 1) {
+				__inc_node_page_state(page, NR_FILE_THPS);
+				filemap_nr_thps_inc(mapping);
+			}
+		}
 unlock:
 		xas_unlock_irq(&xas);
 	} while (xas_nomem(&xas, gfp_mask & GFP_RECLAIM_MASK));
@@ -886,7 +906,7 @@ static int __add_to_page_cache_locked(struct page *page,
 	/* Leave page->index set: truncation relies upon it */
 	if (!huge)
 		mem_cgroup_cancel_charge(page, memcg, false);
-	put_page(page);
+	page_ref_sub(page, nr);
 	return xas_error(&xas);
 }
 ALLOW_ERROR_INJECTION(__add_to_page_cache_locked, ERRNO);
Signature · 1 line
2.25.0
author maintainer
#64 lore raw

PATCH v2 23/25 mm: Allow large pages to be removed from the page cache

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

page_cache_free_page() assumes compound pages are PMD_SIZE; fix that assumption.

---
 mm/filemap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/filemap.c +1 -1
index 08b5cd4ce47b..e74a22af7e4e 100644
@@ -248,7 +248,7 @@ static void page_cache_free_page(struct address_space *mapping,
 		freepage(page);
 
 	if (PageTransHuge(page) && !PageHuge(page)) {
-		page_ref_sub(page, HPAGE_PMD_NR);
+		page_ref_sub(page, hpage_nr_pages(page));
 		VM_BUG_ON_PAGE(page_count(page) <= 0, page);
 	} else {
 		put_page(page);
Signature · 1 line
2.25.0
author maintainer
#65 lore raw

PATCH v2 24/25 mm: Add large page readahead

In reply to Matthew Wilcox

From: "Matthew Wilcox (Oracle)" <[email protected]>

If the filesystem supports large pages, allocate larger pages in the readahead code when it seems worth doing. The heuristic for choosing larger page sizes will surely need some tuning, but this aggressive ramp-up seems good for testing.

---
 mm/readahead.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 5 deletions(-)
mm/readahead.c +93 -5
index 29ca25c8f01e..b582f09aa7e3 100644
@@ -406,13 +406,96 @@ static int try_context_readahead(struct address_space *mapping,
 	return 1;
 }
 
+static inline int ra_alloc_page(struct address_space *mapping, pgoff_t offset,
+		pgoff_t mark, unsigned int order, gfp_t gfp)
+{
+	int err;
+	struct page *page = __page_cache_alloc_order(gfp, order);
+
+	if (!page)
+		return -ENOMEM;
+	if (mark - offset < (1UL << order))
+		SetPageReadahead(page);
+	err = add_to_page_cache_lru(page, mapping, offset, gfp);
+	if (err)
+		put_page(page);
+	return err;
+}
+
+#define PMD_ORDER	(PMD_SHIFT - PAGE_SHIFT)
+
+static unsigned long page_cache_readahead_order(struct address_space *mapping,
+		struct file_ra_state *ra, struct file *file, unsigned int order)
+{
+	struct readahead_control rac = {
+		.mapping = mapping,
+		.file = file,
+		.start = ra->start,
+		.nr_pages = 0,
+	};
+	unsigned int old_order = order;
+	pgoff_t offset = ra->start;
+	pgoff_t limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;
+	pgoff_t mark = offset + ra->size - ra->async_size;
+	int err = 0;
+	gfp_t gfp = readahead_gfp_mask(mapping);
+
+	limit = min(limit, offset + ra->size - 1);
+
+	/* Grow page size up to PMD size */
+	if (order < PMD_ORDER) {
+		order += 2;
+		if (order > PMD_ORDER)
+			order = PMD_ORDER;
+		while ((1 << order) > ra->size)
+			order--;
+	}
+
+	/* If size is somehow misaligned, fill with order-0 pages */
+	while (!err && offset & ((1UL << old_order) - 1)) {
+		err = ra_alloc_page(mapping, offset++, mark, 0, gfp);
+		if (!err)
+			rac.nr_pages++;
+	}
+
+	while (!err && offset & ((1UL << order) - 1)) {
+		err = ra_alloc_page(mapping, offset, mark, old_order, gfp);
+		if (!err)
+			rac.nr_pages += 1UL << old_order;
+		offset += 1UL << old_order;
+	}
+
+	while (!err && offset <= limit) {
+		err = ra_alloc_page(mapping, offset, mark, order, gfp);
+		if (!err)
+			rac.nr_pages += 1UL << order;
+		offset += 1UL << order;
+	}
+
+	if (offset > limit) {
+		ra->size += offset - limit - 1;
+		ra->async_size += offset - limit - 1;
+	}
+
+	read_pages(&rac, NULL);
+
+	/*
+	 * If there were already pages in the page cache, then we may have
+	 * left some gaps.  Let the regular readahead code take care of this
+	 * situation.
+	 */
+	if (err)
+		return ra_submit(ra, mapping, file);
+	return 0;
+}
+
 /*
  * A minimal readahead algorithm for trivial sequential/random reads.
  */
 static unsigned long
 ondemand_readahead(struct address_space *mapping,
 		   struct file_ra_state *ra, struct file *filp,
-		   bool hit_readahead_marker, pgoff_t offset,
+		   struct page *page, pgoff_t offset,
 		   unsigned long req_size)
 {
 	struct backing_dev_info *bdi = inode_to_bdi(mapping->host);
@@ -451,7 +534,7 @@ ondemand_readahead(struct address_space *mapping,
 	 * Query the pagecache for async_size, which normally equals to
 	 * readahead size. Ramp it up and use it as the new readahead size.
 	 */
-	if (hit_readahead_marker) {
+	if (page) {
 		pgoff_t start;
 
 		rcu_read_lock();
@@ -520,7 +603,12 @@ ondemand_readahead(struct address_space *mapping,
 		}
 	}
 
-	return ra_submit(ra, mapping, filp);
+	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) || !page ||
+	    !(mapping->host->i_sb->s_type->fs_flags & FS_LARGE_PAGES))
+		return ra_submit(ra, mapping, filp);
+
+	return page_cache_readahead_order(mapping, ra, filp,
+						compound_order(page));
 }
 
 /**
@@ -555,7 +643,7 @@ void page_cache_sync_readahead(struct address_space *mapping,
 	}
 
 	/* do read-ahead */
-	ondemand_readahead(mapping, ra, filp, false, offset, req_size);
+	ondemand_readahead(mapping, ra, filp, NULL, offset, req_size);
 }
 EXPORT_SYMBOL_GPL(page_cache_sync_readahead);
 
@@ -602,7 +690,7 @@ page_cache_async_readahead(struct address_space *mapping,
 		return;
 
 	/* do read-ahead */
-	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
+	ondemand_readahead(mapping, ra, filp, page, offset, req_size);
 }
 EXPORT_SYMBOL_GPL(page_cache_async_readahead);
 
Signature · 1 line
2.25.0
author maintainer
#66 lore raw

PATCH v2 25/25 mm: Align THP mappings for non-DAX

In reply to Matthew Wilcox

From: William Kucharski <[email protected]>

When we have the opportunity to use transparent huge pages to map a file, we want to follow the same rules as DAX.

---
 mm/huge_memory.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
mm/huge_memory.c +1 -4
index b52e007f0856..b8d9e0d76062 100644
@@ -577,13 +577,10 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
 	unsigned long ret;
 	loff_t off = (loff_t)pgoff << PAGE_SHIFT;
 
-	if (!IS_DAX(filp->f_mapping->host) || !IS_ENABLED(CONFIG_FS_DAX_PMD))
-		goto out;
-
 	ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE);
 	if (ret)
 		return ret;
-out:
+
 	return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags);
 }
 EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
Signature · 1 line
2.25.0
#67 lore raw

Re: [PATCH v2 25/25] mm: Align THP mappings for non-DAX

Quoted 9 lines
mm/huge_memory.c +0 -1
index b52e007f0856..b8d9e0d76062 100644
@@ -577,13 +577,10 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
 	unsigned long ret;
 	loff_t off = (loff_t)pgoff << PAGE_SHIFT;
 
-	if (!IS_DAX(filp->f_mapping->host) || !IS_ENABLED(CONFIG_FS_DAX_PMD))

mm/huge_memory.c unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, · line 586

Quoted 10 lines
mm/huge_memory.c +1 -3
-		goto out;
-
 	ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE);
 	if (ret)
 		return ret;
-out:
+
 	return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags);
 }
 EXPORT_SYMBOL_GPL(thp_get_unmapped_area);

There is no point in splitting thp_get_unmapped_area and __thp_get_unmapped_area with this applied (and arguably even before that). But we still have ext2 and ext4 that use thp_get_unmapped_area but only support huge page mappings for DAX, do we need to handle those somehow?