mm/gup: disallow GUP writing to file-backed mappings by default

Writing to file-backed mappings which require folio dirty tracking using GUP is a fundamentally broken operation, as kernel write access to GUP mappings do not adhere to the semantics expected by a file system.

A GUP caller uses the direct mapping to access the folio, which does not cause write notify to trigger, nor does it enforce that the caller marks the folio dirty.

The problem arises when, after an initial write to the folio, writeback results in the folio being cleaned and then the caller, via the GUP interface, writes to the folio again.

As a result of the use of this secondary, direct, mapping to the folio no write notify will occur, and if the caller does mark the folio dirty, this will be done so unexpectedly.

For example, consider the following scenario:-

1. A folio is written to via GUP which write-faults the memory, notifying the file system and dirtying the folio.
2. Later, writeback is triggered, resulting in the folio being cleaned and the PTE being marked read-only.
3. The GUP caller writes to the folio, as it is mapped read/write via the direct mapping.
4. The GUP caller, now done with the page, unpins it and sets it dirty (though it does not have to).

This results in both data being written to a folio without writenotify, and the folio being dirtied unexpectedly (if the caller decides to do so).

This issue was first reported by Jan Kara [1] in 2018, where the problem resulted in file system crashes.

This is only relevant when the mappings are file-backed and the underlying file system requires folio dirty tracking. File systems which do not, such as shmem or hugetlb, are not at risk and therefore can be written to without issue.

Unfortunately this limitation of GUP has been present for some time and requires future rework of the GUP API in order to provide correct write access to such mappings.

However, for the time being we introduce this check to prevent the most egregious case of this occurring, use of the FOLL_LONGTERM pin.

These mappings are considerably more likely to be written to after folios are cleaned and thus simply must not be permitted to do so.

This patch changes only the slow-path GUP functions, a following patch adapts the GUP-fast path along similar lines.

[1]:https://lore.kernel.org/linux-mm/[email protected]/

Suggested-by Jason Gunthorpe <[email protected]>

Signed-off-by Lorenzo Stoakes <[email protected]>

Reviewed-by John Hubbard <[email protected]>

Reviewed-by Mika Penttilä <[email protected]>

Reviewed-by Jan Kara <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

---
  mm/gup.c | 41 ++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 40 insertions(+), 1 deletion(-)

mm/gup.c +15 -0

index ff689c88a357..0f09dec0906c 100644
@@ -959,16 +959,51 @@ static int faultin_page(struct vm_area_struct *vma,
  	return 0;
  }
 
+/*
+ * Writing to file-backed mappings which require folio dirty tracking using GUP
+ * is a fundamentally broken operation, as kernel write access to GUP mappings
+ * do not adhere to the semantics expected by a file system.
+ *
+ * Consider the following scenario:-
+ *
+ * 1. A folio is written to via GUP which write-faults the memory, notifying
+ *    the file system and dirtying the folio.
+ * 2. Later, writeback is triggered, resulting in the folio being cleaned and
+ *    the PTE being marked read-only.
+ * 3. The GUP caller writes to the folio, as it is mapped read/write via the
+ *    direct mapping.
+ * 4. The GUP caller, now done with the page, unpins it and sets it dirty
+ *    (though it does not have to).

Quoted 85 lines · Lorenzo Stoakes

Writing to file-backed mappings which require folio dirty tracking using GUP is a fundamentally broken operation, as kernel write access to GUP mappings do not adhere to the semantics expected by a file system.

A GUP caller uses the direct mapping to access the folio, which does not cause write notify to trigger, nor does it enforce that the caller marks the folio dirty.

The problem arises when, after an initial write to the folio, writeback results in the folio being cleaned and then the caller, via the GUP interface, writes to the folio again.

As a result of the use of this secondary, direct, mapping to the folio no write notify will occur, and if the caller does mark the folio dirty, this will be done so unexpectedly.

For example, consider the following scenario:-

1. A folio is written to via GUP which write-faults the memory, notifying the file system and dirtying the folio.
2. Later, writeback is triggered, resulting in the folio being cleaned and the PTE being marked read-only.
3. The GUP caller writes to the folio, as it is mapped read/write via the direct mapping.
4. The GUP caller, now done with the page, unpins it and sets it dirty (though it does not have to).

This results in both data being written to a folio without writenotify, and the folio being dirtied unexpectedly (if the caller decides to do so).

This issue was first reported by Jan Kara [1] in 2018, where the problem resulted in file system crashes.

This is only relevant when the mappings are file-backed and the underlying file system requires folio dirty tracking. File systems which do not, such as shmem or hugetlb, are not at risk and therefore can be written to without issue.

Unfortunately this limitation of GUP has been present for some time and requires future rework of the GUP API in order to provide correct write access to such mappings.

However, for the time being we introduce this check to prevent the most egregious case of this occurring, use of the FOLL_LONGTERM pin.

These mappings are considerably more likely to be written to after folios are cleaned and thus simply must not be permitted to do so.

This patch changes only the slow-path GUP functions, a following patch adapts the GUP-fast path along similar lines.

[1]:https://lore.kernel.org/linux-mm/[email protected]/

Suggested-by Jason Gunthorpe <[email protected]>

Signed-off-by Lorenzo Stoakes <[email protected]>

Reviewed-by John Hubbard <[email protected]>

Reviewed-by Mika Penttilä <[email protected]>

Reviewed-by Jan Kara <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

---
  mm/gup.c | 41 ++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 40 insertions(+), 1 deletion(-)

mm/gup.c +15 -0

index ff689c88a357..0f09dec0906c 100644
@@ -959,16 +959,51 @@ static int faultin_page(struct vm_area_struct *vma,
  	return 0;
  }
+/*
+ * Writing to file-backed mappings which require folio dirty tracking using GUP
+ * is a fundamentally broken operation, as kernel write access to GUP mappings
+ * do not adhere to the semantics expected by a file system.
+ *
+ * Consider the following scenario:-
+ *
+ * 1. A folio is written to via GUP which write-faults the memory, notifying
+ *    the file system and dirtying the folio.
+ * 2. Later, writeback is triggered, resulting in the folio being cleaned and
+ *    the PTE being marked read-only.
+ * 3. The GUP caller writes to the folio, as it is mapped read/write via the
+ *    direct mapping.
+ * 4. The GUP caller, now done with the page, unpins it and sets it dirty
+ *    (though it does not have to).

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 985

Quoted 10 lines

mm/gup.c +10 -0

+ *
+ * This results in both data being written to a folio without writenotify, and
+ * the folio being dirtied unexpectedly (if the caller decides to do so).
+ */
+static bool writeable_file_mapping_allowed(struct vm_area_struct *vma,
+					   unsigned long gup_flags)
+{
+	/* If we aren't pinning then no problematic write can occur. */
+	if (!(gup_flags & (FOLL_GET | FOLL_PIN)))
+		return true;

I think we should really not look at FOLL_GET here. Just check for FOLL_PIN (as said, even FOLL_LONGTERM would be sufficient, but I understand the reasoning to keep it, although I would drop it :P ). It also better matches your comment regarding pinning ...

See the comment in is_valid_gup_args() regarding "LONGTERM can only be specified when pinning". (well, there we also check that FOLL_PIN has to be set ... ;) )

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 993

Quoted 8 lines

mm/gup.c +7 -0

+
+	/* We limit this check to the most egregious case - a long term pin. */
+	if (!(gup_flags & FOLL_LONGTERM))
+		return true;
+
+	/* If the VMA requires dirty tracking then GUP will be problematic. */
+	return vma_needs_dirty_tracking(vma);

... should that be "!vma_needs_dirty_tracking(vma)" ?

If the fs needs dirty tracking, it should be disallowed.

Maybe that explains why it's still working for Matthew in his s390x test. ... or I am too tired and messed up :)

--
Thanks,

David / dhildenb

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

We have access to less information in the fast path as we cannot examine the VMA containing the mapping, however we can determine whether the folio is anonymous and then whitelist known-good mappings - specifically hugetlb and shmem mappings.

While we obtain a stable folio for this check, the mapping might not be, as a truncate could nullify it at any time. Since doing so requires mappings to be zapped, we can synchronise against a TLB shootdown operation.

For some architectures TLB shootdown is synchronised by IPI, against which we are protected as the GUP-fast operation is performed with interrupts disabled. However, other architectures which specify CONFIG_MMU_GATHER_RCU_TABLE_FREE use an RCU lock for this operation.

In these instances, we acquire an RCU lock while performing our checks. If we cannot get a stable mapping, we fall back to the slow path, as otherwise we'd have to walk the page tables again and it's simpler and more effective to just fall back.

It's important to note that there are no APIs allowing users to specify FOLL_FAST_ONLY for a PUP-fast let alone with FOLL_LONGTERM, so we can always rely on the fact that if we fail to pin on the fast path, the code will fall back to the slow path which can perform the more thorough check.

Suggested-by David Hildenbrand <[email protected]>

Suggested-by Kirill A . Shutemov <[email protected]>

Signed-off-by Lorenzo Stoakes <[email protected]>

---
  mm/gup.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
  1 file changed, 85 insertions(+), 2 deletions(-)

mm/gup.c +85 -2

index 0f09dec0906c..431618048a03 100644
@@ -18,6 +18,7 @@
  #include <linux/migrate.h>
  #include <linux/mm_inline.h>
  #include <linux/sched/mm.h>
+#include <linux/shmem_fs.h>
 
  #include <asm/mmu_context.h>
  #include <asm/tlbflush.h>

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
  	return folio;
  }
 
+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);
+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}
+
+static void unlock_rcu(void)
+{
+}
+#endif
+
+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this
+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+
+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);
+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */
+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;
+}
+
  /**
   * try_grab_folio() - Attempt to get or pin a folio.
   * @page:  pointer to page to be grabbed

@@ -123,6 +195,8 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
   */
  struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  {
+	bool is_longterm = flags & FOLL_LONGTERM;
+
  	if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page)))
  		return NULL;
 

@@ -136,8 +210,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  		 * right zone, so fail and let the caller fall back to the slow
  		 * path.
  		 */
-		if (unlikely((flags & FOLL_LONGTERM) &&
-			     !is_longterm_pinnable_page(page)))
+		if (unlikely(is_longterm && !is_longterm_pinnable_page(page)))
  			return NULL;
 
  		/*

@@ -148,6 +221,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  		if (!folio)
  			return NULL;
 
+		/*
+		 * Can this folio be safely pinned? We need to perform this
+		 * check after the folio is stabilised.
+		 */
+		if ((flags & FOLL_WRITE) && is_longterm &&
+		    !folio_longterm_write_pin_allowed(folio)) {
+			folio_put_refs(folio, refs);
+			return NULL;
+		}
+
  		/*
  		 * When pinning a large folio, use an exact count to track it.
  		 *

Hi Lorenzo,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Lorenzo-Stoakes/mm-mmap-separate-writenotify-and-dirty-tracking-logic/20230502-071520
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/dee4f4ad6532b0f94d073da263526de334d5d7e0.1682981880.git.lstoakes%40gmail.com
patch subject: [PATCH v6 3/3] mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings
config: arm-randconfig-r013-20230502 (https://download.01.org/0day-ci/archive/20230502/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project b1465cd49efcbc114a75220b153f5a055ce7911f)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/intel-lab-lkp/linux/commit/602bb9fab888bd9176b8eeb80a0da68499c343ed
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Lorenzo-Stoakes/mm-mmap-separate-writenotify-and-dirty-tracking-logic/20230502-071520 git checkout 602bb9fab888bd9176b8eeb80a0da68499c343ed
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

Quoted 1 line

Quoted 1 line

mm/gup.c:114:49: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]

static bool stabilise_mapping_rcu(struct folio *)

                                                   ^

1 warning generated.

vim +114 mm/gup.c

   108
   109	static void unlock_rcu(void)
   110	{
   111		rcu_read_unlock();
   112	}
   113	#else

Quoted 1 line

114 static bool stabilise_mapping_rcu(struct folio *)

   115	{
   116		return true;
   117	}
   118

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests

mm/gup.c +6 -0

+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}

mm/gup.c +17 -0

+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

(quoted diff) +1 -0

+}

Quoted 5 lines · Lorenzo Stoakes

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 167

Quoted 10 lines

mm/gup.c +10 -0

+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;

then becomes:

	if (folio_test_anon(folio))
		return true;

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*

	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

	return false;

Quoted 4 lines · Lorenzo Stoakes

mm/gup.c +0 -0

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
 	return folio;
 }
 

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 106

Quoted 8 lines

mm/gup.c +8 -0

+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);

This doesn't make sense; why bother reading the same thing twice?

Who cares if the thing changes from before; what you care about is that the value you see has stable storage, this doesn't help with that.

Quoted 11 lines

mm/gup.c +11 -0

+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 122

Quoted 5 lines

mm/gup.c +5 -0

+
+static void unlock_rcu(void)
+{
+}
+#endif

Anyway, this all can go away. RCU can't progress while you have interrupts disabled anyway.

Quoted 18 lines

mm/gup.c +17 -0

+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 150

Quoted 10 lines

mm/gup.c +10 -0

+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+

This:

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 152

Quoted 2 lines

mm/gup.c +2 -0

+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);

And this is 3rd read of folio->mapping, just for giggles?

Quoted 6 lines

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 167

Quoted 10 lines

mm/gup.c +10 -0

+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;

then becomes:

	if (folio_test_anon(folio))
		return true;

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

	return false;

Quoted 1 line

(quoted diff) +1 -0

+}

Quoted 5 lines · Peter Zijlstra

Quoted 4 lines · Lorenzo Stoakes

mm/gup.c +0 -0

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
 	return folio;
 }
 

Quoted 10 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 106

Quoted 8 lines

mm/gup.c +8 -0

+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);

This doesn't make sense; why bother reading the same thing twice?

The intent is to see whether the folio->mapping has been truncated from underneath us, as per the futex code that Kirill referred to which does something similar [1].

Quoted 15 lines

Who cares if the thing changes from before; what you care about is that the value you see has stable storage, this doesn't help with that.

Quoted 11 lines

mm/gup.c +11 -0

+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}

Quoted 8 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 122

Quoted 5 lines

mm/gup.c +5 -0

+
+static void unlock_rcu(void)
+{
+}
+#endif

Anyway, this all can go away. RCU can't progress while you have interrupts disabled anyway.

There seems to be other code in the kernel that assumes that this is not the case, i.e. the futex code, though not sure if that's being run with IRQs disabled... if not and it's absolutely certain that we need no special handling for the RCU case, then happy days and more than glad to remove this bit.

I'm far from an expert on RCU (I need to gain a better understanding of it) so I'm deferring how best to proceed on _this part_ to the community.

Quoted 31 lines

Quoted 18 lines

mm/gup.c +17 -0

+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 150

Quoted 10 lines

mm/gup.c +10 -0

+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+

This:

Quoted 4 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 152

Quoted 2 lines

mm/gup.c +2 -0

+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);

And this is 3rd read of folio->mapping, just for giggles?

I like to giggle :)

Actually this is to handle the various cases in which the mapping might not be what we want (i.e. have PAGE_MAPPING_FLAGS set) which doesn't appear to have a helper exposed for a check. Given previous review about duplication I felt best to reuse this even though it does access again... yes I felt weird about doing that.

Quoted 20 lines

Quoted 6 lines

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 167

Quoted 10 lines

mm/gup.c +10 -0

+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;

then becomes:

Quoted 2 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Quoted 16 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

Quoted 3 lines

	return false;

Quoted 1 line

(quoted diff) +1 -0

+}

I'm more than happy to do this (I'd rather drop the RCU bits if possible) but need to be sure it's safe.

Quoted 5 lines · Peter Zijlstra

Quoted 4 lines · Lorenzo Stoakes

mm/gup.c +0 -0

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
 	return folio;
 }
 

Quoted 10 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 106

Quoted 8 lines

mm/gup.c +8 -0

+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);

This doesn't make sense; why bother reading the same thing twice?

The intent is to see whether the folio->mapping has been truncated from underneath us, as per the futex code that Kirill referred to which does something similar [1].

Quoted 14 lines

Who cares if the thing changes from before; what you care about is that the value you see has stable storage, this doesn't help with that.

Quoted 11 lines

mm/gup.c +11 -0

+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}

Quoted 8 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 122

Quoted 5 lines

mm/gup.c +5 -0

+
+static void unlock_rcu(void)
+{
+}
+#endif

Anyway, this all can go away. RCU can't progress while you have interrupts disabled anyway.

There seems to be other code in the kernel that assumes that this is not the case,

i.e. the futex code, though not sure if that's being run with IRQs disabled...

Quoted 30 lines

Quoted 17 lines

mm/gup.c +17 -0

+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 150

Quoted 10 lines

mm/gup.c +10 -0

+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+

This:

Quoted 4 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 152

Quoted 2 lines

mm/gup.c +2 -0

+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);

And this is 3rd read of folio->mapping, just for giggles?

I like to giggle :)

Actually this is to handle the various cases in which the mapping might not be what we want (i.e. have PAGE_MAPPING_FLAGS set) which doesn't appear to have a helper exposed for a check. Given previous review about duplication I felt best to reuse this even though it does access again... yes I felt weird about doing that.

Quoted 19 lines

Quoted 5 lines

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 167

Quoted 10 lines

mm/gup.c +10 -0

+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;

then becomes:

Quoted 2 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Quoted 16 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

Quoted 3 lines

	return false;

Quoted 1 line

(quoted diff) +1 -0

+}

I'm more than happy to do this (I'd rather drop the RCU bits if possible) but need to be sure it's safe.

Quoted 17 lines · Lorenzo Stoakes

Quoted 15 lines · Peter Zijlstra

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 106

Quoted 12 lines · Lorenzo Stoakes

mm/gup.c +8 -0

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
 	return folio;
 }
 
+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);

This doesn't make sense; why bother reading the same thing twice?

Quoted 3 lines

The intent is to see whether the folio->mapping has been truncated from underneath us, as per the futex code that Kirill referred to which does something similar [1].

Yeah, but per that 3rd load you got nothing here. Also that futex code did the early load to deal with the !mapping case, but you're not doing that.

Quoted 23 lines

Quoted 14 lines

Who cares if the thing changes from before; what you care about is that the value you see has stable storage, this doesn't help with that.

Quoted 11 lines

mm/gup.c +11 -0

+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}

Quoted 8 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 122

Quoted 5 lines

mm/gup.c +5 -0

+
+static void unlock_rcu(void)
+{
+}
+#endif

Anyway, this all can go away. RCU can't progress while you have interrupts disabled anyway.

Quoted 2 lines

There seems to be other code in the kernel that assumes that this is not the case,

Yeah, so Paul went back on forth on that a bit. It used to be true in the good old days when everything was simple. Then Paul made things complicated by separating out sched-RCU bh-RCU and 'regular' RCU flavours.

At that point disabling IRQs would only (officially) inhibit sched and bh RCU flavours, but not the regular RCU.

But then some years ago Linus convinced Paul that having all these separate RCU flavours with separate QS rules was a big pain in the backside and Paul munged them all together again.

So now, anything that inhibits any of the RCU flavours inhibits them all. So disabling IRQs is sufficient.

Quoted 2 lines

i.e. the futex code, though not sure if that's being run with IRQs disabled...

That futex code runs in preemptible context, per the lock_page() that can sleep etc.. :-)

Quoted 36 lines

Quoted 31 lines

Quoted 18 lines

mm/gup.c +17 -0

+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 150

Quoted 10 lines

mm/gup.c +10 -0

+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+

This:

Quoted 4 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 152

Quoted 2 lines

mm/gup.c +2 -0

+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);

And this is 3rd read of folio->mapping, just for giggles?

Quoted 7 lines

I like to giggle :)

Actually this is to handle the various cases in which the mapping might not be what we want (i.e. have PAGE_MAPPING_FLAGS set) which doesn't appear to have a helper exposed for a check. Given previous review about duplication I felt best to reuse this even though it does access again... yes I felt weird about doing that.

Right, I had a peek inside folio_mapping(), but the point is that this 3rd load might see yet *another* value of mapping from the prior two loads, rendering them somewhat worthless.

Quoted 19 lines

Quoted 19 lines

Quoted 5 lines

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 167

Quoted 10 lines

mm/gup.c +10 -0

+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;

then becomes:

Quoted 4 lines

Quoted 2 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Oh, right you are.

Quoted 16 lines

Quoted 16 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

Quoted 6 lines

Quoted 3 lines

	return false;

Quoted 1 line

(quoted diff) +1 -0

+}

I'm more than happy to do this (I'd rather drop the RCU bits if possible) but need to be sure it's safe.

GUP-fast as a whole relies on it :-)

Quoted 6 lines

Quoted 4 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Oh, right you are.

Quoted 19 lines

Quoted 19 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

	return false;

Quoted 1 line

+}

Quoted 25 lines · Peter Zijlstra

Quoted 6 lines

Quoted 4 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Oh, right you are.

Quoted 16 lines

Quoted 16 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

	return false;

Quoted 1 line

+}

So arguably you should do *one* READ_ONCE() load of mapping and consistently use that, this means open-coding both folio_test_anon() and folio_mapping().

Quoted 31 lines · Peter Zijlstra

Quoted 28 lines · Peter Zijlstra

Quoted 6 lines

Quoted 4 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Oh, right you are.

Quoted 19 lines

Quoted 19 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

	return false;

Quoted 1 line

+}

Quoted 3 lines

So arguably you should do *one* READ_ONCE() load of mapping and consistently use that, this means open-coding both folio_test_anon() and folio_mapping().

Open-coding folio_test_anon() should not be required. We only care about PAGE_MAPPING_FLAGS stored alongside folio->mapping, that will stick around until the anon page was freed.

@Lorenzo, you might also want to special-case hugetlb directly using folio_test_hugetlb().

--
Thanks,

David / dhildenb

Quoted 32 lines · David Hildenbrand

Quoted 31 lines · Peter Zijlstra

Quoted 28 lines · Peter Zijlstra

Quoted 6 lines

Quoted 4 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Oh, right you are.

Quoted 19 lines

Quoted 19 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

	return false;

Quoted 1 line

+}

Quoted 8 lines

Quoted 3 lines

So arguably you should do *one* READ_ONCE() load of mapping and consistently use that, this means open-coding both folio_test_anon() and folio_mapping().

Open-coding folio_test_anon() should not be required. We only care about PAGE_MAPPING_FLAGS stored alongside folio->mapping, that will stick around until the anon page was freed.

Ack, good point!

Quoted 3 lines

@Lorenzo, you might also want to special-case hugetlb directly using folio_test_hugetlb().

I already am :) I guess you mean when I respin along these lines? Will port that across to.

Quoted 17 lines · Lorenzo Stoakes

Quoted 5 lines · Peter Zijlstra

Quoted 4 lines · Lorenzo Stoakes

mm/gup.c +0 -0

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
 	return folio;
 }
 

Quoted 10 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 106

Quoted 8 lines

mm/gup.c +8 -0

+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);

This doesn't make sense; why bother reading the same thing twice?

Quoted 3 lines

The intent is to see whether the folio->mapping has been truncated from underneath us, as per the futex code that Kirill referred to which does something similar [1].

Yeah, but per that 3rd load you got nothing here. Also that futex code did the early load to deal with the !mapping case, but you're not doing that.

Quoted 23 lines

Quoted 22 lines

Who cares if the thing changes from before; what you care about is that the value you see has stable storage, this doesn't help with that.

Quoted 6 lines

mm/gup.c +6 -0

+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 122

Quoted 10 lines

mm/gup.c +10 -0

+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}
+
+static void unlock_rcu(void)
+{
+}
+#endif

Anyway, this all can go away. RCU can't progress while you have interrupts disabled anyway.

Quoted 2 lines

There seems to be other code in the kernel that assumes that this is not the case,

Yeah, so Paul went back on forth on that a bit. It used to be true in the good old days when everything was simple. Then Paul made things complicated by separating out sched-RCU bh-RCU and 'regular' RCU flavours.

At that point disabling IRQs would only (officially) inhibit sched and bh RCU flavours, but not the regular RCU.

But then some years ago Linus convinced Paul that having all these separate RCU flavours with separate QS rules was a big pain in the backside and Paul munged them all together again.

So now, anything that inhibits any of the RCU flavours inhibits them all. So disabling IRQs is sufficient.

Quoted 2 lines

i.e. the futex code, though not sure if that's being run with IRQs disabled...

That futex code runs in preemptible context, per the lock_page() that can sleep etc.. :-)

Quoted 35 lines

Quoted 30 lines

Quoted 17 lines

mm/gup.c +17 -0

+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 150

Quoted 10 lines

mm/gup.c +10 -0

+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+

This:

Quoted 4 lines

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 152

Quoted 2 lines

mm/gup.c +2 -0

+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);

And this is 3rd read of folio->mapping, just for giggles?

Quoted 7 lines

I like to giggle :)

Actually this is to handle the various cases in which the mapping might not be what we want (i.e. have PAGE_MAPPING_FLAGS set) which doesn't appear to have a helper exposed for a check. Given previous review about duplication I felt best to reuse this even though it does access again... yes I felt weird about doing that.

Right, I had a peek inside folio_mapping(), but the point is that this 3rd load might see yet *another* value of mapping from the prior two loads, rendering them somewhat worthless.

Quoted 19 lines

Quoted 19 lines

Quoted 5 lines

mm/gup.c +5 -0

+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */

mm/gup.c static inline struct folio *try_get_folio(struct page *page, int refs) · line 167

Quoted 10 lines

mm/gup.c +10 -0

+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;

then becomes:

Quoted 4 lines

Quoted 2 lines

	if (folio_test_anon(folio))
		return true;

This relies on the mapping so belongs below the lockdep assert imo.

Oh, right you are.

Quoted 16 lines

Quoted 16 lines

	/*
	 * Having IRQs disabled (as per GUP-fast) also inhibits RCU
	 * grace periods from making progress, IOW. they imply
	 * rcu_read_lock().
	 */
	lockdep_assert_irqs_disabled();

	/*
	 * Inodes and thus address_space are RCU freed and thus safe to
	 * access at this point.
	 */
	mapping = folio_mapping(folio);
	if (mapping && shmem_mapping(mapping))
		return true;

Quoted 6 lines

Quoted 3 lines

	return false;

Quoted 1 line

(quoted diff) +1 -0

+}

I'm more than happy to do this (I'd rather drop the RCU bits if possible) but need to be sure it's safe.

GUP-fast as a whole relies on it :-)

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

We have access to less information in the fast path as we cannot examine the VMA containing the mapping, however we can determine whether the folio is anonymous and then whitelist known-good mappings - specifically hugetlb and shmem mappings.

While we obtain a stable folio for this check, the mapping might not be, as a truncate could nullify it at any time. Since doing so requires mappings to be zapped, we can synchronise against a TLB shootdown operation.

For some architectures TLB shootdown is synchronised by IPI, against which we are protected as the GUP-fast operation is performed with interrupts disabled. However, other architectures which specify CONFIG_MMU_GATHER_RCU_TABLE_FREE use an RCU lock for this operation.

In these instances, we acquire an RCU lock while performing our checks. If we cannot get a stable mapping, we fall back to the slow path, as otherwise we'd have to walk the page tables again and it's simpler and more effective to just fall back.

It's important to note that there are no APIs allowing users to specify FOLL_FAST_ONLY for a PUP-fast let alone with FOLL_LONGTERM, so we can always rely on the fact that if we fail to pin on the fast path, the code will fall back to the slow path which can perform the more thorough check.

Suggested-by David Hildenbrand <[email protected]>

Suggested-by Kirill A . Shutemov <[email protected]>

Signed-off-by Lorenzo Stoakes <[email protected]>

---
 mm/gup.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 85 insertions(+), 2 deletions(-)

mm/gup.c +15 -0

index 0f09dec0906c..431618048a03 100644
@@ -18,6 +18,7 @@
 #include <linux/migrate.h>
 #include <linux/mm_inline.h>
 #include <linux/sched/mm.h>
+#include <linux/shmem_fs.h>
 
 #include <asm/mmu_context.h>
 #include <asm/tlbflush.h>

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
 	return folio;
 }
 
+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);
+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}

mm/gup.c +60 -2

+
+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this
+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+
+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);
+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */
+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;
+}
+
 /**
  * try_grab_folio() - Attempt to get or pin a folio.
  * @page:  pointer to page to be grabbed

@@ -123,6 +195,8 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
  */
 struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
 {
+	bool is_longterm = flags & FOLL_LONGTERM;
+
 	if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page)))
 		return NULL;
 

@@ -136,8 +210,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
 		 * right zone, so fail and let the caller fall back to the slow
 		 * path.
 		 */
-		if (unlikely((flags & FOLL_LONGTERM) &&
-			     !is_longterm_pinnable_page(page)))
+		if (unlikely(is_longterm && !is_longterm_pinnable_page(page)))
 			return NULL;
 
 		/*

@@ -148,6 +221,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
 		if (!folio)
 			return NULL;
 
+		/*
+		 * Can this folio be safely pinned? We need to perform this
+		 * check after the folio is stabilised.
+		 */
+		if ((flags & FOLL_WRITE) && is_longterm &&
+		    !folio_longterm_write_pin_allowed(folio)) {
+			folio_put_refs(folio, refs);
+			return NULL;
+		}
+
 		/*
 		 * When pinning a large folio, use an exact count to track it.
 		 *

Signature · 1 line

2.40.1

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

We have access to less information in the fast path as we cannot examine the VMA containing the mapping, however we can determine whether the folio is anonymous and then whitelist known-good mappings - specifically hugetlb and shmem mappings.

While we obtain a stable folio for this check, the mapping might not be, as a truncate could nullify it at any time. Since doing so requires mappings to be zapped, we can synchronise against a TLB shootdown operation.

For some architectures TLB shootdown is synchronised by IPI, against which we are protected as the GUP-fast operation is performed with interrupts disabled. However, other architectures which specify CONFIG_MMU_GATHER_RCU_TABLE_FREE use an RCU lock for this operation.

In these instances, we acquire an RCU lock while performing our checks. If we cannot get a stable mapping, we fall back to the slow path, as otherwise we'd have to walk the page tables again and it's simpler and more effective to just fall back.

It's important to note that there are no APIs allowing users to specify FOLL_FAST_ONLY for a PUP-fast let alone with FOLL_LONGTERM, so we can always rely on the fact that if we fail to pin on the fast path, the code will fall back to the slow path which can perform the more thorough check.

Suggested-by David Hildenbrand <[email protected]>

Suggested-by Kirill A . Shutemov <[email protected]>

Signed-off-by Lorenzo Stoakes <[email protected]>

---
  mm/gup.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
  1 file changed, 85 insertions(+), 2 deletions(-)

mm/gup.c +85 -2

index 0f09dec0906c..431618048a03 100644
@@ -18,6 +18,7 @@
  #include <linux/migrate.h>
  #include <linux/mm_inline.h>
  #include <linux/sched/mm.h>
+#include <linux/shmem_fs.h>
 
  #include <asm/mmu_context.h>
  #include <asm/tlbflush.h>

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
  	return folio;
  }
 
+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);
+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}
+
+static void unlock_rcu(void)
+{
+}
+#endif
+
+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this
+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+
+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);
+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */
+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;
+}
+
  /**
   * try_grab_folio() - Attempt to get or pin a folio.
   * @page:  pointer to page to be grabbed

@@ -123,6 +195,8 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
   */
  struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  {
+	bool is_longterm = flags & FOLL_LONGTERM;
+
  	if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page)))
  		return NULL;
 

@@ -136,8 +210,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  		 * right zone, so fail and let the caller fall back to the slow
  		 * path.
  		 */
-		if (unlikely((flags & FOLL_LONGTERM) &&
-			     !is_longterm_pinnable_page(page)))
+		if (unlikely(is_longterm && !is_longterm_pinnable_page(page)))
  			return NULL;
 
  		/*

@@ -148,6 +221,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  		if (!folio)
  			return NULL;
 
+		/*
+		 * Can this folio be safely pinned? We need to perform this
+		 * check after the folio is stabilised.
+		 */
+		if ((flags & FOLL_WRITE) && is_longterm &&
+		    !folio_longterm_write_pin_allowed(folio)) {
+			folio_put_refs(folio, refs);
+			return NULL;
+		}
+
  		/*
  		 * When pinning a large folio, use an exact count to track it.
  		 *

Am 02.05.23 um 01:11 schrieb Lorenzo Stoakes:

Quoted 7 lines

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

Hmm, does this interfer with KVM on s390 and PCI interpretion of interrupt delivery? It would no longer work with file backed memory, correct?

See
arch/s390/kvm/pci.c

kvm_s390_pci_aif_enable
which does have
FOLL_WRITE | FOLL_LONGTERM
to

Quoted 153 lines

We have access to less information in the fast path as we cannot examine the VMA containing the mapping, however we can determine whether the folio is anonymous and then whitelist known-good mappings - specifically hugetlb and shmem mappings.

While we obtain a stable folio for this check, the mapping might not be, as a truncate could nullify it at any time. Since doing so requires mappings to be zapped, we can synchronise against a TLB shootdown operation.

For some architectures TLB shootdown is synchronised by IPI, against which we are protected as the GUP-fast operation is performed with interrupts disabled. However, other architectures which specify CONFIG_MMU_GATHER_RCU_TABLE_FREE use an RCU lock for this operation.

In these instances, we acquire an RCU lock while performing our checks. If we cannot get a stable mapping, we fall back to the slow path, as otherwise we'd have to walk the page tables again and it's simpler and more effective to just fall back.

It's important to note that there are no APIs allowing users to specify FOLL_FAST_ONLY for a PUP-fast let alone with FOLL_LONGTERM, so we can always rely on the fact that if we fail to pin on the fast path, the code will fall back to the slow path which can perform the more thorough check.

Suggested-by David Hildenbrand <[email protected]>

Suggested-by Kirill A . Shutemov <[email protected]>

Signed-off-by Lorenzo Stoakes <[email protected]>

---
  mm/gup.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
  1 file changed, 85 insertions(+), 2 deletions(-)

mm/gup.c +85 -2

index 0f09dec0906c..431618048a03 100644
@@ -18,6 +18,7 @@
  #include <linux/migrate.h>
  #include <linux/mm_inline.h>
  #include <linux/sched/mm.h>
+#include <linux/shmem_fs.h>
  #include <asm/mmu_context.h>
  #include <asm/tlbflush.h>

@@ -95,6 +96,77 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
  	return folio;
  }
+#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
+static bool stabilise_mapping_rcu(struct folio *folio)
+{
+	struct address_space *mapping = READ_ONCE(folio->mapping);
+
+	rcu_read_lock();
+
+	return mapping == READ_ONCE(folio->mapping);
+}
+
+static void unlock_rcu(void)
+{
+	rcu_read_unlock();
+}
+#else
+static bool stabilise_mapping_rcu(struct folio *)
+{
+	return true;
+}
+
+static void unlock_rcu(void)
+{
+}
+#endif
+
+/*
+ * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM |
+ * FOLL_WRITE pin is permitted for a specific folio.
+ *
+ * This assumes the folio is stable and pinned.
+ *
+ * Writing to pinned file-backed dirty tracked folios is inherently problematic
+ * (see comment describing the writeable_file_mapping_allowed() function). We
+ * therefore try to avoid the most egregious case of a long-term mapping doing
+ * so.
+ *
+ * This function cannot be as thorough as that one as the VMA is not available
+ * in the fast path, so instead we whitelist known good cases.
+ *
+ * The folio is stable, but the mapping might not be. When truncating for
+ * instance, a zap is performed which triggers TLB shootdown. IRQs are disabled
+ * so we are safe from an IPI, but some architectures use an RCU lock for this
+ * operation, so we acquire an RCU lock to ensure the mapping is stable.
+ */
+static bool folio_longterm_write_pin_allowed(struct folio *folio)
+{
+	bool ret;
+
+	/* hugetlb mappings do not require dirty tracking. */
+	if (folio_test_hugetlb(folio))
+		return true;
+
+	if (stabilise_mapping_rcu(folio)) {
+		struct address_space *mapping = folio_mapping(folio);
+
+		/*
+		 * Neither anonymous nor shmem-backed folios require
+		 * dirty tracking.
+		 */
+		ret = folio_test_anon(folio) ||
+			(mapping && shmem_mapping(mapping));
+	} else {
+		/* If the mapping is unstable, fallback to the slow path. */
+		ret = false;
+	}
+
+	unlock_rcu();
+
+	return ret;
+}
+
  /**
   * try_grab_folio() - Attempt to get or pin a folio.
   * @page:  pointer to page to be grabbed

@@ -123,6 +195,8 @@ static inline struct folio *try_get_folio(struct page *page, int refs)
   */
  struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  {
+	bool is_longterm = flags & FOLL_LONGTERM;
+
  	if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page)))
  		return NULL;

@@ -136,8 +210,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  		 * right zone, so fail and let the caller fall back to the slow
  		 * path.
  		 */
-		if (unlikely((flags & FOLL_LONGTERM) &&
-			     !is_longterm_pinnable_page(page)))
+		if (unlikely(is_longterm && !is_longterm_pinnable_page(page)))
  			return NULL;
  		/*

@@ -148,6 +221,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
  		if (!folio)
  			return NULL;
+		/*
+		 * Can this folio be safely pinned? We need to perform this
+		 * check after the folio is stabilised.
+		 */
+		if ((flags & FOLL_WRITE) && is_longterm &&
+		    !folio_longterm_write_pin_allowed(folio)) {
+			folio_put_refs(folio, refs);
+			return NULL;
+		}
+
  		/*
  		 * When pinning a large folio, use an exact count to track it.
  		 *

Quoted 12 lines · Christian Borntraeger

Am 02.05.23 um 01:11 schrieb Lorenzo Stoakes:

Quoted 7 lines

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

Hmm, does this interfer with KVM on s390 and PCI interpretion of interrupt delivery? It would no longer work with file backed memory, correct?

Quoted 8 lines

See
arch/s390/kvm/pci.c

kvm_s390_pci_aif_enable
which does have
FOLL_WRITE | FOLL_LONGTERM
to

Does this memory map a dirty-tracked file? It's kind of hard to dig into where the address originates from without going through a ton of code. In worst case if the fast code doesn't find a whitelist it'll fall back to slow path which explicitly checks for dirty-tracked filesystem.

Quoted 12 lines

Am 02.05.23 um 01:11 schrieb Lorenzo Stoakes:

Quoted 7 lines

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

Hmm, does this interfer with KVM on s390 and PCI interpretion of interrupt delivery? It would no longer work with file backed memory, correct?

Quoted 8 lines

See
arch/s390/kvm/pci.c

kvm_s390_pci_aif_enable
which does have
FOLL_WRITE | FOLL_LONGTERM
to

Does this memory map a dirty-tracked file? It's kind of hard to dig into where the address originates from without going through a ton of code. In worst case if the fast code doesn't find a whitelist it'll fall back to slow path which explicitly checks for dirty-tracked filesystem.

We can reintroduce a flag to permit exceptions if this is really broken, are you able to test? I don't have an s390 sat around :)

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having
<memoryBacking>

    <source type="file"/>

</memoryBacking>

On Tue, May 02, 2023 at 03:04:27PM +0200, Christian Borntraeger wrote:
\> > We can reintroduce a flag to permit exceptions if this is really broken, are you

Quoted 6 lines

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having
<memoryBacking>

     <source type="file"/>

</memoryBacking>

This s390 code is the least of the problems, after this series VFIO won't startup at all with this configuration.

Quoted 9 lines · Jason Gunthorpe

On Tue, May 02, 2023 at 03:04:27PM +0200, Christian Borntraeger wrote:
\> > We can reintroduce a flag to permit exceptions if this is really broken, are you

Quoted 6 lines

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having
<memoryBacking>

     <source type="file"/>

</memoryBacking>

Quoted 2 lines

This s390 code is the least of the problems, after this series VFIO won't startup at all with this configuration.

Good question if the domain would fail to start. I recall that IOMMUs for zPCI are special on s390x. [1]

Well, zPCI is special. I cannot immediately tell when we would trigger long-term pinning.

[1] https://www.mail-archive.com/[email protected]/msg875728.html

Quoted 10 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

On Tue, May 02, 2023 at 03:04:27PM +0200, Christian Borntraeger wrote:
\> > We can reintroduce a flag to permit exceptions if this is really broken, are you

Quoted 6 lines

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having

    <memoryBacking>
      <source type="file"/>
    </memoryBacking>

Quoted 5 lines

Quoted 2 lines

This s390 code is the least of the problems, after this series VFIO won't startup at all with this configuration.

Good question if the domain would fail to start. I recall that IOMMUs for zPCI are special on s390x. [1]

Not upstream they aren't.

Quoted 2 lines

Well, zPCI is special. I cannot immediately tell when we would trigger long-term pinning.

zPCI uses the standard IOMMU stuff, so it uses a normal VFIO container and the normal pin_user_pages() path.

Quoted 19 lines · Jason Gunthorpe

Quoted 10 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

On Tue, May 02, 2023 at 03:04:27PM +0200, Christian Borntraeger wrote:
\> > We can reintroduce a flag to permit exceptions if this is really broken, are you

Quoted 6 lines

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having <memoryBacking>
      <source type="file"/>
    </memoryBacking>

Quoted 5 lines

Quoted 2 lines

This s390 code is the least of the problems, after this series VFIO won't startup at all with this configuration.

Good question if the domain would fail to start. I recall that IOMMUs for zPCI are special on s390x. [1]

Not upstream they aren't.

Quoted 5 lines

Quoted 2 lines

Well, zPCI is special. I cannot immediately tell when we would trigger long-term pinning.

zPCI uses the standard IOMMU stuff, so it uses a normal VFIO container and the normal pin_user_pages() path.

@Christian, Matthew: would we pin all guest memory when starting the domain (IIRC, like on x86-64) and fail early, or only when the guest issues rpcit instructions to map individual pages?

Quoted 23 lines · David Hildenbrand

Quoted 22 lines · Jason Gunthorpe

Quoted 10 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

On Tue, May 02, 2023 at 03:04:27PM +0200, Christian Borntraeger wrote:
\> > We can reintroduce a flag to permit exceptions if this is really broken, are you

Quoted 6 lines

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having
    <memoryBacking>
      <source type="file"/>
    </memoryBacking>

Quoted 5 lines

Quoted 2 lines

This s390 code is the least of the problems, after this series VFIO won't startup at all with this configuration.

Good question if the domain would fail to start. I recall that IOMMUs for zPCI are special on s390x. [1]

Not upstream they aren't.

Quoted 2 lines

Well, zPCI is special. I cannot immediately tell when we would trigger long-term pinning.

Quoted 6 lines

Quoted 2 lines

zPCI uses the standard IOMMU stuff, so it uses a normal VFIO container and the normal pin_user_pages() path.

@Christian, Matthew: would we pin all guest memory when starting the domain (IIRC, like on x86-64) and fail early, or only when the guest issues rpcit instructions to map individual pages?

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

However, per Jason's prior suggestion, the initial implementation for s390 nesting via iommufd will pin all of guest memory when starting the domain. I have something already working via iommufd built on top of the nesting infrastructure patches and QEMU iommufd series that are floating around; needs some cleanup, hoping to send an RFC in the coming weeks. I can CC you if you'd like.

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

Quoted 9 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 5 lines

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

IIUC, that pinning is not for ordinary IOMMU / KVM memory access. It's for passthrough of (adapter) interrupts.

I have to speculate, but I guess for hardware to forward interrupts to the VM, it has to pin the special guest memory page that will receive the indications, to then configure (interrupt) hardware to target the interrupt indications to that special guest page (using a host physical address).

Quoted 16 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 5 lines

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

Quoted 7 lines

IIUC, that pinning is not for ordinary IOMMU / KVM memory access. It's for passthrough of (adapter) interrupts.

I have to speculate, but I guess for hardware to forward interrupts to the VM, it has to pin the special guest memory page that will receive the indications, to then configure (interrupt) hardware to target the interrupt indications to that special guest page (using a host physical address).

Either the emulated access is "CPU" based happening through the KVM page table so it should use mmu_notifier locking.

Or it is "DMA" and should go through an IOVA through iommufd pinning and locking.

There is no other ground, nothing in KVM should be inventing its own access methodology.

Quoted 31 lines · Jason Gunthorpe

Quoted 16 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 5 lines

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

Quoted 7 lines

IIUC, that pinning is not for ordinary IOMMU / KVM memory access. It's for passthrough of (adapter) interrupts.

I have to speculate, but I guess for hardware to forward interrupts to the VM, it has to pin the special guest memory page that will receive the indications, to then configure (interrupt) hardware to target the interrupt indications to that special guest page (using a host physical address).

Either the emulated access is "CPU" based happening through the KVM page table so it should use mmu_notifier locking.

Or it is "DMA" and should go through an IOVA through iommufd pinning and locking.

Quoted 2 lines

There is no other ground, nothing in KVM should be inventing its own access methodology.

I might be wrong, but this seems to be a bit different.

It cannot tolerate page faults (needs a host physical address), so memory notifiers don't really apply. (as a side note, KVM on s390x does not use mmu notifiers as we know them)

It's kind-of like DMA, but it's not really DMA.  It's the CPU delivering interrupts for a specific device. So we're configuring the interrupt controller I guess to target a guest memory page.

But I have way too little knowledge about zPCI and the code in question here. And if it could be converted to iommufd (and if that's really the right mechanism to use here).

Hopefully Matthew knows the details and if this really needs to be special :)

Quoted 32 lines · David Hildenbrand

Quoted 31 lines · Jason Gunthorpe

Quoted 16 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 5 lines

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

Quoted 7 lines

IIUC, that pinning is not for ordinary IOMMU / KVM memory access. It's for passthrough of (adapter) interrupts.

I have to speculate, but I guess for hardware to forward interrupts to the VM, it has to pin the special guest memory page that will receive the indications, to then configure (interrupt) hardware to target the interrupt indications to that special guest page (using a host physical address).

Either the emulated access is "CPU" based happening through the KVM page table so it should use mmu_notifier locking.

Or it is "DMA" and should go through an IOVA through iommufd pinning and locking.

Quoted 8 lines

Quoted 2 lines

There is no other ground, nothing in KVM should be inventing its own access methodology.

I might be wrong, but this seems to be a bit different.

It cannot tolerate page faults (needs a host physical address), so memory notifiers don't really apply. (as a side note, KVM on s390x does not use mmu notifiers as we know them)

The host physical address is one shared between underlying firmware and the host kvm. Either might make changes to the referenced page and then issue an alert to the guest via a mechanism called GISA, giving impetus to the guest to look at that page and process the event. As you say, firmware can't tolerate the page being unavailable; it's expecting that once we feed it that location it's always available until we remove it (kvm_s390_pci_aif_disable).

Quoted 42 lines · Matthew Rosato

Quoted 32 lines · David Hildenbrand

Quoted 31 lines · Jason Gunthorpe

Quoted 16 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 5 lines

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

Quoted 7 lines

IIUC, that pinning is not for ordinary IOMMU / KVM memory access. It's for passthrough of (adapter) interrupts.

I have to speculate, but I guess for hardware to forward interrupts to the VM, it has to pin the special guest memory page that will receive the indications, to then configure (interrupt) hardware to target the interrupt indications to that special guest page (using a host physical address).

Either the emulated access is "CPU" based happening through the KVM page table so it should use mmu_notifier locking.

Or it is "DMA" and should go through an IOVA through iommufd pinning and locking.

Quoted 8 lines

Quoted 2 lines

There is no other ground, nothing in KVM should be inventing its own access methodology.

I might be wrong, but this seems to be a bit different.

It cannot tolerate page faults (needs a host physical address), so memory notifiers don't really apply. (as a side note, KVM on s390x does not use mmu notifiers as we know them)

Quoted 7 lines

The host physical address is one shared between underlying firmware and the host kvm. Either might make changes to the referenced page and then issue an alert to the guest via a mechanism called GISA, giving impetus to the guest to look at that page and process the event. As you say, firmware can't tolerate the page being unavailable; it's expecting that once we feed it that location it's always available until we remove it (kvm_s390_pci_aif_disable).

That is a CPU access delegated to the FW without any locking scheme to make it safe with KVM :\

It would have been better if FW could inject it through the kvm page tables so it has some coherency.

Otherwise you have to call this "DMA", I think.

How does s390 avoid mmu notifiers without having lots of problems?? It is not really optional to hook the invalidations if you need to build a shadow page table..

Quoted 3 lines

How does s390 avoid mmu notifiers without having lots of problems?? It is not really optional to hook the invalidations if you need to build a shadow page table..

Totally no idea on s390 details, but.. per my read above, if the firmware needs to make sure the page is always available (so no way to fault it in on demand), which means a longterm pinning seems appropriate here.

Then if pinned a must, there's no need for mmu notifiers (as the page will simply not be invalidated anyway)?

Quoted 8 lines · Peter Xu

Quoted 3 lines

How does s390 avoid mmu notifiers without having lots of problems?? It is not really optional to hook the invalidations if you need to build a shadow page table..

Totally no idea on s390 details, but.. per my read above, if the firmware needs to make sure the page is always available (so no way to fault it in on demand), which means a longterm pinning seems appropriate here.

Quoted 2 lines

Then if pinned a must, there's no need for mmu notifiers (as the page will simply not be invalidated anyway)?

And what if someone deliberately changes the mapping? memory hotplug in the VM, or whatever?

Quoted 9 lines · Jason Gunthorpe

Quoted 8 lines · Peter Xu

Quoted 3 lines

How does s390 avoid mmu notifiers without having lots of problems?? It is not really optional to hook the invalidations if you need to build a shadow page table..

Totally no idea on s390 details, but.. per my read above, if the firmware needs to make sure the page is always available (so no way to fault it in on demand), which means a longterm pinning seems appropriate here.

Quoted 5 lines

Quoted 2 lines

Then if pinned a must, there's no need for mmu notifiers (as the page will simply not be invalidated anyway)?

And what if someone deliberately changes the mapping? memory hotplug in the VM, or whatever?

Besides s390 not supporting memory hotplug in VMs (yet): if the guest wants a different guest physical address, I guess that's the problem of the guest, and it can update it:

KVM_S390_ZPCIOP_REG_AEN is triggered from QEMU via s390_pci_kvm_aif_enable(), triggered by the guest via a special instruction.

If the hypervisor changes the mapping, it's just the same thing as mixing e.g. MADV_DONTNEED with longterm pinning in vfio: don't do it. And if you do it, you get to keep the mess you created for your VM.

Linux will make sure to not change the mapping: for example, page migration of a pinned page will fail.

But maybe I am missing something important here.

Quoted 30 lines · David Hildenbrand

Quoted 9 lines · Jason Gunthorpe

Quoted 8 lines · Peter Xu

Quoted 3 lines

How does s390 avoid mmu notifiers without having lots of problems?? It is not really optional to hook the invalidations if you need to build a shadow page table..

Totally no idea on s390 details, but.. per my read above, if the firmware needs to make sure the page is always available (so no way to fault it in on demand), which means a longterm pinning seems appropriate here.

Quoted 5 lines

Quoted 2 lines

Then if pinned a must, there's no need for mmu notifiers (as the page will simply not be invalidated anyway)?

And what if someone deliberately changes the mapping? memory hotplug in the VM, or whatever?

Besides s390 not supporting memory hotplug in VMs (yet): if the guest wants a different guest physical address, I guess that's the problem of the guest, and it can update it:

KVM_S390_ZPCIOP_REG_AEN is triggered from QEMU via s390_pci_kvm_aif_enable(), triggered by the guest via a special instruction.

If the hypervisor changes the mapping, it's just the same thing as mixing e.g. MADV_DONTNEED with longterm pinning in vfio: don't do it. And if you do it, you get to keep the mess you created for your VM.

Linux will make sure to not change the mapping: for example, page migration of a pinned page will fail.

Quoted 1 line

But maybe I am missing something important here.

It missses the general architectural point why we have all these shootdown mechanims in other places - plares are not supposed to make these kinds of assumptions. When the userspace unplugs the memory from KVM or unmaps it from VFIO it is not still being accessed by the kernel.

Functional bug or not, it is inconsistent with how this is designed to work.

Quoted 5 lines

It missses the general architectural point why we have all these shootdown mechanims in other places - plares are not supposed to make these kinds of assumptions. When the userspace unplugs the memory from KVM or unmaps it from VFIO it is not still being accessed by the kernel.

Yes. Like having memory in a vfio iommu v1 and doing the same (mremap, munmap, MADV_DONTNEED, ...). Which is why we disable MADV_DONTNEED (e.g., virtio-balloon) in QEMU with vfio.

Quoted 9 lines · David Hildenbrand

Quoted 5 lines

It missses the general architectural point why we have all these shootdown mechanims in other places - plares are not supposed to make these kinds of assumptions. When the userspace unplugs the memory from KVM or unmaps it from VFIO it is not still being accessed by the kernel.

Yes. Like having memory in a vfio iommu v1 and doing the same (mremap, munmap, MADV_DONTNEED, ...). Which is why we disable MADV_DONTNEED (e.g., virtio-balloon) in QEMU with vfio.

That is different, VFIO has it's own contract how it consumes the memory from the MM and VFIO breaks all this stuff.

But when you tell VFIO to unmap the memory it doesn't keep accessing it in the background like this does.

Quoted 12 lines · Jason Gunthorpe

Quoted 9 lines · David Hildenbrand

Quoted 5 lines

It missses the general architectural point why we have all these shootdown mechanims in other places - plares are not supposed to make these kinds of assumptions. When the userspace unplugs the memory from KVM or unmaps it from VFIO it is not still being accessed by the kernel.

Yes. Like having memory in a vfio iommu v1 and doing the same (mremap, munmap, MADV_DONTNEED, ...). Which is why we disable MADV_DONTNEED (e.g., virtio-balloon) in QEMU with vfio.

Quoted 5 lines

That is different, VFIO has it's own contract how it consumes the memory from the MM and VFIO breaks all this stuff.

But when you tell VFIO to unmap the memory it doesn't keep accessing it in the background like this does.

To me, this is similar to when QEMU (user space) triggers KVM_S390_ZPCIOP_DEREG_AEN, to tell KVM to disable AIF and stop using the page (1) When triggered by the guest explicitly (2) when resetting the VM
(3) when resetting the virtual PCI device / configuration.

Interrupt gets unregistered from HW (which stops using the page), the pages get unpinned. Pages get no longer used.

I guess I am still missing (a) how this is fundamentally different (b) how it could be done differently.

I'd really be happy to learn how a better approach would look like that does not use longterm pinnings.

Quoted 24 lines · David Hildenbrand

Quoted 12 lines · Jason Gunthorpe

Quoted 9 lines · David Hildenbrand

Quoted 5 lines

It missses the general architectural point why we have all these shootdown mechanims in other places - plares are not supposed to make these kinds of assumptions. When the userspace unplugs the memory from KVM or unmaps it from VFIO it is not still being accessed by the kernel.

Yes. Like having memory in a vfio iommu v1 and doing the same (mremap, munmap, MADV_DONTNEED, ...). Which is why we disable MADV_DONTNEED (e.g., virtio-balloon) in QEMU with vfio.

Quoted 5 lines

That is different, VFIO has it's own contract how it consumes the memory from the MM and VFIO breaks all this stuff.

But when you tell VFIO to unmap the memory it doesn't keep accessing it in the background like this does.

To me, this is similar to when QEMU (user space) triggers KVM_S390_ZPCIOP_DEREG_AEN, to tell KVM to disable AIF and stop using the page (1) When triggered by the guest explicitly (2) when resetting the VM
(3) when resetting the virtual PCI device / configuration.

Quoted 5 lines

Interrupt gets unregistered from HW (which stops using the page), the pages get unpinned. Pages get no longer used.

I guess I am still missing (a) how this is fundamentally different (b) how it could be done differently.

It uses an address that is already scoped within the KVM memory map and uses KVM's gpa_to_gfn() to translate it to some pinnable page

It is not some independent thing like VFIO, it is explicitly scoped within the existing KVM structure and it does not follow any mutations that are done to the gpa map through the usual KVM APIs.

Quoted 2 lines

I'd really be happy to learn how a better approach would look like that does not use longterm pinnings.

Sounds like the FW sadly needs pinnings. This is why I said it looks like DMA. If possible it would be better to get the pinning through VFIO, eg as a mdev

Otherwise, it would have been cleaner if this was divorced from KVM and took in a direct user pointer, then maybe you could make the argument is its own thing with its own lifetime rules. (then you are kind of making your own mdev)

Or, perhaps, this is really part of some radical "irqfd" that we've been on and off talking about specifically to get this area of interrupt bypass uAPI'd properly..

Quoted 6 lines

Quoted 2 lines

I'd really be happy to learn how a better approach would look like that does not use longterm pinnings.

Sounds like the FW sadly needs pinnings. This is why I said it looks like DMA. If possible it would be better to get the pinning through VFIO, eg as a mdev

Hrm, these are today handled as a vfio-pci device (e.g. no mdev) so that would be a pretty significant change.

Quoted 4 lines

Otherwise, it would have been cleaner if this was divorced from KVM and took in a direct user pointer, then maybe you could make the argument is its own thing with its own lifetime rules. (then you are kind of making your own mdev)

Problem there is that firmware needs both the location (where to indicate the event) and the identity of the KVM instance (what guest to ship the GISA alert to) so I don't think we can completely divorce them.

Quoted 3 lines

Or, perhaps, this is really part of some radical "irqfd" that we've been on and off talking about specifically to get this area of interrupt bypass uAPI'd properly..

I investigated that at one point and could not seem to get it to fit; I'll have another look there.

Quoted 24 lines · David Hildenbrand

Quoted 12 lines · Jason Gunthorpe

Quoted 9 lines · David Hildenbrand

Quoted 5 lines

It missses the general architectural point why we have all these shootdown mechanims in other places - plares are not supposed to make these kinds of assumptions. When the userspace unplugs the memory from KVM or unmaps it from VFIO it is not still being accessed by the kernel.

Yes. Like having memory in a vfio iommu v1 and doing the same (mremap, munmap, MADV_DONTNEED, ...). Which is why we disable MADV_DONTNEED (e.g., virtio-balloon) in QEMU with vfio.

Quoted 5 lines

That is different, VFIO has it's own contract how it consumes the memory from the MM and VFIO breaks all this stuff.

But when you tell VFIO to unmap the memory it doesn't keep accessing it in the background like this does.

To me, this is similar to when QEMU (user space) triggers KVM_S390_ZPCIOP_DEREG_AEN, to tell KVM to disable AIF and stop using the page (1) When triggered by the guest explicitly (2) when resetting the VM
(3) when resetting the virtual PCI device / configuration.

Quoted 5 lines

Interrupt gets unregistered from HW (which stops using the page), the pages get unpinned. Pages get no longer used.

I guess I am still missing (a) how this is fundamentally different (b) how it could be done differently.

It uses an address that is already scoped within the KVM memory map and uses KVM's gpa_to_gfn() to translate it to some pinnable page

It is not some independent thing like VFIO, it is explicitly scoped within the existing KVM structure and it does not follow any mutations that are done to the gpa map through the usual KVM APIs.

Quoted 3 lines

I'd really be happy to learn how a better approach would look like that does not use longterm pinnings.

Sounds like the FW sadly needs pinnings. This is why I said it looks like DMA. If possible it would be better to get the pinning through VFIO, eg as a mdev

Otherwise, it would have been cleaner if this was divorced from KVM and took in a direct user pointer, then maybe you could make the argument is its own thing with its own lifetime rules. (then you are kind of making your own mdev)

Or, perhaps, this is really part of some radical "irqfd" that we've been on and off talking about specifically to get this area of interrupt bypass uAPI'd properly..

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

Which, again, makes it weird/wrong that this KVM code is pinning it again :\

Jason

Quoted 9 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Quoted 2 lines

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

I might have mis-explained above.

With iommufd nesting, we will pin everything upfront as a starting point.

The current usage of vfio type1 iommu for s390 does not pin the entirety of guest memory upfront, it happens as guest RPCITs occur / type1 mappings are made.

Quoted 15 lines · Matthew Rosato

Quoted 11 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

I might have mis-explained above.

Quoted 3 lines

With iommufd nesting, we will pin everything upfront as a starting point.

The current usage of vfio type1 iommu for s390 does not pin the entirety of guest memory upfront, it happens as guest RPCITs occur / type1 mappings are made.

... so, after the domain started successfully on the libvirt/QEMU side ? :/

It would be great to confirm that. There might be a BUG in patch #2 (see my reply to patch #2) that might not allow you to reproduce it right now.

--
Thanks,

David / dhildenb

Quoted 16 lines · David Hildenbrand

Quoted 15 lines · Matthew Rosato

Quoted 11 lines · Jason Gunthorpe

Quoted 7 lines · David Hildenbrand

Quoted 1 line

Eventually we want to implement a mechanism where we can dynamically pin in response to RPCIT.

Okay, so IIRC we'll fail starting the domain early, that's good. And if we pin all guest memory (instead of small pieces dynamically), there is little existing use for file-backed RAM in such zPCI configurations (because memory cannot be reclaimed either way if it's all pinned), so likely there are no real existing users.

Right, this is VFIO, the physical HW can't tolerate not having pinned memory, so something somewhere is always pinning it.

I might have mis-explained above.

Quoted 9 lines

Quoted 3 lines

With iommufd nesting, we will pin everything upfront as a starting point.

The current usage of vfio type1 iommu for s390 does not pin the entirety of guest memory upfront, it happens as guest RPCITs occur / type1 mappings are made.

... so, after the domain started successfully on the libvirt/QEMU side ? :/

It would be great to confirm that. There might be a BUG in patch #2 (see my reply to patch #2) that might not allow you to reproduce it right now.

Yes apologies - thank you VERY much for doing this Matthew, but apologies, I made rather a clanger in patch 2 which would mean fast patch degrading to slow path would pass even for file-backed.

Will respin a v7 + cc you on that, if you could be so kind as to test that?

Quoted 9 lines · Jason Gunthorpe

On Tue, May 02, 2023 at 03:04:27PM +0200, Christian Borntraeger wrote:
\> > We can reintroduce a flag to permit exceptions if this is really broken, are you

Quoted 6 lines

Quoted 1 line

able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having <memoryBacking>
     <source type="file"/>
   </memoryBacking>

Quoted 2 lines

This s390 code is the least of the problems, after this series VFIO won't startup at all with this configuration.

Good question if the domain would fail to start. I recall that IOMMUs for zPCI are special on s390x. [1]

Well, zPCI is special. I cannot immediately tell when we would trigger long-term pinning.

[1] https://www.mail-archive.com/[email protected]/msg875728.html

Am 02.05.23 um 14:54 schrieb Lorenzo Stoakes:

Quoted 22 lines · Christian Borntraeger

Quoted 12 lines

Am 02.05.23 um 01:11 schrieb Lorenzo Stoakes:

Quoted 7 lines

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

Hmm, does this interfer with KVM on s390 and PCI interpretion of interrupt delivery? It would no longer work with file backed memory, correct?

Quoted 8 lines

See
arch/s390/kvm/pci.c

kvm_s390_pci_aif_enable
which does have
FOLL_WRITE | FOLL_LONGTERM
to

Quoted 4 lines

Does this memory map a dirty-tracked file? It's kind of hard to dig into where the address originates from without going through a ton of code. In worst case if the fast code doesn't find a whitelist it'll fall back to slow path which explicitly checks for dirty-tracked filesystem.

It does pin from whatever QEMU uses as backing for the guest.

Quoted 3 lines

We can reintroduce a flag to permit exceptions if this is really broken, are you able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having <memoryBacking>
    <source type="file"/>
  </memoryBacking>

In libvirt I guess.

Quoted 35 lines · Christian Borntraeger

Am 02.05.23 um 14:54 schrieb Lorenzo Stoakes:

Quoted 22 lines · Christian Borntraeger

Quoted 12 lines

Am 02.05.23 um 01:11 schrieb Lorenzo Stoakes:

Quoted 7 lines

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

Hmm, does this interfer with KVM on s390 and PCI interpretion of interrupt delivery? It would no longer work with file backed memory, correct?

Quoted 8 lines

See
arch/s390/kvm/pci.c

kvm_s390_pci_aif_enable
which does have
FOLL_WRITE | FOLL_LONGTERM
to

Quoted 4 lines

Does this memory map a dirty-tracked file? It's kind of hard to dig into where the address originates from without going through a ton of code. In worst case if the fast code doesn't find a whitelist it'll fall back to slow path which explicitly checks for dirty-tracked filesystem.

It does pin from whatever QEMU uses as backing for the guest.

Quoted 3 lines

We can reintroduce a flag to permit exceptions if this is really broken, are you able to test? I don't have an s390 sat around :)

Quoted 6 lines

Matt (Rosato on cc) probably can. In the end, it would mean having
  <memoryBacking>
    <source type="file"/>
  </memoryBacking>

In libvirt I guess.

I am running with this series applied using a QEMU guest with memory-backend-file (using the above libvirt snippet) for a few different PCI device types and AEN forwarding (e.g. what is setup in kvm_s390_pci_aif_enable) is still working.

Quoted 41 lines · Matthew Rosato

Quoted 40 lines · Christian Borntraeger

Am 02.05.23 um 14:54 schrieb Lorenzo Stoakes:

Quoted 22 lines · Christian Borntraeger

Quoted 12 lines

Am 02.05.23 um 01:11 schrieb Lorenzo Stoakes:

Quoted 7 lines

Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly.

This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path.

Hmm, does this interfer with KVM on s390 and PCI interpretion of interrupt delivery? It would no longer work with file backed memory, correct?

Quoted 8 lines

See
arch/s390/kvm/pci.c

kvm_s390_pci_aif_enable
which does have
FOLL_WRITE | FOLL_LONGTERM
to

Quoted 4 lines

Does this memory map a dirty-tracked file? It's kind of hard to dig into where the address originates from without going through a ton of code. In worst case if the fast code doesn't find a whitelist it'll fall back to slow path which explicitly checks for dirty-tracked filesystem.

It does pin from whatever QEMU uses as backing for the guest.

Quoted 3 lines

We can reintroduce a flag to permit exceptions if this is really broken, are you able to test? I don't have an s390 sat around :)

Matt (Rosato on cc) probably can. In the end, it would mean having <memoryBacking>
     <source type="file"/>
   </memoryBacking>

Quoted 4 lines

Quoted 1 line

In libvirt I guess.

I am running with this series applied using a QEMU guest with memory-backend-file (using the above libvirt snippet) for a few different PCI device types and AEN forwarding (e.g. what is setup in kvm_s390_pci_aif_enable) is still working.

That's ... unexpected. :)

Either this series doesn't work as expected or you end up using a filesystem that is still compatible. But I guess most applicable filesystems (ext4, btrfs, xfs) all have a page_mkwrite callback and should, therefore, disallow long-term pinning with this series.