Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

regression 6.1.y Salvatore Bonaccorso 3 participants 6 messages Apr 15, 2025 – Apr 17, 2025

Subsystems (5)

Stable branch Memory management VFS Block layer +1 more subsystems

lore

Share & tools

lore

Salvatore Bonaccorso

author Reviewed-by Lorenzo Stoakes Reviewed-by Christoph Hellwig Acked-by David Hildenbrand

2025-04-15 18:59 UTC #0 lore raw

[Apologies if this has been reported already but I have not found an already filled corresponding report]

After updating from the 6.1.129 based version to 6.1.133, various users have reported that their VMs do not boot anymore up (both KVM and under Xen) if pci-passthrough is involved. The reports are at:

https://bugs.debian.org/1102889
https://bugs.debian.org/1102914
https://bugs.debian.org/1103153

Milan Broz bisected the issues and found that the commit introducing the problems can be tracked down to backport of c8070b787519 ("mm:
Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60

#regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774

476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774

Author: David Howells <[email protected]>
Date:   Fri May 26 22:41:40 2023 +0100

mm: Don't pin ZERO_PAGE in pin_user_pages()

[ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]

Make pin_user_pages*() leave a ZERO_PAGE unpinned if it extracts a pointer to it from the page tables and make unpin_user_page*() correspondingly ignore a ZERO_PAGE when unpinning. We don't want to risk overrunning a zero page's refcount as we're only allowed ~2 million pins on it - something that userspace can conceivably trigger.

Add a pair of functions to test whether a page or a folio is a ZERO_PAGE.

Signed-off-by David Howells <[email protected]>

cc Christoph Hellwig <[email protected]>

cc David Hildenbrand <[email protected]>

cc Lorenzo Stoakes <[email protected]>

cc Andrew Morton <[email protected]>

cc Jens Axboe <[email protected]>

cc Al Viro <[email protected]>

cc Matthew Wilcox <[email protected]>

cc Jan Kara <[email protected]>

cc Jeff Layton <[email protected]>

cc Jason Gunthorpe <[email protected]>

cc Logan Gunthorpe <[email protected]>

cc Hillf Danton <[email protected]>

cc Christian Brauner <[email protected]>

cc Linus Torvalds <[email protected]>

cc [email protected]

Reviewed-by Lorenzo Stoakes <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

Acked-by David Hildenbrand <[email protected]>

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Jens Axboe <[email protected]>

Stable-dep-of: bddf10d26e6e ("uprobes: Reject the shared zeropage in uprobe_write_opcode()")

Signed-off-by Sasha Levin <[email protected]>

 Documentation/core-api/pin_user_pages.rst |  6 ++++++
 include/linux/mm.h                        | 26 ++++++++++++++++++++++++--
 mm/gup.c                                  | 31 ++++++++++++++++++++++++++++++-
 3 files changed, 60 insertions(+), 3 deletions(-)

Milan verified that the issue persists in 6.1.134 so far and the patch itself cannot be just reverted.

The failures all have a similar pattern, when pci-passthrough is used for a pci devide, for instance under qemu the bootup will fail with:

qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: VFIO_MAP_DMA failed: Cannot allocate memory
qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: vfio 0000:03:00.0: failed to setup container

(in the case as reported by Milan).

Any ideas here?

Regards,
Salvatore

Salvatore Bonaccorso

author

2025-04-15 19:08 UTC #1 lore raw

[resent with correct [email protected] list]

Quoted 85 lines · Salvatore Bonaccorso

Hi
[Apologies if this has been reported already but I have not found an already filled corresponding report]
After updating from the 6.1.129 based version to 6.1.133, various users have reported that their VMs do not boot anymore up (both KVM and under Xen) if pci-passthrough is involved. The reports are at:
https://bugs.debian.org/1102889
https://bugs.debian.org/1102914
https://bugs.debian.org/1103153
Milan Broz bisected the issues and found that the commit introducing the problems can be tracked down to backport of c8070b787519 ("mm:
Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
#regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
Author: David Howells <[email protected]>
Date:   Fri May 26 22:41:40 2023 +0100
mm: Don't pin ZERO_PAGE in pin_user_pages()
[ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]
Make pin_user_pages*() leave a ZERO_PAGE unpinned if it extracts a pointer to it from the page tables and make unpin_user_page*() correspondingly ignore a ZERO_PAGE when unpinning. We don't want to risk overrunning a zero page's refcount as we're only allowed ~2 million pins on it - something that userspace can conceivably trigger.
Add a pair of functions to test whether a page or a folio is a ZERO_PAGE.
Signed-off-by David Howells <[email protected]>
cc Christoph Hellwig <[email protected]>
cc David Hildenbrand <[email protected]>
cc Lorenzo Stoakes <[email protected]>
cc Andrew Morton <[email protected]>
cc Jens Axboe <[email protected]>
cc Al Viro <[email protected]>
cc Matthew Wilcox <[email protected]>
cc Jan Kara <[email protected]>
cc Jeff Layton <[email protected]>
cc Jason Gunthorpe <[email protected]>
cc Logan Gunthorpe <[email protected]>
cc Hillf Danton <[email protected]>
cc Christian Brauner <[email protected]>
cc Linus Torvalds <[email protected]>
cc [email protected]
cc [email protected]
cc [email protected]
cc [email protected]
Reviewed-by Lorenzo Stoakes <[email protected]>
Reviewed-by Christoph Hellwig <[email protected]>
Acked-by David Hildenbrand <[email protected]>
Link https://lore.kernel.org/r/[email protected] open here
Signed-off-by Jens Axboe <[email protected]>
Stable-dep-of: bddf10d26e6e ("uprobes: Reject the shared zeropage in uprobe_write_opcode()")
Signed-off-by Sasha Levin <[email protected]>
 Documentation/core-api/pin_user_pages.rst |  6 ++++++
 include/linux/mm.h                        | 26 ++++++++++++++++++++++++--
 mm/gup.c                                  | 31 ++++++++++++++++++++++++++++++-
 3 files changed, 60 insertions(+), 3 deletions(-)
Milan verified that the issue persists in 6.1.134 so far and the patch itself cannot be just reverted.
The failures all have a similar pattern, when pci-passthrough is used for a pci devide, for instance under qemu the bootup will fail with:
qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: VFIO_MAP_DMA failed: Cannot allocate memory
qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: vfio 0000:03:00.0: failed to setup container
(in the case as reported by Milan).
Any ideas here?
Regards,
Salvatore

Milan Broz

2025-04-15 19:11 UTC #2 lore raw

In reply to Salvatore Bonaccorso

Quoted 2 lines · Salvatore Bonaccorso

Milan verified that the issue persists in 6.1.134 so far and the patch itself cannot be just reverted.

Just an update:

With 6.1.134 stable as a base, I reverted that commit (upstream c8070b78751955e59b42457b974bea4a4fe00187)
+ applied upstream commit 873aefb376bbc0ed1dd2381ea1d6ec88106fdbd4 (vfio/type1: Unpin zero pages)
+ reverted upstream bddf10d26e6e5114e7415a0e442ec6f51a559468 (uprobes: Reject the shared zeropage in uprobe_write_opcode()) to be able compile code without errors.

With these changes, kernel works with my NVMe passthrough test again. This confirms the issue.

Milan

Quoted 13 lines

The failures all have a similar pattern, when pci-passthrough is used for a pci devide, for instance under qemu the bootup will fail with:
qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: VFIO_MAP_DMA failed: Cannot allocate memory
qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: vfio 0000:03:00.0: failed to setup container
(in the case as reported by Milan).
Any ideas here?
Regards,
Salvatore

Alex Williamson

2025-04-16 20:26 UTC #3 lore raw

In reply to Salvatore Bonaccorso

Quoted 21 lines · On Tue, 15 Apr 2025 20:59:19 +0200 Salvatore Bonaccorso <[email protected]>…

Hi
[Apologies if this has been reported already but I have not found an already filled corresponding report]
After updating from the 6.1.129 based version to 6.1.133, various users have reported that their VMs do not boot anymore up (both KVM and under Xen) if pci-passthrough is involved. The reports are at:
https://bugs.debian.org/1102889
https://bugs.debian.org/1102914
https://bugs.debian.org/1103153
Milan Broz bisected the issues and found that the commit introducing the problems can be tracked down to backport of c8070b787519 ("mm:
Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
#regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774

Quoted 8 lines

476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
Author: David Howells <[email protected]>
Date:   Fri May 26 22:41:40 2023 +0100
mm: Don't pin ZERO_PAGE in pin_user_pages()
[ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]

It's a bad backport, I've debugged and posted the fix for stable here:

https://lore.kernel.org/all/[email protected]/

Thanks,
Alex

Salvatore Bonaccorso

author

2025-04-16 21:51 UTC #4 lore raw

Hi Alex,

Quoted 29 lines · Alex Williamson

Quoted 26 lines · On Tue, 15 Apr 2025 20:59:19 +0200 Salvatore Bonaccorso <[email protected]>…
Hi
[Apologies if this has been reported already but I have not found an already filled corresponding report]
After updating from the 6.1.129 based version to 6.1.133, various users have reported that their VMs do not boot anymore up (both KVM and under Xen) if pci-passthrough is involved. The reports are at:
https://bugs.debian.org/1102889
https://bugs.debian.org/1102914
https://bugs.debian.org/1103153
Milan Broz bisected the issues and found that the commit introducing the problems can be tracked down to backport of c8070b787519 ("mm:
Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
#regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
Author: David Howells <[email protected]>
Date:   Fri May 26 22:41:40 2023 +0100

Quoted 7 lines

Quoted 3 lines
mm: Don't pin ZERO_PAGE in pin_user_pages()
[ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]
It's a bad backport, I've debugged and posted the fix for stable here:
https://lore.kernel.org/all/[email protected]/

Thank you, that worked (replying here as well mainly to fix my mistake in the CC to [email protected], which got truncated to [email protected] in my initial submission).

Quoted 4 lines

Thanks,
Alex

Signature · 4 lines

  .-.  Salvatore Bonaccorso --------------- Debian GNU/Linux Developer
  oo|  ----------------------------------------- http://www.debian.org
 /`'\  GPG key ID: 0x789D6F057FD863FE --------------------------------
(\_;/) Fingerprint: 04A4 407C B914 2C23 030C  17AE 789D 6F05 7FD8 63FE

Milan Broz

Tested-by

2025-04-17 05:14 UTC #5 lore raw

Quoted 36 lines · Salvatore Bonaccorso

Hi Alex,
Quoted 33 lines · Alex Williamson
Quoted 29 lines · On Tue, 15 Apr 2025 20:59:19 +0200 Salvatore Bonaccorso <[email protected]>…
Hi
[Apologies if this has been reported already but I have not found an already filled corresponding report]
After updating from the 6.1.129 based version to 6.1.133, various users have reported that their VMs do not boot anymore up (both KVM and under Xen) if pci-passthrough is involved. The reports are at:
https://bugs.debian.org/1102889
https://bugs.debian.org/1102914
https://bugs.debian.org/1103153
Milan Broz bisected the issues and found that the commit introducing the problems can be tracked down to backport of c8070b787519 ("mm:
Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
#regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
Author: David Howells <[email protected]>
Date:   Fri May 26 22:41:40 2023 +0100
mm: Don't pin ZERO_PAGE in pin_user_pages()
[ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]

Quoted 7 lines

Quoted 3 lines
It's a bad backport, I've debugged and posted the fix for stable here:
https://lore.kernel.org/all/[email protected]/
Thank you, that worked (replying here as well mainly to fix my mistake in the CC to [email protected], which got truncated to [email protected] in my initial submission).

I can also confirm the fix works, thanks for the quick response!

Tested-by Milan Broz <[email protected]>

Milan