HMM_MIRROR has less than useful help text
Subsystems (2)
Hi!
Commit c0b124054f9e42eb6da545a10fe9122a7d7c3f72 has very nice commit message, explaining what HMM_MIRROR is and when it is needed. Unfortunately, it did not make it into Kconfig help:
CONFIG_HMM_MIRROR:
Select HMM_MIRROR if you want to mirror range of the CPU page table of a
process into a device page table. Here, mirror means "keep synchronized".
Prerequisites: the device must provide the ability to write-protect its
page tables (at PAGE_SIZE granularity), and must be able to recover from
the resulting potential page faults.
Could that be fixed?
This is key information for me:
# This is a heterogeneous memory management (HMM) process address space
# mirroring. # This is useful for NVidia GPU >= Pascal, Mellanox IB >= mlx5 and more # hardware in the future.
Thanks,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 19 lines · Pavel Machek
Hi!
Commit
c0b124054f9e42eb6da545a10fe9122a7d7c3f72has very nice commit message, explaining whatHMM_MIRRORis and when it is needed. Unfortunately, it did not make it intoKconfighelp:
CONFIG_HMM_MIRROR:Select
HMM_MIRRORif you want to mirror range of the CPU page table of a
process into a device page table. Here, mirror means "keep synchronized".
Prerequisites: the device must provide the ability to write-protect its
page tables (atPAGE_SIZEgranularity), and must be able to recover from
the resulting potential page faults.
Quoted 9 lines
Could that be fixed?
This is key information for me:
# This is a heterogeneous memory management (HMM) process address space
# mirroring. # This is useful for NVidia GPU >= Pascal, Mellanox IB >= mlx5 and more # hardware in the future.
That seems like a reasonable request
Balbir
Quoted 5 lines
Thanks,
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 25 lines · Balbir Singh
Quoted 23 lines · Pavel Machek
Hi!
Commit
c0b124054f9e42eb6da545a10fe9122a7d7c3f72has very nice commit message, explaining whatHMM_MIRRORis and when it is needed. Unfortunately, it did not make it intoKconfighelp:
CONFIG_HMM_MIRROR:Select
HMM_MIRRORif you want to mirror range of the CPU page table of a
process into a device page table. Here, mirror means "keep synchronized".
Prerequisites: the device must provide the ability to write-protect its
page tables (atPAGE_SIZEgranularity), and must be able to recover from
the resulting potential page faults.Could that be fixed?
This is key information for me:
Quoted 7 lines
Quoted 5 lines
# This is a heterogeneous memory management (HMM) process address space
# mirroring. # This is useful for NVidia GPU >= Pascal, Mellanox IB >= mlx5 and more # hardware in the future.That seems like a reasonable request
Hi Pavel, care to send a patch?
Quoted 32 lines · Dan Williams
Quoted 31 lines · Balbir Singh
Quoted 28 lines · Pavel Machek
Hi!
Commit
c0b124054f9e42eb6da545a10fe9122a7d7c3f72has very nice commit message, explaining whatHMM_MIRRORis and when it is needed. Unfortunately, it did not make it intoKconfighelp:
CONFIG_HMM_MIRROR:Select
HMM_MIRRORif you want to mirror range of the CPU page table of a
process into a device page table. Here, mirror means "keep synchronized".
Prerequisites: the device must provide the ability to write-protect its
page tables (atPAGE_SIZEgranularity), and must be able to recover from
the resulting potential page faults.Could that be fixed?
This is key information for me:
# This is a heterogeneous memory management (HMM) process address space
# mirroring. # This is useful for NVidia GPU >= Pascal, Mellanox IB >= mlx5 and more # hardware in the future.
Quoted 3 lines
Quoted 1 line
That seems like a reasonable request
Hi Pavel, care to send a patch?
I hoped patch author would fix up their code. I'm not HMM expert, he should be...
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
PATCH 1/1 mm/Kconfig: additional help text for HMM_MIRROR option
In reply to Pavel MachekFrom: John Hubbard <[email protected]>
The HMM_MIRROR option in Kconfig is a little underdocumented and mysterious, and leaves people wondering whether to enable it.
Add text explaining just a little bit more about HMM, and also mention which hardware would benefit from having HMM_MIRROR enabled.
---
Hi Pavel and all, does this help? I've tried to capture the key missing bits of documentation, but still keep it small, for Kconfig.
thanks,
John Hubbard
NVIDIA
mm/Kconfig | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
mm/Kconfig +12 -5
@@ -681,11 +681,18 @@ config HMM_MIRROR depends on MMU && 64BIT select MMU_NOTIFIER help - Select HMM_MIRROR if you want to mirror range of the CPU page table of a - process into a device page table. Here, mirror means "keep synchronized". - Prerequisites: the device must provide the ability to write-protect its - page tables (at PAGE_SIZE granularity), and must be able to recover from - the resulting potential page faults. + This is Heterogeneous Memory Management (HMM) process address space + mirroring. + + HMM_MIRROR provides a way to mirror ranges of the CPU page tables + of a process into a device page table. Here, mirror means "keep + synchronized". Prerequisites: the device must provide the ability + to write-protect its page tables (at PAGE_SIZE granularity), and + must be able to recover from the resulting potential page faults. + + Select HMM_MIRROR if you have hardware that meets the above + description. An early, partial list of such hardware is: + an NVIDIA GPU >= Pascal, Mellanox IB >= mlx5, or an AMD GPU. config DEVICE_PRIVATE bool "Unaddressable device memory (GPU memory, ...)"
Signature · 1 line
2.22.0
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 43 lines · On Thu, Jul 18, 2019 at 06:32:53PM -0700, [email protected] wrote:
From: John Hubbard <[email protected]>
The
HMM_MIRRORoption inKconfigis a little underdocumented and mysterious, and leaves people wondering whether to enable it.Add text explaining just a little bit more about HMM, and also mention which hardware would benefit from having
HMM_MIRRORenabled.Suggested-by Pavel Machek <[email protected]>Cc Balbir Singh <[email protected]>Cc Dan Williams <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Jerome Glisse <[email protected]>Cc Christoph Hellwig <[email protected]>Signed-off-by John Hubbard <[email protected]>---
Hi Pavel and all, does this help? I've tried to capture the key missing bits of documentation, but still keep it small, for
Kconfig.thanks,
John Hubbard
NVIDIAmm/Kconfig | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)mm/Kconfig +2
-5@@ -681,11 +681,18 @@ config HMM_MIRROR depends on MMU && 64BIT select MMU_NOTIFIER help - Select HMM_MIRROR if you want to mirror range of the CPU page table of a - process into a device page table. Here, mirror means "keep synchronized". - Prerequisites: the device must provide the ability to write-protect its - page tables (at PAGE_SIZE granularity), and must be able to recover from - the resulting potential page faults. + This is Heterogeneous Memory Management (HMM) process address space + mirroring.
mm/Kconfig config HMM_MIRROR · line 695
Quoted 10 lines
mm/Kconfig +10
-0+ + HMM_MIRROR provides a way to mirror ranges of the CPU page tables + of a process into a device page table. Here, mirror means "keep + synchronized". Prerequisites: the device must provide the ability + to write-protect its page tables (at PAGE_SIZE granularity), and + must be able to recover from the resulting potential page faults. + + Select HMM_MIRROR if you have hardware that meets the above + description. An early, partial list of such hardware is: + an NVIDIA GPU >= Pascal, Mellanox IB >= mlx5, or an AMD GPU.
I don't think we want to put device information here. If we want that information in Kconfig best to put it in the devices themselves. Otherwise this list will get stale.
Other than that, looks good.
Ira
Quoted 6 lines
mm/Kconfig +0
-0config DEVICE_PRIVATE bool "Unaddressable device memory (GPU memory, ...)"Signature · 1 line
2.22.0
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 2 lines · Ira Weiny
Quoted 1 line · On Thu, Jul 18, 2019 at 06:32:53PM -0700, [email protected] wrote:
From: John Hubbard <[email protected]>
...
Quoted 8 lines
mm/Kconfig config HMM_MIRROR · line 695
Quoted 3 lines
mm/Kconfig +3
-0+ Select HMM_MIRROR if you have hardware that meets the above + description. An early, partial list of such hardware is: + an NVIDIA GPU >= Pascal, Mellanox IB >= mlx5, or an AMD GPU.I don't think we want to put device information here. If we want that information in
Kconfigbest to put it in the devices themselves. Otherwise this list will get stale.
Quoted 4 lines
Other than that, looks good.
Reviewed-by Ira Weiny <[email protected]>
Hi Ira, thanks for the review, I'll remove that last sentence. I'll post a v2 with your reviewed by tag, in a new email thread. But first I'll wait to see if there are other replies.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
In reply to [email protected]mm/Kconfig config HMM_MIRROR · line 695
Quoted 9 lines · On Thu, Jul 18, 2019 at 06:32:53PM -0700, [email protected] wrote:
mm/Kconfig +9
-0+ HMM_MIRROR provides a way to mirror ranges of the CPU page tables + of a process into a device page table. Here, mirror means "keep + synchronized". Prerequisites: the device must provide the ability + to write-protect its page tables (at PAGE_SIZE granularity), and + must be able to recover from the resulting potential page faults. + + Select HMM_MIRROR if you have hardware that meets the above + description. An early, partial list of such hardware is: + an NVIDIA GPU >= Pascal, Mellanox IB >= mlx5, or an AMD GPU.
Nevermind that the Nvidia support is stagaging and looks rather broken, there is no Mellanox user of this either at this point.
But either way this has no business in a common kconfig help. Just drop the fine grained details and leave it to the overview.
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 14 lines · Christoph Hellwig
mm/Kconfig config HMM_MIRROR · line 695
Quoted 9 lines · On Thu, Jul 18, 2019 at 06:32:53PM -0700, [email protected] wrote:
mm/Kconfig +9
-0+ HMM_MIRROR provides a way to mirror ranges of the CPU page tables + of a process into a device page table. Here, mirror means "keep + synchronized". Prerequisites: the device must provide the ability + to write-protect its page tables (at PAGE_SIZE granularity), and + must be able to recover from the resulting potential page faults. + + Select HMM_MIRROR if you have hardware that meets the above + description. An early, partial list of such hardware is: + an NVIDIA GPU >= Pascal, Mellanox IB >= mlx5, or an AMD GPU.Nevermind that the Nvidia support is stagaging and looks rather broken, there is no Mellanox user of this either at this point.
Quoted 2 lines
But either way this has no business in a common kconfig help. Just drop the fine grained details and leave it to the overview.
I disagree here. This explains what kind of hardware this is for (very new). Partial list does not hurt, and I know that I probably don't need to enable this.
How else am I supposed to know if my computer needs page tables synchronized?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 18 lines · Pavel Machek
Quoted 14 lines · Christoph Hellwig
mm/Kconfig config HMM_MIRROR · line 695
Quoted 9 lines · On Thu, Jul 18, 2019 at 06:32:53PM -0700, [email protected] wrote:
mm/Kconfig +9
-0+ HMM_MIRROR provides a way to mirror ranges of the CPU page tables + of a process into a device page table. Here, mirror means "keep + synchronized". Prerequisites: the device must provide the ability + to write-protect its page tables (at PAGE_SIZE granularity), and + must be able to recover from the resulting potential page faults. + + Select HMM_MIRROR if you have hardware that meets the above + description. An early, partial list of such hardware is: + an NVIDIA GPU >= Pascal, Mellanox IB >= mlx5, or an AMD GPU.Nevermind that the Nvidia support is stagaging and looks rather broken, there is no Mellanox user of this either at this point.
Quoted 2 lines
But either way this has no business in a common kconfig help. Just drop the fine grained details and leave it to the overview.
Quoted 6 lines
I disagree here. This explains what kind of hardware this is for (very new). Partial list does not hurt, and I know that I probably don't need to enable this.
How else am I supposed to know if my computer needs page tables synchronized?
It is like MMU_NOTIFIERS, if something needs it, then it will select it.
Maybe it should just be a hidden kconfig anyhow as there is no reason to turn it on without also turning on a using driver.
Jason
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 5 lines · Jason Gunthorpe
It is like
MMU_NOTIFIERS, if something needs it, then it will select it.Maybe it should just be a hidden kconfig anyhow as there is no reason to turn it on without also turning on a using driver.
We can't just select it due to the odd X86_64 || PPC64 dependency.
Which also answers Pavels question: you never really need it, as we can only use it for optional functionality due to that.
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
It is like
MMU_NOTIFIERS, if something needs it, then it will select it.Maybe it should just be a hidden kconfig anyhow as there is no reason to turn it on without also turning on a using driver.
We can't just select it due to the odd
X86_64|| PPC64 dependency.Which also answers Pavels question: you never really need it, as we can only use it for optional functionality due to that.
Okay, just explain it in the help text :-)..
Alternatively... you can have WANT_HMM_MIRROR option drivers select, and option HMM_MIRROR which is yes if WANT_HMM_MIRROR && (X86_64 || PPC64), no?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 10 lines · Pavel Machek
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
It is like
MMU_NOTIFIERS, if something needs it, then it will select it.Maybe it should just be a hidden kconfig anyhow as there is no reason to turn it on without also turning on a using driver.
We can't just select it due to the odd
X86_64|| PPC64 dependency.
Quoted 9 lines
Quoted 2 lines
Which also answers Pavels question: you never really need it, as we can only use it for optional functionality due to that.
Okay, just explain it in the help text :-)..
Alternatively... you can have
WANT_HMM_MIRRORoption drivers select, and optionHMM_MIRRORwhich is yes ifWANT_HMM_MIRROR&& (X86_64|| PPC64), no?
Yes. This really should be a hidden option that just auto-enables. It's not ideal to require people to both *find* HMM_MIRROR, *and* figure out that they need it. (I think it's just this way due to the history of how HMM got merged--it started off as a kind of experimental sandbox, so it had it's own config options, to avoid bothering anything else.)
I'll send out a new patch to just auto-select. The WANT_HMM_MIRROR approach seems accurate, given the (X86_64 || PPC64) complication, probably after -rc1 is ready (I don't see the ODP code using HMM yet, so that must not have been merged yet.)
Longer term, I vaguely recall that there is no strong reason preventing HMM from being made to work on other arches, and am hoping that it was just done this way to save development time. I don't want to leave it this way unless there's a good reason to.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 32 lines · John Hubbard
Quoted 10 lines · Pavel Machek
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
It is like
MMU_NOTIFIERS, if something needs it, then it will select it.Maybe it should just be a hidden kconfig anyhow as there is no reason to turn it on without also turning on a using driver.
We can't just select it due to the odd
X86_64|| PPC64 dependency.Quoted 9 lines
Quoted 2 lines
Which also answers Pavels question: you never really need it, as we can only use it for optional functionality due to that.
Okay, just explain it in the help text :-)..
Alternatively... you can have
WANT_HMM_MIRRORoption drivers select, and optionHMM_MIRRORwhich is yes ifWANT_HMM_MIRROR&& (X86_64|| PPC64), no?Yes. This really should be a hidden option that just auto-enables. It's not ideal to require people to both *find*
HMM_MIRROR, *and* figure out that they need it. (I think it's just this way due to the history of how HMM got merged--it started off as a kind of experimental sandbox, so it had it's own config options, to avoid bothering anything else.)I'll send out a new patch to just auto-select. The
WANT_HMM_MIRRORapproach seems accurate, given the (X86_64|| PPC64) complication, probably after -rc1 is ready (I don't see the ODP code using HMM yet, so that must not have been merged yet.)
Quoted 4 lines
Longer term, I vaguely recall that there is no strong reason preventing HMM from being made to work on other arches, and am hoping that it was just done this way to save development time. I don't want to leave it this way unless there's a good reason to.
No one has given me a satisfactory answer about the restriction either.
The only thing this kconfig controls that could possibly be arch specific is the page walking code in hmm_range_snapshot and related.
Maybe there is/was some arch entanglement there?
Jason
Re: [PATCH] mm/Kconfig: additional help text for HMM_MIRROR option
Quoted 8 lines · Jason Gunthorpe
No one has given me a satisfactory answer about the restriction either.
The only thing this kconfig controls that could possibly be arch specific is the page walking code in
hmm_range_snapshotand related.Maybe there is/was some arch entanglement there?
The page walking code is supposed to be platform independent. I did push a branch to the buildbot a few days ago to catch issues, and the only one found so far is an abuse of pte_index() that can be trivially fixed. The other thing I noticed is that the use of some of the p??_none/protnone/present checks seems inconsistent, but I did not have time to audit that yet.