infiniband/mthca : Fix userland mapping of mthca infiniband cards in Xen dom0
PATCH 1/1 infiniband/mthca : Fix userland mapping of mthca infiniband cards in Xen dom0
Infiniband allows the mapping of PCI memory regions to userland to
achieve faster speeds. Using this feature with mthca cards in a Xen dom0
kernel leads to a pretty bad crash, and the process being killed (see
http://xen.1045712.n5.nabble.com/Infiniband-from-userland-in-dom0-process-killed-bad-pagetable-td3259124.html )
This patch fixes the issue by setting the VM_IO flag on the vma, and refreshing the vm_page_prot field. The vm_page_prot will then have the _PAGE_IOMAP bit set, and Xen will therefore map the actual PCI space rather than a piece of non-existent RAM.
---
This has been heavily tested on a Fedora Core modified 2.6.32 kernel (with pvops patches : http://fedorapeople.org/~myoung/dom0/src/ ).I have also checked that it does not break anything on a stock kernel (2.6.37-rc7 from kernel.org), although only outside of Xen.
There has also been some discussion about this issue on the Xen mailing list :
http://lists.xensource.com/archives/html/xen-devel/2010-11/msg01739.html
drivers/infiniband/hw/mthca/mthca_provider.c +2 -0
@@ -391,6 +391,8 @@ static int mthca_mmap_uar(struct ib_uconif (vma->vm_end - vma->vm_start !=PAGE_SIZE )return -EINVAL ; + vma->vm_flags |=VM_IO ; + vma->vm_page_prot =vm_get_page_prot (vma->vm_flags); vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); if (io_remap_pfn_range(vma, vma->vm_start,
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
drivers/infiniband/hw/mthca/mthca_provider.c static int mthca_mmap_uar(struct ib_ucon · line 398
Quoted 11 lines
drivers/infiniband/hw/mthca/mthca_provider.c +2
-0@@ -391,6 +391,8 @@ static int mthca_mmap_uar(struct ib_uconif (vma->vm_end - vma->vm_start !=PAGE_SIZE )return -EINVAL ; + vma->vm_flags |=VM_IO ; + vma->vm_page_prot =vm_get_page_prot (vma->vm_flags); vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); if (io_remap_pfn_range(vma, vma->vm_start,
This is kind of unappealing -- is there no way to make io_remap_pfn_range do the right thing under Xen? It seems unfortunate to make drivers both set VM_IO and call io_remap_pfn_range. Or maybe we should get rid of io_remap_pfn_range() and just have drivers set VM_IO and call remap_pfn_range() all the time?
There are quite a few drivers under drivers/infiniband/hw that call io_remap_pfn_range() -- presumably they all need the analogous fix?
Finally as a stylistic thing I would probably prefer to see the vm_page_prot manipulation written as
vma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
and if this fix is really the right thing to do, this should probably be wrapped up in a helper function; presumably every instance of
vma->vm_page_prot = pgprot_FOO(vma->vm_page_prot)
under drivers/ would need the same fix for Xen.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Thanks for your reply.
Quoted 5 lines
This is kind of unappealing -- is there no way to make
io_remap_pfn_rangedo the right thing under Xen? It seems unfortunate to make drivers both setVM_IOand callio_remap_pfn_range. Or maybe we should get rid ofio_remap_pfn_range()and just have drivers setVM_IOand callremap_pfn_range()all the time?
Aside from IB drivers, the devices that need userland mapping are mainly graphic cards. There has been quite a few issues with that and the topic is discussed here :
http://wiki.xensource.com/xenwiki/XenPVOPSDRM
AFAIK, on x86 io_remap_pfn_range() is just #defined as remap_pfn_range(). remap_pfn_range() sets the VM_IO flag, but does not refresh the vm_page_prot field (mm/memory.c:1875). I think it could make sense to patch remap_pfn_range(), but I'm afraid it might break a lot of other drivers.
drivers/infiniband/hw/mthca/mthca_provider.c static int mthca_mmap_uar(struct ib_ucon · line 396
Quoted 3 lines
drivers/infiniband/hw/mthca/mthca_provider.c +2
-0+ vma->vm_flags |=VM_IO ; + vma->vm_page_prot =vm_get_page_prot (vma->vm_flags); vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot);
Under Xen, when the VM_IO flag is set, the vm_get_page_prot() macro will set the _PAGE_IOMAP bit on the the vm_page_prot.
Xen needs that bit to be set in order to understand that we actually want the real memory space to be mapped. The use of this macro was pointed out to me by the Xen team (see last post on
http://xen.1045712.n5.nabble.com/Kernel-Oops-when-reading-kernel-page-tables-debugfs-file-td3279750.html
), and, I assume, is the proper way of handling things.
To sum up :
- Xen needs the _PAGE_IOMAP flag set on the vm_page_prot for the
device to work.
- I understand the proper way to do this is to set the VM_IO flag,
and then call vm_get_page_prot()
Quoted 2 lines
There are quite a few drivers under
drivers/infiniband/hwthat callio_remap_pfn_range()-- presumably they all need the analogous fix?
I only have access to Mellanox Infinihost HBAs for testing, but once we figure out how we want this fixed, I can take a look at the other drivers.
Quoted 5 lines
Finally as a stylistic thing I would probably prefer to see the
vm_page_protmanipulation written asvma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
Makes sense.
While digging, I've also seen things like :
vma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags | VM_IO));
But it seems a bit dense.
Quoted 6 lines
and if this fix is really the right thing to do, this should probably be wrapped up in a helper function; presumably every instance of
vma->vm_page_prot =
pgprot_FOO(vma->vm_page_prot)under
drivers/would need the same fix for Xen.
The issue is actually only related to mappings of bus memory to userland. In kernel space, ioremap() is generaly used, which I understand sets the _PAGE_IOMAP flag.
Signature · 5 lines
Vivien Bernet-Rollande -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines
To sum up :
- Xen needs the_PAGE_IOMAPflag set on thevm_page_protfor the
device to work.
- I understand the proper way to do this is to set theVM_IOflag,
and then callvm_get_page_prot()Quoted 2 lines
There are quite a few drivers under
drivers/infiniband/hwthat callio_remap_pfn_range()-- presumably they all need the analogous fix?I only have access to Mellanox Infinihost HBAs for testing, but once
There is another way, which I hope can fix this issue (it fixes
the graphics drivers). Basically making the Xen MMU be aware of the E820 and consider any access to PFNs that are not in RAM to have VM_IO implicitly set.
The patches are at git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/p2m-identity.v4.3
And posted on LKML )https://lkml.org/lkml/2010/12/30/163)
They _should_ take care of your problem as long as the PFNs you are mapping fall within the PCI BAR regions (which I would presume they do since you are using ioremap).
Please tell me if they work (or not).
Quoted 6 lines
There is another way, which I hope can fix this issue (it fixes
the graphics drivers). Basically making the Xen MMU be aware of the E820 and consider any access to PFNs that are not in RAM to haveVM_IOimplicitly set.The patches are at
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
Quoted 7 lines
stable/p2m-identity.v4.3
And posted on LKML )https://lkml.org/lkml/2010/12/30/163)
They _should_ take care of your problem as long as the PFNs you are mapping fall within the PCI BAR regions (which I would presume they do since you are using ioremap).
Yes, that should work fine for all the cases I know of in the RDMA drivers. We are just mapping some PCI BAR space into userspace for direct access to hardware (just like graphics, I think).
I definitely prefer a solution that doesn't require driver authors to add a Xen-specific fix that isn't required on any other architecture.
- R.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Roland Dreier
Quoted 10 lines
There is another way, which I hope can fix this issue (it fixes
the graphics drivers). Basically making the Xen MMU be aware of the E820 and consider any access to PFNs that are not in RAM to haveVM_IOimplicitly set.The patches are at
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.gitstable/p2m-identity.v4.3
And posted on LKML )https://lkml.org/lkml/2010/12/30/163)
Quoted 7 lines
Quoted 3 lines
They _should_ take care of your problem as long as the PFNs you are mapping fall within the PCI BAR regions (which I would presume they do since you are using ioremap).
Yes, that should work fine for all the cases I know of in the RDMA drivers. We are just mapping some PCI BAR space into userspace for direct access to hardware (just like graphics, I think).
Allright, then it will work.
I've respun another version that fundamentally does the same thing but accounts for "what PFN is non-RAM" in a much cleaner way.
It is @:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/p2m-identity.v4.5
will post it soon on LKML.
Jack, Vivien: i you do merge it in your tree for testing with Xen and it works properly can I attach your 'Tested-by' on the patch-set?
Quoted 3 lines
I definitely prefer a solution that doesn't require driver authors to add a Xen-specific fix that isn't required on any other architecture.
Heh. You and me are on the same wavelength here.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20110107191559.GA27871-Z2dCqgVfyhJWk0Htik3J/[email protected]
We have no plans at present to test mthca on Xen. Our current Xen testing is focused on Connectx.
-Jack
-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:[email protected]]
Sent: Friday, January 07, 2011 9:16 PM
To: Roland Dreier
Cc: Vivien Bernet-Rollande; Jack Morgenstein; Jeremy Fitzhardinge; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Xen-devel] Re: [PATCH] infiniband/mthca : Fix userland mapping of mthca infiniband cards in Xen dom0
Quoted 10 lines · Roland Dreier
Quoted 10 lines
There is another way, which I hope can fix this issue (it fixes
the graphics drivers). Basically making the Xen MMU be aware of the E820 and consider any access to PFNs that are not in RAM to haveVM_IOimplicitly set.The patches are at
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.gitstable/p2m-identity.v4.3
And posted on LKML )https://lkml.org/lkml/2010/12/30/163)
Quoted 7 lines
Quoted 3 lines
They _should_ take care of your problem as long as the PFNs you are mapping fall within the PCI BAR regions (which I would presume they do since you are using ioremap).
Yes, that should work fine for all the cases I know of in the RDMA drivers. We are just mapping some PCI BAR space into userspace for direct access to hardware (just like graphics, I think).
Allright, then it will work.
I've respun another version that fundamentally does the same thing but accounts for "what PFN is non-RAM" in a much cleaner way.
It is @:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/p2m-identity.v4.5
will post it soon on LKML.
Jack, Vivien: i you do merge it in your tree for testing with Xen and it works properly can I attach your 'Tested-by' on the patch-set?
Quoted 3 lines
I definitely prefer a solution that doesn't require driver authors to add a Xen-specific fix that isn't required on any other architecture.
Heh. You and me are on the same wavelength here.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): AANLkTi=zMiVJfTRDeYvbxT46b0iASKG9jPgXqG81bqAK-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 10 lines
Quoted 5 lines
Finally as a stylistic thing I would probably prefer to see the
vm_page_protmanipulation written asvma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
Makes sense.
While digging, I've also seen things like :
vma->vm_page_prot=pgprot_noncached(vm_get_page_prot(vma->vm_flags | VM_IO));
This isn't the same, since it doesn't set VM_IO in vm_flags.
Quoted 9 lines
Quoted 5 lines
wrapped up in a helper function; presumably every instance of
vma->vm_page_prot =
pgprot_FOO(vma->vm_page_prot)under
drivers/would need the same fix for Xen.The issue is actually only related to mappings of bus memory to userland. In kernel space,
ioremap()is generaly used, which I understand sets the_PAGE_IOMAPflag.
Right, but setting vm_page_prot means a driver is doing userspace mapping of something, and if it's setting pgprot, that would generally mean it's not mapping RAM.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html