Updated InfiniBand/RDMA merge plans for 2.6.24
Since 2.6.23 still isn't out, and I've managed to reduce my patch review backlog a bit, it's probably a good idea to give another update about what I have queued for 2.6.24 already and what I hope to get to before the merge window opens.
Core:
- My user_mad P_Key index support patch. Merged this, although I still owe Sasha a patch to update libraries to use this.
- A fix to the user_mad 32-bit big-endian userspace 64/32 problem with the method_mask when registering agents. Merged.
- Sean's QoS changes. Merged.
- Sean's IB CM MRA interface changes. I merged these -- what the heck, if it breaks we can back them out.
ULPs:
- Pradeep's IPoIB CM support for devices that don't have SRQs. Sean started reviewing but I didn't see any updated patches.
- Moni's IPoIB bonding support. Seems like we found a clean set of changes, and these will go in via another (Jeff Garzik's?) tree.
- Rolf's IPoIB MGID scope changes. No review progress here.
- Eli and Michael's IPoIB stateless offload (checksum offload, LSO, LRO, etc). Not much review progress here; I'll try to chip away at the series and see what we can get into 2.6.24.
- Or's IPoIB/userspace multicast coexistence stuff. I think we've converged on this; I'll merge this once a final version of the patch appears.
HW specific:
- I already merged patches to enable MSI-X by default for mthca and mlx4. I hope there aren't too many systems that get hosed if a MSI-X interrupt is generated.
- Jack and Michael's mlx4 FMR support. Merged. I guess the fix for running in Xen domU may need to wait for 2.6.25, but I'll see what I can do.
- ehca patch queue. Merged everything I think.
- Steve's mthca router mode support. No one looked at it, seems like it's at risk of missing the window.
- Arthur's mthca doorbell alignment fixes. I still need to check various approaches; I'll definitely merge something for 2.6.24.
- Michael's mlx4 WQE shrinking patch. May miss the window and go for 2.6.25, I'll see if I can get to it.
Here are a few topics that I believe will not be ready in time for the 2.6.24 window and will need to wait for 2.6.25:
- Multiple CQ event vector support. I haven't seen any discussions about how ULPs or userspace apps should decide which vector to use, and hence no progress has been made since we deferred this during the 2.6.23 merge window.
- XRC. Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.
BOILERPLATE
===========
I keep patches in a git tree, available from
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git
There are several branches of interest in this tree:
for-2.6.23 - changes queued for merging into the current kernel release for-2.6.24 - changes queued for the next merge window for-linus - changes I have asked Linus to pull upstream
for-mm - pulled by Andrew for inclusion in -mm
I frequently rewrite history and rebase my tree, so the best way to track it is to keep a clone of Linus's tree around and then pull a fresh copy of my tree with
git clone --reference /path/to/linus/tree \
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.gitIf you would like me to merge a patch, please send it to me as soon as it is ready. Do NOT wait for the merge window to open; if your change is not strictly a fix and you send it to me after the merge window opens, then it will likely have to wait for the next merge window. Please let me know if your patch is a fix that should go into the current release or if it can wait for the next merge window; if it is a fix, please describe the severity of the issue your are fixing, so I can make a good judgement about which release it should go into.
Including a good changelog entry that explains what you are changing, why you are changing it, and how your change accomplishes your goal will greatly increase the chance of your patch being merged promptly. Getting an independent review and a Reviewed-by: line also helps a lot.
The files Documentation/SubmittingPatches and Documentation/SubmitChecklist in kernel source tree also have a lot of good advice that makes it easier for me to handle your patches.
DETAILS
=======
Here is the complete list of patches I have in my for-2.6.24 branch:
Ali Ayoub (1):
IB/sa: Error handling thinko fix
Anton Blanchard (3):
IB/fmr_pool: Clean up some error messages in fmr_pool.c IB/ehca: Make output clearer by removing some debug messages IB/ehca: Export module parameters in sysfs
Dotan Barak (1):
mlx4_core: Use enum value GO_BIT_TIMEOUT_MSECS
Eli Cohen (2):
IPoIB: Fix typo to end statement with ';' instead of ','
IPoIB: Fix error path memory leak
Hoang-Nam Nguyen (4):
IB/ehca: Use remap_4k_pfn() to map firmware contexts to user space IB/ehca: Fix large page HW cap defines
IB/ehca: Fix mem leak of firmware ctrlblock in ehca_create_srq() IB/ehca: Adjust 64-bit alignment of create QP response for userspace
Jack Morgenstein (5):
IB/mlx4: Display misc device information under /sys/class/infiniband/
mlx4_core: Support ICM tables in coherent memory
mlx4_core: Write MTTs from CPU instead with of WRITE_MTT FW command IB/mlx4: Implement FMRs
mlx4_core: Increase max number of QPs per multicast group to 56
Joachim Fenkes (11):
IB/ehca: Refactor hvcall tracing
IB/ehca: Print return codes as signed decimal integers
IB/ehca: ehca_gen_warn() should always print
IB/ehca: Add check for max #SGE to create_qp()
IB/ehca: Path migration support
IB/ehca: Serialize MR alloc and MR free hvCalls
IB/ehca: Replace get_paca()->paca_index by the more portable raw_smp_processor_id() IB/ehca: Bump version number and change its format
IB/umem: Add hugetlb flag to struct ib_umem
IB/ehca: Only use MR large pages for hugetlb regions
IB/ehca: Return srq_attr->max_sge in ehca_query_srq()
Michael S. Tsirkin (2):
mlx4_core: Enable MSI-X by default
IB/mthca: Enable MSI-X by default
Peter Oruba (1):
IB/mthca: Use PCI-X/PCI-Express read control interfaces
Ralph Campbell (1):
IB/core: Fix handling of multicast response failures
Roland Dreier (14):
IPoIB: Make sure no receives are handled when stopping device
IB: find_first_zero_bit() takes unsigned pointer
mlx4_core: Don't free special QPs in QP number bitmap IB/mlx4: Use __set_data_seg() in mlx4_ib_post_recv() IB/ehca: Include <linux/mutex.h> from ehca_classes.h IB/mlx4: Fix up SRQ limit_watermark endianness
IB/iser: Remove unnecessary includes
mlx4_core: Change capability decoding: SRC->XRC
IB/umad: Add P_Key index support
IB/umad: Fix bit ordering and 32-on-64 problems on big endian systems IB/uverbs: Make ib_uverbs_release_event_file() static
mlx4_core: Reserve the correct number of MTT segments
mlx4_core: Fix meaning of dev->caps.reserved_mtts IB/mthca: Increase max number of QPs per multicast group to 56
Satyam Sharma (1):
IB/ehca: Misc cpuinit section annotations and #ifdef cleanups
Sean Hefty (7):
IPoIB: Specify Traffic Class with path record queries for QoS support IB/sa: Add new QoS fields to path record
RDMA/cma: Add ability to specify type of service RDMA/ucma: Allow user space to set service type
IB/srp: Add QoS support through service ID
IB/cm: Modify interface to send MRAs in response to duplicate messages RDMA/cma: Queue IB CM MRAs to avoid unnecessary remote retries
Stefan Roscher (2):
IB/ehca: Small QP userspace support
IB/ehca: Support more than 4k QPs for userspace and kernelspace
Steve Wise (2):
RDMA/cxgb3: Make the iw_cxgb3 module parameters writable RDMA/cma: Use neigh_event_send() to start neighbour discovery
Quoted 7 lines
ULPs:
- Pradeep's IPoIB CM support for devices that don't have SRQs. Sean started reviewing but I didn't see any updated patches.
Roland, I submitted an updated patch incorporating some of Sean's comments within a day or two. Rest of comments pertained to restructuring the code and adding some additional module parameters.
This would require more discussions since some of these had been already discussed previously. We had decided upon this code structure after a lot of discussions and incorporating these would be undoing some of that.
We can discuss and revisit the comments after the merge.
Pradeep
No mention about the iwarp port space issue?
Here is the status of the current proposed patch:
- needs another round of changes based on Sean's feedback
- Arkady raised issues about the pain this puts on admins
- it forces services like nfs-rdma, which already separates the nfs-rdma server by port number, to needlessly use a separate subnet for the rdma service.
I'm at a loss as to how to proceed.
Any ideas?
Steve.
Quoted 205 lines · Roland Dreier
Since 2.6.23 still isn't out, and I've managed to reduce my patch review backlog a bit, it's probably a good idea to give another update about what I have queued for 2.6.24 already and what I hope to get to before the merge window opens.
Core:
- My
user_madP_Keyindex support patch. Merged this, although I still owe Sasha a patch to update libraries to use this.- A fix to the
user_mad32-bit big-endian userspace 64/32 problem with themethod_maskwhen registering agents. Merged.- Sean's QoS changes. Merged.
- Sean's IB CM MRA interface changes. I merged these -- what the heck, if it breaks we can back them out.
ULPs:
- Pradeep's IPoIB CM support for devices that don't have SRQs. Sean started reviewing but I didn't see any updated patches.
- Moni's IPoIB bonding support. Seems like we found a clean set of changes, and these will go in via another (Jeff Garzik's?) tree.
- Rolf's IPoIB MGID scope changes. No review progress here.
- Eli and Michael's IPoIB stateless offload (checksum offload, LSO, LRO, etc). Not much review progress here; I'll try to chip away at the series and see what we can get into 2.6.24.
- Or's IPoIB/userspace multicast coexistence stuff. I think we've converged on this; I'll merge this once a final version of the patch appears.
HW specific:
- I already merged patches to enable MSI-X by default for mthca and mlx4. I hope there aren't too many systems that get hosed if a MSI-X interrupt is generated.
- Jack and Michael's mlx4 FMR support. Merged. I guess the fix for running in Xen domU may need to wait for 2.6.25, but I'll see what I can do.
- ehca patch queue. Merged everything I think.
- Steve's mthca router mode support. No one looked at it, seems like it's at risk of missing the window.
- Arthur's mthca doorbell alignment fixes. I still need to check various approaches; I'll definitely merge something for 2.6.24.
- Michael's mlx4 WQE shrinking patch. May miss the window and go for 2.6.25, I'll see if I can get to it.
Here are a few topics that I believe will not be ready in time for the 2.6.24 window and will need to wait for 2.6.25:
- Multiple CQ event vector support. I haven't seen any discussions about how ULPs or userspace apps should decide which vector to use, and hence no progress has been made since we deferred this during the 2.6.23 merge window.
- XRC. Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.
BOILERPLATE
===========I keep patches in a git tree, available from
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.gitThere are several branches of interest in this tree:
for-2.6.23 - changes queued for merging into the current kernel release for-2.6.24 - changes queued for the next merge window for-linus - changes I have asked Linus to pull upstream
for-mm - pulled by Andrew for inclusion in -mmI frequently rewrite history and rebase my tree, so the best way to track it is to keep a clone of Linus's tree around and then pull a fresh copy of my tree with
git clone --reference /path/to/linus/tree \ git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.gitIf you would like me to merge a patch, please send it to me as soon as it is ready. Do NOT wait for the merge window to open; if your change is not strictly a fix and you send it to me after the merge window opens, then it will likely have to wait for the next merge window. Please let me know if your patch is a fix that should go into the current release or if it can wait for the next merge window; if it is a fix, please describe the severity of the issue your are fixing, so I can make a good judgement about which release it should go into.
Including a good changelog entry that explains what you are changing, why you are changing it, and how your change accomplishes your goal will greatly increase the chance of your patch being merged promptly. Getting an independent review and a Reviewed-by: line also helps a lot.
The files
Documentation/SubmittingPatchesandDocumentation/SubmitChecklistin kernel source tree also have a lot of good advice that makes it easier for me to handle your patches.DETAILS
=======Here is the complete list of patches I have in my for-2.6.24 branch:
Ali Ayoub (1):
IB/sa: Error handling thinko fixAnton Blanchard (3):
IB/fmr_pool: Clean up some error messages infmr_pool.cIB/ehca: Make output clearer by removing some debug messages IB/ehca: Export module parameters in sysfsDotan Barak (1):
mlx4_core: Use enum valueGO_BIT_TIMEOUT_MSECSEli Cohen (2):
IPoIB: Fix typo to end statement with ';' instead of ','
IPoIB: Fix error path memory leakHoang-Nam Nguyen (4):
IB/ehca: Useremap_4k_pfn()to map firmware contexts to user space IB/ehca: Fix large page HW cap defines
IB/ehca: Fix mem leak of firmware ctrlblock inehca_create_srq()IB/ehca: Adjust 64-bit alignment of create QP response for userspaceJack Morgenstein (5):
IB/mlx4: Display misc device information under/sys/class/infiniband/
mlx4_core: Support ICM tables in coherent memory
mlx4_core: Write MTTs from CPU instead with ofWRITE_MTTFW command IB/mlx4: Implement FMRs
mlx4_core: Increase max number of QPs per multicast group to 56Joachim Fenkes (11):
IB/ehca: Refactor hvcall tracing
IB/ehca: Print return codes as signed decimal integers
IB/ehca:ehca_gen_warn()should always print
IB/ehca: Add check for max #SGE tocreate_qp()
IB/ehca: Path migration support
IB/ehca: Serialize MR alloc and MR free hvCalls
IB/ehca: Replace get_paca()->paca_index by the more portableraw_smp_processor_id()IB/ehca: Bump version number and change its format
IB/umem: Add hugetlb flag to structib_umem
IB/ehca: Only use MR large pages for hugetlb regions
IB/ehca: Returnsrq_attr->max_sgeinehca_query_srq()Michael S. Tsirkin (2):
mlx4_core: Enable MSI-X by default
IB/mthca: Enable MSI-X by defaultPeter Oruba (1):
IB/mthca: Use PCI-X/PCI-Express read control interfacesRalph Campbell (1):
IB/core: Fix handling of multicast response failuresRoland Dreier (14):
IPoIB: Make sure no receives are handled when stopping device
IB:find_first_zero_bit()takes unsigned pointer
mlx4_core: Don't free special QPs in QP number bitmap IB/mlx4: Use__set_data_seg()inmlx4_ib_post_recv()IB/ehca: Include <linux/mutex.h> fromehca_classes.hIB/mlx4: Fix up SRQlimit_watermarkendianness
IB/iser: Remove unnecessary includes
mlx4_core: Change capability decoding:SRC->XRC
IB/umad: AddP_Keyindex support
IB/umad: Fix bit ordering and 32-on-64 problems on big endian systems IB/uverbs: Makeib_uverbs_release_event_file()static
mlx4_core: Reserve the correct number of MTT segments
mlx4_core: Fix meaning ofdev->caps.reserved_mttsIB/mthca: Increase max number of QPs per multicast group to 56Satyam Sharma (1):
IB/ehca: Misc cpuinit section annotations and #ifdef cleanupsSean Hefty (7):
IPoIB: Specify Traffic Class with path record queries for QoS support IB/sa: Add new QoS fields to path record
RDMA/cma: Add ability to specify type of service RDMA/ucma: Allow user space to set service type
IB/srp: Add QoS support through service ID
IB/cm: Modify interface to send MRAs in response to duplicate messages RDMA/cma: Queue IB CM MRAs to avoid unnecessary remote retriesStefan Roscher (2):
IB/ehca: Small QP userspace support
IB/ehca: Support more than 4k QPs for userspace and kernelspaceSteve Wise (2):
RDMA/cxgb3: Make theiw_cxgb3module parameters writable RDMA/cma: Useneigh_event_send()to start neighbour discovery
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/generalTo unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Quoted 4 lines · Roland Dreier
- XRC. Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.
I posted the first draft patch set to the OpenFabrics list on September 18:
[ofa-general] [PATCH 0 of 5] XRC implementation patches (libibverbs, libmlx4, core, mlx4) Jack Morgenstein
[ofa-general] [PATCH 1 of 5] libibverbs: XRC implementation Jack Morgenstein
[ofa-general] [PATCH 2 of 5] libmlx4: XRC implementation Jack Morgenstein
[ofa-general] [PATCH 3 of 5] core: XRC implementation for fd = -1 when opening an xrc domain Jack Morgenstein
[ofa-general] [PATCH 4 of 5] core: XRC implementation -- add support for working with file descriptors Jack Morgenstein
[ofa-general] [PATCH 5 of 5] mlx4: XRC implementation Jack Morgenstein
The above patch set implements XRC for userspace verbs layer and below. The Kernel-space verbs implementation (1 more patch) is finished, but as yet untested.
- Jack
Quoted 5 lines
Quoted 4 lines
- XRC. Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.
I posted the first draft patch set to the OpenFabrics list on September 18:
Sorry, I didn't update that text. But my backlog is still too big to get XRC into 2.6.24.
- R.
Quoted 1 line
No mention about the iwarp port space issue?
I don't think we're at a stage where I'm prepared to merge something-- we all agree the latest patch has serious drawbacks, and it commits us to a suboptimal interface that is userspace-visible.
Quoted 1 line
I'm at a loss as to how to proceed.
Could we try to do some cleanups to the net core to make the alias stuff less painful? eg is there any sane way to make it possible for a device that creates 'eth0' to also create an 'iw0' alias without an assigning an address?
- R.
Quoted 6 lines · Roland Dreier
Quoted 1 line
No mention about the iwarp port space issue?
I don't think we're at a stage where I'm prepared to merge something-- we all agree the latest patch has serious drawbacks, and it commits us to a suboptimal interface that is userspace-visible.
Fair enough.
Quoted 7 lines
Quoted 1 line
I'm at a loss as to how to proceed.
Could we try to do some cleanups to the net core to make the alias stuff less painful? eg is there any sane way to make it possible for a device that creates 'eth0' to also create an 'iw0' alias without an assigning an address?
Well, "alias" interfaces really don't exist. ethX:iw is really just adding a address record (struct in_ifaddr) to ethX. So in the current core design, adding an alias without an address is really adding the alias with address 0.0.0.0. And I think the core net code assumes if an in_ifaddr struct exists for a device, then its IP address is indeed valid.
So I think the changes wouldn't be small to enhance the design to add a concept of an alias interface.
I'll look into this more though.
Steve.
Quoted 7 lines
Roland, I submitted an updated patch incorporating some of Sean's comments within a day or two. Rest of comments pertained to restructuring the code and adding some additional module parameters.
This would require more discussions since some of these had been already discussed previously. We had decided upon this code structure after a lot of discussions and incorporating these would be undoing some of that.
Can you give a link to your current final version of the patch?
Sean, what's your opinion of where we stand?
Since module parameters create a userspace-visible interface that we are stuck with for a long time, we definitely have to get at least that much right before merging.
- R.
Quoted 3 lines
Can you give a link to your current final version of the patch?
Sean, what's your opinion of where we stand?
Let me look back over the last version that was sent and reply back later today or tomorrow. Several of my initial comments were on code structure.
Quoted 3 lines
Since module parameters create a userspace-visible interface that we are stuck with for a long time, we definitely have to get at least that much right before merging.
I was taking a slightly different view of the design. It would be nice to agree on whether SRQ should be separated from the QP type before merging upstream, even if the implementation doesn't immediately support all available options.
- Sean
Quoted 10 lines · Roland Dreier
Quoted 7 lines
Roland, I submitted an updated patch incorporating some of Sean's comments within a day or two. Rest of comments pertained to restructuring the code and adding some additional module parameters.
This would require more discussions since some of these had been already discussed previously. We had decided upon this code structure after a lot of discussions and incorporating these would be undoing some of that.
Can you give a link to your current final version of the patch?
Roland,
This is the link to the last one that I submitted on 09/18.
http://lists.openfabrics.org/pipermail/general/2007-September/040917.html
Pradeep