Skip to contentSkip to conversation

Updated InfiniBand/RDMA merge plans for 2.6.24

DISCUSSION Roland Dreier 5 participants 10 messages Oct 5, 2007 – Oct 9, 2007

lore
Share & tools
lore
#0 lore raw

Since 2.6.23 still isn't out, and I've managed to reduce my patch review backlog a bit, it's probably a good idea to give another update about what I have queued for 2.6.24 already and what I hope to get to before the merge window opens.

Core:

- My user_mad P_Key index support patch. Merged this, although I still owe Sasha a patch to update libraries to use this.

- A fix to the user_mad 32-bit big-endian userspace 64/32 problem with the method_mask when registering agents. Merged.

- Sean's QoS changes. Merged.

- Sean's IB CM MRA interface changes. I merged these -- what the heck, if it breaks we can back them out.

ULPs:

- Pradeep's IPoIB CM support for devices that don't have SRQs. Sean started reviewing but I didn't see any updated patches.

- Moni's IPoIB bonding support. Seems like we found a clean set of changes, and these will go in via another (Jeff Garzik's?) tree.

- Rolf's IPoIB MGID scope changes. No review progress here.

- Eli and Michael's IPoIB stateless offload (checksum offload, LSO, LRO, etc). Not much review progress here; I'll try to chip away at the series and see what we can get into 2.6.24.

- Or's IPoIB/userspace multicast coexistence stuff. I think we've converged on this; I'll merge this once a final version of the patch appears.

HW specific:

- I already merged patches to enable MSI-X by default for mthca and mlx4. I hope there aren't too many systems that get hosed if a MSI-X interrupt is generated.

- Jack and Michael's mlx4 FMR support. Merged. I guess the fix for running in Xen domU may need to wait for 2.6.25, but I'll see what I can do.

- ehca patch queue. Merged everything I think.

- Steve's mthca router mode support. No one looked at it, seems like it's at risk of missing the window.

- Arthur's mthca doorbell alignment fixes. I still need to check various approaches; I'll definitely merge something for 2.6.24.

- Michael's mlx4 WQE shrinking patch. May miss the window and go for 2.6.25, I'll see if I can get to it.

Here are a few topics that I believe will not be ready in time for the 2.6.24 window and will need to wait for 2.6.25:

- Multiple CQ event vector support. I haven't seen any discussions about how ULPs or userspace apps should decide which vector to use, and hence no progress has been made since we deferred this during the 2.6.23 merge window.

- XRC. Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.

BOILERPLATE
===========

I keep patches in a git tree, available from

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

There are several branches of interest in this tree:

for-2.6.23 - changes queued for merging into the current kernel release for-2.6.24 - changes queued for the next merge window for-linus - changes I have asked Linus to pull upstream
for-mm - pulled by Andrew for inclusion in -mm

I frequently rewrite history and rebase my tree, so the best way to track it is to keep a clone of Linus's tree around and then pull a fresh copy of my tree with

    git clone --reference /path/to/linus/tree \
        git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

If you would like me to merge a patch, please send it to me as soon as it is ready. Do NOT wait for the merge window to open; if your change is not strictly a fix and you send it to me after the merge window opens, then it will likely have to wait for the next merge window. Please let me know if your patch is a fix that should go into the current release or if it can wait for the next merge window; if it is a fix, please describe the severity of the issue your are fixing, so I can make a good judgement about which release it should go into.

Including a good changelog entry that explains what you are changing, why you are changing it, and how your change accomplishes your goal will greatly increase the chance of your patch being merged promptly. Getting an independent review and a Reviewed-by: line also helps a lot.

The files Documentation/SubmittingPatches and Documentation/SubmitChecklist in kernel source tree also have a lot of good advice that makes it easier for me to handle your patches.

DETAILS
=======

Here is the complete list of patches I have in my for-2.6.24 branch:

Ali Ayoub (1):
IB/sa: Error handling thinko fix

Anton Blanchard (3):
IB/fmr_pool: Clean up some error messages in fmr_pool.c IB/ehca: Make output clearer by removing some debug messages IB/ehca: Export module parameters in sysfs

Dotan Barak (1):
mlx4_core: Use enum value GO_BIT_TIMEOUT_MSECS

Eli Cohen (2):
IPoIB: Fix typo to end statement with ';' instead of ','
IPoIB: Fix error path memory leak

Hoang-Nam Nguyen (4):
IB/ehca: Use remap_4k_pfn() to map firmware contexts to user space IB/ehca: Fix large page HW cap defines
IB/ehca: Fix mem leak of firmware ctrlblock in ehca_create_srq() IB/ehca: Adjust 64-bit alignment of create QP response for userspace

Jack Morgenstein (5):
IB/mlx4: Display misc device information under /sys/class/infiniband/
mlx4_core: Support ICM tables in coherent memory
mlx4_core: Write MTTs from CPU instead with of WRITE_MTT FW command IB/mlx4: Implement FMRs
mlx4_core: Increase max number of QPs per multicast group to 56

Joachim Fenkes (11):
IB/ehca: Refactor hvcall tracing
IB/ehca: Print return codes as signed decimal integers
IB/ehca: ehca_gen_warn() should always print
IB/ehca: Add check for max #SGE to create_qp()
IB/ehca: Path migration support
IB/ehca: Serialize MR alloc and MR free hvCalls
IB/ehca: Replace get_paca()->paca_index by the more portable raw_smp_processor_id() IB/ehca: Bump version number and change its format
IB/umem: Add hugetlb flag to struct ib_umem
IB/ehca: Only use MR large pages for hugetlb regions
IB/ehca: Return srq_attr->max_sge in ehca_query_srq()

Michael S. Tsirkin (2):
mlx4_core: Enable MSI-X by default
IB/mthca: Enable MSI-X by default

Peter Oruba (1):
IB/mthca: Use PCI-X/PCI-Express read control interfaces

Ralph Campbell (1):
IB/core: Fix handling of multicast response failures

Roland Dreier (14):
IPoIB: Make sure no receives are handled when stopping device
IB: find_first_zero_bit() takes unsigned pointer
mlx4_core: Don't free special QPs in QP number bitmap IB/mlx4: Use __set_data_seg() in mlx4_ib_post_recv() IB/ehca: Include <linux/mutex.h> from ehca_classes.h IB/mlx4: Fix up SRQ limit_watermark endianness
IB/iser: Remove unnecessary includes
mlx4_core: Change capability decoding: SRC->XRC
IB/umad: Add P_Key index support
IB/umad: Fix bit ordering and 32-on-64 problems on big endian systems IB/uverbs: Make ib_uverbs_release_event_file() static
mlx4_core: Reserve the correct number of MTT segments
mlx4_core: Fix meaning of dev->caps.reserved_mtts IB/mthca: Increase max number of QPs per multicast group to 56

Satyam Sharma (1):
IB/ehca: Misc cpuinit section annotations and #ifdef cleanups

Sean Hefty (7):
IPoIB: Specify Traffic Class with path record queries for QoS support IB/sa: Add new QoS fields to path record
RDMA/cma: Add ability to specify type of service RDMA/ucma: Allow user space to set service type
IB/srp: Add QoS support through service ID
IB/cm: Modify interface to send MRAs in response to duplicate messages RDMA/cma: Queue IB CM MRAs to avoid unnecessary remote retries

Stefan Roscher (2):
IB/ehca: Small QP userspace support
IB/ehca: Support more than 4k QPs for userspace and kernelspace

Steve Wise (2):
RDMA/cxgb3: Make the iw_cxgb3 module parameters writable RDMA/cma: Use neigh_event_send() to start neighbour discovery

#1 lore raw
Quoted 7 lines

ULPs:

- Pradeep's IPoIB CM support for devices that don't have SRQs. Sean started reviewing but I didn't see any updated patches.

Roland, I submitted an updated patch incorporating some of Sean's comments within a day or two. Rest of comments pertained to restructuring the code and adding some additional module parameters.

This would require more discussions since some of these had been already discussed previously. We had decided upon this code structure after a lot of discussions and incorporating these would be undoing some of that.

We can discuss and revisit the comments after the merge.

Pradeep

#5 lore raw
In reply to Roland Dreier

No mention about the iwarp port space issue?

Here is the status of the current proposed patch:

- needs another round of changes based on Sean's feedback

- Arkady raised issues about the pain this puts on admins

- it forces services like nfs-rdma, which already separates the nfs-rdma server by port number, to needlessly use a separate subnet for the rdma service.

I'm at a loss as to how to proceed.

Any ideas?

Steve.

Quoted 205 lines · Roland Dreier

Since 2.6.23 still isn't out, and I've managed to reduce my patch review backlog a bit, it's probably a good idea to give another update about what I have queued for 2.6.24 already and what I hope to get to before the merge window opens.

Core:

- My user_mad P_Key index support patch. Merged this, although I still owe Sasha a patch to update libraries to use this.

- A fix to the user_mad 32-bit big-endian userspace 64/32 problem with the method_mask when registering agents. Merged.

- Sean's QoS changes. Merged.

- Sean's IB CM MRA interface changes. I merged these -- what the heck, if it breaks we can back them out.

ULPs:

- Pradeep's IPoIB CM support for devices that don't have SRQs. Sean started reviewing but I didn't see any updated patches.

- Moni's IPoIB bonding support. Seems like we found a clean set of changes, and these will go in via another (Jeff Garzik's?) tree.

- Rolf's IPoIB MGID scope changes. No review progress here.

- Eli and Michael's IPoIB stateless offload (checksum offload, LSO, LRO, etc). Not much review progress here; I'll try to chip away at the series and see what we can get into 2.6.24.

- Or's IPoIB/userspace multicast coexistence stuff. I think we've converged on this; I'll merge this once a final version of the patch appears.

HW specific:

- I already merged patches to enable MSI-X by default for mthca and mlx4. I hope there aren't too many systems that get hosed if a MSI-X interrupt is generated.

- Jack and Michael's mlx4 FMR support. Merged. I guess the fix for running in Xen domU may need to wait for 2.6.25, but I'll see what I can do.

- ehca patch queue. Merged everything I think.

- Steve's mthca router mode support. No one looked at it, seems like it's at risk of missing the window.

- Arthur's mthca doorbell alignment fixes. I still need to check various approaches; I'll definitely merge something for 2.6.24.

- Michael's mlx4 WQE shrinking patch. May miss the window and go for 2.6.25, I'll see if I can get to it.

Here are a few topics that I believe will not be ready in time for the 2.6.24 window and will need to wait for 2.6.25:

- Multiple CQ event vector support. I haven't seen any discussions about how ULPs or userspace apps should decide which vector to use, and hence no progress has been made since we deferred this during the 2.6.23 merge window.

- XRC. Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.

BOILERPLATE
===========

I keep patches in a git tree, available from

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

There are several branches of interest in this tree:

for-2.6.23 - changes queued for merging into the current kernel release for-2.6.24 - changes queued for the next merge window for-linus - changes I have asked Linus to pull upstream
for-mm - pulled by Andrew for inclusion in -mm

I frequently rewrite history and rebase my tree, so the best way to track it is to keep a clone of Linus's tree around and then pull a fresh copy of my tree with

    git clone --reference /path/to/linus/tree \
        git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

If you would like me to merge a patch, please send it to me as soon as it is ready. Do NOT wait for the merge window to open; if your change is not strictly a fix and you send it to me after the merge window opens, then it will likely have to wait for the next merge window. Please let me know if your patch is a fix that should go into the current release or if it can wait for the next merge window; if it is a fix, please describe the severity of the issue your are fixing, so I can make a good judgement about which release it should go into.

Including a good changelog entry that explains what you are changing, why you are changing it, and how your change accomplishes your goal will greatly increase the chance of your patch being merged promptly. Getting an independent review and a Reviewed-by: line also helps a lot.

The files Documentation/SubmittingPatches and Documentation/SubmitChecklist in kernel source tree also have a lot of good advice that makes it easier for me to handle your patches.

DETAILS
=======

Here is the complete list of patches I have in my for-2.6.24 branch:

Ali Ayoub (1):
IB/sa: Error handling thinko fix

Anton Blanchard (3):
IB/fmr_pool: Clean up some error messages in fmr_pool.c IB/ehca: Make output clearer by removing some debug messages IB/ehca: Export module parameters in sysfs

Dotan Barak (1):
mlx4_core: Use enum value GO_BIT_TIMEOUT_MSECS

Eli Cohen (2):
IPoIB: Fix typo to end statement with ';' instead of ','
IPoIB: Fix error path memory leak

Hoang-Nam Nguyen (4):
IB/ehca: Use remap_4k_pfn() to map firmware contexts to user space IB/ehca: Fix large page HW cap defines
IB/ehca: Fix mem leak of firmware ctrlblock in ehca_create_srq() IB/ehca: Adjust 64-bit alignment of create QP response for userspace

Jack Morgenstein (5):
IB/mlx4: Display misc device information under /sys/class/infiniband/
mlx4_core: Support ICM tables in coherent memory
mlx4_core: Write MTTs from CPU instead with of WRITE_MTT FW command IB/mlx4: Implement FMRs
mlx4_core: Increase max number of QPs per multicast group to 56

Joachim Fenkes (11):
IB/ehca: Refactor hvcall tracing
IB/ehca: Print return codes as signed decimal integers
IB/ehca: ehca_gen_warn() should always print
IB/ehca: Add check for max #SGE to create_qp()
IB/ehca: Path migration support
IB/ehca: Serialize MR alloc and MR free hvCalls
IB/ehca: Replace get_paca()->paca_index by the more portable raw_smp_processor_id() IB/ehca: Bump version number and change its format
IB/umem: Add hugetlb flag to struct ib_umem
IB/ehca: Only use MR large pages for hugetlb regions
IB/ehca: Return srq_attr->max_sge in ehca_query_srq()

Michael S. Tsirkin (2):
mlx4_core: Enable MSI-X by default
IB/mthca: Enable MSI-X by default

Peter Oruba (1):
IB/mthca: Use PCI-X/PCI-Express read control interfaces

Ralph Campbell (1):
IB/core: Fix handling of multicast response failures

Roland Dreier (14):
IPoIB: Make sure no receives are handled when stopping device
IB: find_first_zero_bit() takes unsigned pointer
mlx4_core: Don't free special QPs in QP number bitmap IB/mlx4: Use __set_data_seg() in mlx4_ib_post_recv() IB/ehca: Include <linux/mutex.h> from ehca_classes.h IB/mlx4: Fix up SRQ limit_watermark endianness
IB/iser: Remove unnecessary includes
mlx4_core: Change capability decoding: SRC->XRC
IB/umad: Add P_Key index support
IB/umad: Fix bit ordering and 32-on-64 problems on big endian systems IB/uverbs: Make ib_uverbs_release_event_file() static
mlx4_core: Reserve the correct number of MTT segments
mlx4_core: Fix meaning of dev->caps.reserved_mtts IB/mthca: Increase max number of QPs per multicast group to 56

Satyam Sharma (1):
IB/ehca: Misc cpuinit section annotations and #ifdef cleanups

Sean Hefty (7):
IPoIB: Specify Traffic Class with path record queries for QoS support IB/sa: Add new QoS fields to path record
RDMA/cma: Add ability to specify type of service RDMA/ucma: Allow user space to set service type
IB/srp: Add QoS support through service ID
IB/cm: Modify interface to send MRAs in response to duplicate messages RDMA/cma: Queue IB CM MRAs to avoid unnecessary remote retries

Stefan Roscher (2):
IB/ehca: Small QP userspace support
IB/ehca: Support more than 4k QPs for userspace and kernelspace

Steve Wise (2):
RDMA/cxgb3: Make the iw_cxgb3 module parameters writable RDMA/cma: Use neigh_event_send() to start neighbour discovery
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

#8 lore raw
In reply to Roland Dreier
Quoted 4 lines · Roland Dreier

 - XRC.  Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.

I posted the first draft patch set to the OpenFabrics list on September 18:

[ofa-general] [PATCH 0 of 5] XRC implementation patches (libibverbs, libmlx4, core, mlx4) Jack Morgenstein
[ofa-general] [PATCH 1 of 5] libibverbs: XRC implementation Jack Morgenstein
[ofa-general] [PATCH 2 of 5] libmlx4: XRC implementation Jack Morgenstein
[ofa-general] [PATCH 3 of 5] core: XRC implementation for fd = -1 when opening an xrc domain Jack Morgenstein
[ofa-general] [PATCH 4 of 5] core: XRC implementation -- add support for working with file descriptors Jack Morgenstein
[ofa-general] [PATCH 5 of 5] mlx4: XRC implementation Jack Morgenstein

The above patch set implements XRC for userspace verbs layer and below. The Kernel-space verbs implementation (1 more patch) is finished, but as yet untested.

- Jack

#9 lore raw
Quoted 5 lines
Quoted 4 lines

 - XRC.  Given the length of the backlog above and the fact that a first draft of this code has not been posted yet, I don't see any way that we could have something this major ready in time.

I posted the first draft patch set to the OpenFabrics list on September 18:

Sorry, I didn't update that text. But my backlog is still too big to get XRC into 2.6.24.

- R.

#6 lore raw
In reply to Steve Wise
Quoted 1 line

No mention about the iwarp port space issue?

I don't think we're at a stage where I'm prepared to merge something-- we all agree the latest patch has serious drawbacks, and it commits us to a suboptimal interface that is userspace-visible.

Quoted 1 line

I'm at a loss as to how to proceed.

Could we try to do some cleanups to the net core to make the alias stuff less painful? eg is there any sane way to make it possible for a device that creates 'eth0' to also create an 'iw0' alias without an assigning an address?

- R.

#7 lore raw
Quoted 6 lines · Roland Dreier
Quoted 1 line

No mention about the iwarp port space issue?

I don't think we're at a stage where I'm prepared to merge something-- we all agree the latest patch has serious drawbacks, and it commits us to a suboptimal interface that is userspace-visible.

Fair enough.

Quoted 7 lines
Quoted 1 line

I'm at a loss as to how to proceed.

Could we try to do some cleanups to the net core to make the alias stuff less painful? eg is there any sane way to make it possible for a device that creates 'eth0' to also create an 'iw0' alias without an assigning an address?

Well, "alias" interfaces really don't exist. ethX:iw is really just adding a address record (struct in_ifaddr) to ethX. So in the current core design, adding an alias without an address is really adding the alias with address 0.0.0.0. And I think the core net code assumes if an in_ifaddr struct exists for a device, then its IP address is indeed valid.

So I think the changes wouldn't be small to enhance the design to add a concept of an alias interface.

I'll look into this more though.

Steve.

#2 lore raw
In reply to Pradeep Satyanarayana
Quoted 7 lines

Roland, I submitted an updated patch incorporating some of Sean's comments within a day or two. Rest of comments pertained to restructuring the code and adding some additional module parameters.

This would require more discussions since some of these had been already discussed previously. We had decided upon this code structure after a lot of discussions and incorporating these would be undoing some of that.

Can you give a link to your current final version of the patch?

Sean, what's your opinion of where we stand?

Since module parameters create a userspace-visible interface that we are stuck with for a long time, we definitely have to get at least that much right before merging.

- R.

#3 lore raw
Quoted 3 lines

Can you give a link to your current final version of the patch?

Sean, what's your opinion of where we stand?

Let me look back over the last version that was sent and reply back later today or tomorrow. Several of my initial comments were on code structure.

Quoted 3 lines

Since module parameters create a userspace-visible interface that we are stuck with for a long time, we definitely have to get at least that much right before merging.

I was taking a slightly different view of the design. It would be nice to agree on whether SRQ should be separated from the QP type before merging upstream, even if the implementation doesn't immediately support all available options.

- Sean

#4 lore raw
In reply to Roland Dreier
Quoted 10 lines · Roland Dreier
Quoted 7 lines

Roland, I submitted an updated patch incorporating some of Sean's comments within a day or two. Rest of comments pertained to restructuring the code and adding some additional module parameters.

This would require more discussions since some of these had been already discussed previously. We had decided upon this code structure after a lot of discussions and incorporating these would be undoing some of that.

Can you give a link to your current final version of the patch?

Roland,

This is the link to the last one that I submitted on 09/18.

http://lists.openfabrics.org/pipermail/general/2007-September/040917.html

Pradeep