Skip to contentSkip to conversation

please pull infiniband.git

GIT PULL Roland Dreier 9 participants 31 messages Jul 12, 2007 – Jul 18, 2007 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Linus, please pull from

master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

This tree is also available from kernel.org mirrors at:

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

This will get the first batch of changes for the 2.6.23 merge window:

Andrew Morton (1):

      IB: Fix ib_umem_get() when npages == 0

Arthur Jones (3):
IB/ipath: Update MAINTAINERS entry
IB/ipath: Test interrupts at driver startup IB/ipath: Remove bogus RD_ATOMIC checks from modify_qp

Bryan O'Sullivan (1):
IB/ipath: Include <linux/vmalloc.h> to fix ppc64 build

Dave Olson (5):
IB/ipath: Support the IBA6110 revision 4
IB/ipath: Fix the mtrr_add args for chips with 2 buffer sizes IB/ipath: Use S_ABORT not cancel and abort on exit freeze mode after recovery IB/ipath: Be more cautious about coming out of freeze mode IB/ipath: Change version wording to be less confusing with release number

Dotan Barak (2):
mlx4_core: Get the maximum message size from reported device capabilities IB/core: Take sizeof the correct pointer when calling kmalloc()

Hal Rosenstock (1):
IB/mad: Enhance SMI for switch support

Hoang-Nam Nguyen (3):
IB/ehca: Change scaling_code parameter description to match default value IB/ehca: Report RDMA atomic attributes in query_qp() IB/ehca: Improve latency by unlocking after triggering the hardware

Jack Morgenstein (2):
IB/mlx4: Implement query QP
IB/mlx4: Implement query SRQ

Jan Engelhardt (1):
IB: Use menuconfig for InfiniBand menu

Joachim Fenkes (9):
IB/ehca: Refactor "maybe missed event" code
IB/ehca: HW level, HW caps and MTU autodetection
IB/ehca: QP code restructuring in preparation for SRQ
IB/ehca: add Shared Receive Queue support
IB/ehca: Lock renaming, static initializers
IB/ehca: Refactor sync between completions and destroy_cq using atomic_t IB/ehca: Change idr spinlocks into rwlocks
IB/ehca: Return QP pointer in poll_cq()
IB/ehca: Notify consumers of LID/PKEY/SM changes after nondisruptive events

Joan Eslinger (1):
IB/ipath: Change use of constants for TID type to defined values

John Gregor (2):
IB/ipath: Remove incompletely implemented ipath_runtime flags and code IB/ipath: Update copyright dates

Mark Debbage (2):
IB/ipath: Correct checking of swminor version field when using subports IB/ipath: Make handling of one subport consistent

Michael Albaugh (4):
IB/ipath: Support blinking LEDs with an led_override file IB/ipath: Lock and always use shadow copies of GPIO register IB/ipath: Log "active" time and some errors to EEPROM IB/ipath: Add capability to modify PBC word

Michael S. Tsirkin (2):
IB/mlx4: Include linux/mutex.h from mlx4_ib.h
mlx4_core: Include linux/mutex.h from mlx4.h

Ralph Campbell (10):
IB/ipath: Fix problem with next WQE after a UC completion IB/ipath: Fix local loopback bug when waiting for resources IB/ipath: Set M bit in BTH according to IB spec
IB/ipath: Fix RDMA read retry code
IB/ipath: Wait for PIO available interrupt
IB/ipath: Fix possible data corruption if multiple SGEs used for receive IB/ipath: Duplicate RDMA reads can cause responder to NAK inappropriately IB/ipath: Add barrier before updating WC head in shared memory IB/ipath: Lower default number of kernel send buffers IB/ipath: Remove support for preproduction HTX InfiniPath cards

Robert Walsh (5):
IB/ipath: Fix maximum MTU reporting
IB/ipath: Fill in some missing FMR-related fields in query_device IB/ipath: Send ACK invalid where appropriate
IB/ipath: ipath_poll fixups and enhancements
IB/ipath: Clean send flags properly on QP reset

Roland Dreier (5):
IB: Remove garbage non-ASCII characters from comments
IB: Update mailing list address
IPoIB/cm: Fix warning if IPV6 is not enabled
IPoIB: Recycle loopback skbs instead of freeing and reallocating
IB: Update MAINTAINERS with Hal's new email address

Sean Hefty (7):
IB/ipath: return correct PortGUID in NodeInfo
IB/sa: Make sure SA queries use default P_Key
IB/cm: Use spin_lock_irq() instead of spin_lock_irqsave() when possible IB/cm: Include HCA ACK delay in local ACK timeout
IB/cm: cm_msgs.h should include ib_cm.h
IB/cm: Fix handling of duplicate SIDR REQs
IB/cm: Send no match if a SIDR REQ does not match a listen

Shani Moideen (1):
IB/mthca: Replace memset(<addr>, 0, PAGE_SIZE) with clear_page(<addr>)

Stefan Roscher (2):
IB/ehca: Support UD low-latency QPs
IB/ehca: Set SEND_GRH flag for all non-LL UD QPs on eHCA2

Steve Wise (6):
RDMA/cxgb3: Streaming -> RDMA mode transition fixes RDMA/cxgb3: TERMINATE WRs can hang the tx ofld queue RDMA/cxgb3: Don't count neg_adv abort_req_rss messages as real aborts RDMA/cxgb3: ctrl-qp init/clear shouldn't set the gen bit RDMA/cxgb3: Don't post TID_RELEASE message
RDMA/cxgb3: Don't abort after failures sending the mpa reply

WANG Cong (1):
RDMA/cxgb3: Check return of kmalloc() in iwch_register_device()

 MAINTAINERS                                       |   15 +-
 drivers/infiniband/Kconfig                        |   15 +-
 drivers/infiniband/core/agent.c                   |   19 +-
 drivers/infiniband/core/cm.c                      |  247 ++++---
 drivers/infiniband/core/cm_msgs.h                 |    1 +
 drivers/infiniband/core/cma.c                     |    1 -
 drivers/infiniband/core/mad.c                     |   50 ++-
 drivers/infiniband/core/multicast.c               |    2 +-
 drivers/infiniband/core/sa.h                      |    2 +-
 drivers/infiniband/core/sa_query.c                |   87 ++-
 drivers/infiniband/core/smi.c                     |   16 +-
 drivers/infiniband/core/smi.h                     |    2 +
 drivers/infiniband/core/sysfs.c                   |    2 +-
 drivers/infiniband/core/ucm.c                     |    1 -
 drivers/infiniband/core/umem.c                    |    1 +
 drivers/infiniband/hw/amso1100/Kconfig            |    2 +-
 drivers/infiniband/hw/cxgb3/Kconfig               |    2 +-
 drivers/infiniband/hw/cxgb3/cxio_hal.c            |    6 +-
 drivers/infiniband/hw/cxgb3/cxio_wr.h             |    3 +-
 drivers/infiniband/hw/cxgb3/iwch_cm.c             |  108 ++--
 drivers/infiniband/hw/cxgb3/iwch_cm.h             |    1 +
 drivers/infiniband/hw/cxgb3/iwch_provider.c       |    7 +-
 drivers/infiniband/hw/cxgb3/iwch_qp.c             |    7 +-
 drivers/infiniband/hw/ehca/Kconfig                |    2 +-
 drivers/infiniband/hw/ehca/ehca_av.c              |    6 +-
 drivers/infiniband/hw/ehca/ehca_classes.h         |   75 ++-
 drivers/infiniband/hw/ehca/ehca_classes_pSeries.h |    4 +-
 drivers/infiniband/hw/ehca/ehca_cq.c              |   50 +-
 drivers/infiniband/hw/ehca/ehca_hca.c             |   61 ++-
 drivers/infiniband/hw/ehca/ehca_irq.c             |  140 +++--
 drivers/infiniband/hw/ehca/ehca_irq.h             |    1 -
 drivers/infiniband/hw/ehca/ehca_iverbs.h          |   18 +
 drivers/infiniband/hw/ehca/ehca_main.c            |   98 +++-
 drivers/infiniband/hw/ehca/ehca_qp.c              |  751 +++++++++++++++------
 drivers/infiniband/hw/ehca/ehca_reqs.c            |   85 ++-
 drivers/infiniband/hw/ehca/ehca_tools.h           |    1 +
 drivers/infiniband/hw/ehca/ehca_uverbs.c          |   13 +-
 drivers/infiniband/hw/ehca/hcp_if.c               |   58 +-
 drivers/infiniband/hw/ehca/hcp_if.h               |    1 -
 drivers/infiniband/hw/ehca/hipz_hw.h              |   19 +
 drivers/infiniband/hw/ehca/ipz_pt_fn.h            |   28 +-
 drivers/infiniband/hw/ipath/Kconfig               |    2 +-
 drivers/infiniband/hw/ipath/ipath_common.h        |   33 +-
 drivers/infiniband/hw/ipath/ipath_cq.c            |    7 +-
 drivers/infiniband/hw/ipath/ipath_debug.h         |    2 +-
 drivers/infiniband/hw/ipath/ipath_diag.c          |   41 +-
 drivers/infiniband/hw/ipath/ipath_driver.c        |  187 +++++-
 drivers/infiniband/hw/ipath/ipath_eeprom.c        |  303 ++++++++-
 drivers/infiniband/hw/ipath/ipath_file_ops.c      |  205 ++++--
 drivers/infiniband/hw/ipath/ipath_fs.c            |    9 +-
 drivers/infiniband/hw/ipath/ipath_iba6110.c       |  101 ++--
 drivers/infiniband/hw/ipath/ipath_iba6120.c       |   92 ++-
 drivers/infiniband/hw/ipath/ipath_init_chip.c     |   26 +-
 drivers/infiniband/hw/ipath/ipath_intr.c          |  141 ++++-
 drivers/infiniband/hw/ipath/ipath_kernel.h        |   85 +++-
 drivers/infiniband/hw/ipath/ipath_keys.c          |    2 +-
 drivers/infiniband/hw/ipath/ipath_layer.c         |    2 +-
 drivers/infiniband/hw/ipath/ipath_layer.h         |    2 +-
 drivers/infiniband/hw/ipath/ipath_mad.c           |   11 +-
 drivers/infiniband/hw/ipath/ipath_mmap.c          |    2 +-
 drivers/infiniband/hw/ipath/ipath_mr.c            |    2 +-
 drivers/infiniband/hw/ipath/ipath_qp.c            |   19 +-
 drivers/infiniband/hw/ipath/ipath_rc.c            |  116 +++-
 drivers/infiniband/hw/ipath/ipath_registers.h     |    2 +-
 drivers/infiniband/hw/ipath/ipath_ruc.c           |   36 +-
 drivers/infiniband/hw/ipath/ipath_srq.c           |    4 +-
 drivers/infiniband/hw/ipath/ipath_stats.c         |   25 +-
 drivers/infiniband/hw/ipath/ipath_sysfs.c         |   43 ++-
 drivers/infiniband/hw/ipath/ipath_uc.c            |    9 +-
 drivers/infiniband/hw/ipath/ipath_ud.c            |    6 +-
 drivers/infiniband/hw/ipath/ipath_user_pages.c    |    2 +-
 drivers/infiniband/hw/ipath/ipath_verbs.c         |   29 +-
 drivers/infiniband/hw/ipath/ipath_verbs.h         |    3 +-
 drivers/infiniband/hw/ipath/ipath_verbs_mcast.c   |    2 +-
 drivers/infiniband/hw/ipath/ipath_wc_ppc64.c      |    2 +-
 drivers/infiniband/hw/ipath/ipath_wc_x86_64.c     |   29 +-
 drivers/infiniband/hw/mlx4/Kconfig                |    1 -
 drivers/infiniband/hw/mlx4/main.c                 |    6 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h              |    4 +
 drivers/infiniband/hw/mlx4/qp.c                   |  137 ++++
 drivers/infiniband/hw/mlx4/srq.c                  |   18 +
 drivers/infiniband/hw/mthca/Kconfig               |    2 +-
 drivers/infiniband/hw/mthca/mthca_allocator.c     |    2 +-
 drivers/infiniband/hw/mthca/mthca_eq.c            |    2 +-
 drivers/infiniband/ulp/ipoib/Kconfig              |    2 +-
 drivers/infiniband/ulp/ipoib/ipoib_cm.c           |    4 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c           |   33 +-
 drivers/infiniband/ulp/iser/Kconfig               |    2 +-
 drivers/infiniband/ulp/srp/Kconfig                |    2 +-
 drivers/net/cxgb3/version.h                       |    2 +-
 drivers/net/mlx4/fw.c                             |    3 +
 drivers/net/mlx4/fw.h                             |    1 +
 drivers/net/mlx4/main.c                           |    1 +
 drivers/net/mlx4/mlx4.h                           |    1 +
 drivers/net/mlx4/qp.c                             |   21 +
 drivers/net/mlx4/srq.c                            |   30 +
 include/linux/mlx4/device.h                       |    2 +
 include/linux/mlx4/qp.h                           |    3 +
 include/rdma/ib_cm.h                              |    1 -
 include/rdma/ib_mad.h                             |    3 +
 100 files changed, 2812 insertions(+), 1061 deletions(-)
#1 lore raw

Further 2.6.23 merge plans...

As you can see, I just sent my first 2.6.23 pull request for Linus. There are still a few more things I plan to do in before the merge window closes (in ~10 days):

- Write a patch to add P_Key handling to user_mad in the way we discussed (add an ioctl to enable P_Key mode without breaking old apps) -- I hope to do this tomorrow so we can get some review and testing before merging it.

- Take a look at Sean's local SA caching patches. I merged everything else from Sean's tree, but I'm still undecided about these. I haven't read them carefully yet, but even aside from that I don't have a good feeling about whether there's consensus about this yet. Any opinions about merging, for or against, would be appreciated here.

- Merge up pending hardware driver changes, including the cxgb3 and ehca patches I have in my queue, plus Jack's catastrophic error patch for mlx4.

- Try to get to resolution on the IPoIB "CM without SRQ" solution.

Also, if there's something I didn't list and didn't already include in the tree I asked Linus to pull, please remind me. I probably dropped it.

- R.

#2 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 8 lines · Roland Dreier

As you can see, I just sent my first 2.6.23 pull request for Linus. There are still a few more things I plan to do in before the merge window closes (in ~10 days):

- Write a patch to add P_Key handling to user_mad in the way we discussed (add an ioctl to enable P_Key mode without breaking old apps) -- I hope to do this tomorrow so we can get some review and testing before merging it.

Unfortunately, I'll mostly just be able to review it. Not sure how much testing I will be able to do but we'll see...

-- Hal

Quoted 23 lines

- Take a look at Sean's local SA caching patches. I merged everything else from Sean's tree, but I'm still undecided about these. I haven't read them carefully yet, but even aside from that I don't have a good feeling about whether there's consensus about this yet. Any opinions about merging, for or against, would be appreciated here.

- Merge up pending hardware driver changes, including the cxgb3 and ehca patches I have in my queue, plus Jack's catastrophic error patch for mlx4.

- Try to get to resolution on the IPoIB "CM without SRQ" solution.

Also, if there's something I didn't list and didn't already include in the tree I asked Linus to pull, please remind me. I probably dropped it.

- R.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

#3 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

In reply to Roland Dreier
Quoted 6 lines

- Take a look at Sean's local SA caching patches. I merged everything else from Sean's tree, but I'm still undecided about these. I haven't read them carefully yet, but even aside from that I don't have a good feeling about whether there's consensus about this yet. Any opinions about merging, for or against, would be appreciated here.

Obviously I'm biased here, but we've definitely seen local caching of path records (PR) greatly improve performance for large MPI job runs. (Our largest jobs wouldn't run without it.) The development of the feature was requested and paid for by the US national labs. Infinicon/Silverstorm/QLogic also had this feature in their IB stack for scalability reasons as well. PR caching is done in the stack today by IPoIB.

The implementation is hidden under the current kernel ib_sa interface, is disabled by default, and automatically fails over to standard PR queries if needed. Removing the cache later should be fairly easy.

But to be fair, it will be difficult to enable both QoS and local PR caching. To me, this would be the strongest reason against using it. However, QoS places additional burden on the SA, which will make scaling even more challenging.

- Sean

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 10 lines
Quoted 4 lines

But to be fair, it will be difficult to enable both QoS and local PR caching. To me, this would be the strongest reason against using it. However, QoS places additional burden on the SA, which will make scaling even more challenging.

my understanding is that the local sa does a path-query where all the fields except for the SGID are wildcard-ed. This means we expect the result to be a table of all the paths from this port to every other port on the fabrics for every pkey which this port is a member of etc, correct?

Quoted 4 lines

How do you plug here the QoS concept of SID in the path query? are you expecting the SA to realize what are all the services for which this port is a "member"? does the proposed definision for QoS management at the SA defines "services per gids" isn't it "what SL to user per Service"?

Or, thanks for rescuing this post.

I think this is an important question. If we merge the local SA stuff, then are we creating a problem for dealing with QoS? Are we going to have to revert the local SA stuff once the QoS stuff is available? Or is there at least a sketch of a plan on how to handle this?

- R.

#6 lore raw

RE: [ofa-general] Further 2.6.23 merge plans...

Quoted 2 lines

I think this is an important question. If we merge the local SA stuff, then are we creating a problem for dealing with QoS?

Yes - I do believe that merging PR caching and QoS together will be difficult. I don't think the problems are insurmountable, but I can't say that I have a definite solution for how to deal with this.

My current thoughts are that the purpose of the cache is to increase SA scalability on large clusters. We've seen issues running MPI, trying to establish all-to-all connections, on our 256 node cluster. (With 4 processes per node, this results in about 500,000+ PR queries hitting the SA.) The SA was swamped with work, and it wasn't trying to enforce QoS requirements across the cluster.

I just don't see how an SA that is already having trouble scaling to this number of nodes will be able to perform the additional task of providing QoS across the cluster. It may be that, at least initially, an administrator may need to select between enabling PR caching or QoS.

Quoted 2 lines

Are we going to have to revert the local SA stuff once the QoS stuff is available?

In the best case, the local SA will need enhancements added to the base support. In the worst case, a user would have to choose between QoS or PR caching. If all users choose QoS, then it would make sense to remove the local SA.

Quoted 1 line

Or is there at least a sketch of a plan on how to handle this?

This is only a rough idea, and it depends on how the QoS is implemented. The idea is to create a local QoS module on each node. The local QoS modules would be programmed with basic QoS information. For example, which types of queries to handle locally, versus which ones to forward to the SA. Locally handled queries would return PRs based on some QoS mapping table. (I haven't looked into any details of this.)

Ideally, local QoS modules would be programmed by a QoS master. This would require a new vendor-specific protocol, but would allow for a simple distributed QoS manager.

We will have a better idea of the issues and possible solutions once the QoS spec is released, and we can hold discussions on it. I will be working more details on QoS enhancements starting in the next couple of weeks.

- Sean

#7 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 3 lines

We will have a better idea of the issues and possible solutions once the QoS spec is released, and we can hold discussions on it. I will be working more details on QoS enhancements starting in the next couple of weeks.

Based on discussions so far, maybe the best path forward from here is to delay until 2.6.24. This will let us add this version to OFED 1.3 for more widespread testing, plus give us the time that we need to come up with a plan to integrate QoS with the local SA. I don't think we'll have a final implementation for QoS support by that time, but at least we'll have a better idea of the problems.

These patches are based on the same design used with OFED 1.2, but a
fair number of lines of code still changed, plus it added InformInfo registration. I don't believe anyone other than me has tested these patches with the local SA enabled. It's typically running on my systems, but because it automatically fails over to standard SA queries, it would be easy for me to miss problems.

- Sean

#8 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 4 lines

Based on discussions so far, maybe the best path forward from here is to delay until 2.6.24. This will let us add this version to OFED 1.3 for more widespread testing, plus give us the time that we need to come up with a plan to integrate QoS with the local SA.

I spoke with Matt on this, and he agreed with this plan.

- Sean

#9 lore raw

Re: Further 2.6.23 merge plans...

In reply to Roland Dreier
Quoted 2 lines

Also, if there's something I didn't list and didn't already include in the tree I asked Linus to pull, please remind me. I probably dropped it.

Any plans to do something with multiple EQ support in mthca?

Signature · 1 line
MST
#10 lore raw

Re: Further 2.6.23 merge plans...

Quoted 1 line

Any plans to do something with multiple EQ support in mthca?

I haven't done any work on it or seen anything from anyone else, so I expect this will have to wait for 2.6.24.

#11 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Hello Roland,

Quoted 4 lines
Quoted 1 line

Any plans to do something with multiple EQ support in mthca?

I haven't done any work on it or seen anything from anyone else, so I expect this will have to wait for 2.6.24.

We are working on IPoIB to use multiple EQ for multiple
links/connetions scalability. Does this mean this will wait for 2.6.24?

Thanks
Shirley

#12 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 2 lines

We are working on IPoIB to use multiple EQ for multiple
links/connetions scalability. Does this mean this will wait for 2.6.24?

I think so -- I don't want to merge something that first appears in the last few days of the merge window. The idea is to get your stuff queued up *before* the merge window opens.

- R.

#13 lore raw

Re: Further 2.6.23 merge plans...

In reply to Roland Dreier
Quoted 7 lines

Quoting Roland Dreier <[email protected]>:
Subject: Re: Further 2.6.23 merge plans...

Quoted 1 line

Any plans to do something with multiple EQ support in mthca?

I haven't done any work on it or seen anything from anyone else, so I expect this will have to wait for 2.6.24.

I'm surprised to hear this. How about this:
http://lists.openfabrics.org/pipermail/general/2007-May/035757.html

Signature · 1 line
MST

A message that is not in the archive (sent off-list or not yet received): OF72F6B9D1.F60C4EEF-ON8725731A.00506757-8825731A.0024BD1C@us.ibm.com

#15 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 7 lines

Quoting Shirley Ma <[email protected]>:
Subject: Re: [ofa-general] Re: Further 2.6.23 merge plans...

Michael,

I would like to try this patch for one adapter/2 ports scalability performance for IPoIB. Is this patch appliable to OFED-1.2?

Most likely yes.

Signature · 1 line
MST
#16 lore raw

Re: Further 2.6.23 merge plans...

In reply to Michael S. Tsirkin
Quoted 5 lines
Quoted 2 lines

I haven't done any work on it or seen anything from anyone else, so I expect this will have to wait for 2.6.24.

I'm surprised to hear this. How about this:
http://lists.openfabrics.org/pipermail/general/2007-May/035757.html

Sure, I remember that. But I haven't seen anything to suggest that anyone has given any further thought to the issues that were raised in that thread.

- R.

#17 lore raw

Re: Further 2.6.23 merge plans...

Quoted 6 lines

Quoting Roland Dreier <[email protected]>:
Subject: Re: Further 2.6.23 merge plans...

Quoted 2 lines
Quoted 2 lines

I haven't done any work on it or seen anything from anyone else, so I expect this will have to wait for 2.6.24.

Quoted 6 lines
Quoted 2 lines

I'm surprised to hear this. How about this:
http://lists.openfabrics.org/pipermail/general/2007-May/035757.html

Sure, I remember that. But I haven't seen anything to suggest that anyone has given any further thought to the issues that were raised in that thread.

Well, the only issue I recall is about the # of EQs we want to allocate. Was there something else?

Maybe code can be merged as-is (2 EQs) and the number be tuned later as applications start using vectors?

Signature · 1 line
MST
#18 lore raw

Re: Further 2.6.23 merge plans...

Quoted 2 lines

Well, the only issue I recall is about the # of EQs we want to allocate. Was there something else?

Yes, some ideas about how applications should pick which EQ to use. And how to handle CPU affinity. And whether we want to try to do something NUMA-aware.

- R.

#19 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

In reply to Roland Dreier

Hello Roland,

FYI, we are working on several IPoIB performance improvement
patches which are not on the list. Some of the patches are under test, some of the patches are going to be submitted soon. They are:

1. skb aggregations for both dev xmit(networking layer) and IPoIB send (it will be submitted soon, for both UD and RC mode)
2. multiple interrupt vectors in IPoIB for multiple links scalability (working on patch for both UD and RC mode)
3. split CQ and send completion aggregation (for both UD and RC mode)
4. LRO for IPoIB when generic LRO is available in networking layer. (UD mode only)

Some of them might be made in 2.6.23 timeline, some of them might
not, it depends on our test progress and community review feedback.

Thanks
Shirley

#20 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 3 lines

FYI, we are working on several IPoIB performance improvement
patches which are not on the list. Some of the patches are under test, some of the patches are going to be submitted soon. They are:

There is less than a week left in the merge window, and none of these changes has been reviewed yet. So being realistic, I don't think we can expect to get any of this into 2.6.23.

- R.

#21 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

In reply to Roland Dreier
Quoted 5 lines · Roland Dreier

As you can see, I just sent my first 2.6.23 pull request for Linus. There are still a few more things I plan to do in before the merge window closes (in ~10 days):

Till when can we insert mlx4 with FMRs?

Tziporet

#22 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 1 line

Till when can we insert mlx4 with FMRs?

2.6.22 came out on July 8, so I would expect 2.6.23-rc1 (the end of the merge window) to be July 22.

#23 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

In reply to Roland Dreier
Quoted 6 lines

- Take a look at Sean's local SA caching patches. I merged everything else from Sean's tree, but I'm still undecided about these. I haven't read them carefully yet, but even aside from that I don't have a good feeling about whether there's consensus about this yet. Any opinions about merging, for or against, would be appreciated here.

Does anyone other than Sean have an opinion here? If you want this feature, if you've tested it, if you don't think it's ready yet, whatever, please speak up -- I don't feel comfortable making a decision on my own here (although I will if I have to).

#24 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 11 lines · Roland Dreier
Quoted 6 lines

- Take a look at Sean's local SA caching patches. I merged everything else from Sean's tree, but I'm still undecided about these. I haven't read them carefully yet, but even aside from that I don't have a good feeling about whether there's consensus about this yet. Any opinions about merging, for or against, would be appreciated here.

Does anyone other than Sean have an opinion here? If you want this feature, if you've tested it, if you don't think it's ready yet, whatever, please speak up -- I don't feel comfortable making a decision on my own here (although I will if I have to).

Roland,

I would like to see these features moved upstream. DOE funded this
work as part of the items we see needing on our large scale IB deployment (both present and future). So from at least one big customer perspective we see this as useful.

I'll let others comment on specific code/implementation issues.

Thanks,

- Matt

Quoted 6 lines

_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Signature · 4 lines
Matt Leininger, Ph.D.
Lawrence Livermore National Laboratory
[email protected]
V 925-422-4110
#25 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 4 lines

I would like to see these features moved upstream. DOE funded this
work as part of the items we see needing on our large scale IB deployment (both present and future). So from at least one big customer perspective we see this as useful.

Does your reference to "present deployment" mean you are running this code now?

- R.

#26 lore raw

Re: [ofa-general] Further 2.6.23 merge plans...

Quoted 7 lines · Roland Dreier
Quoted 4 lines

I would like to see these features moved upstream. DOE funded this
work as part of the items we see needing on our large scale IB deployment (both present and future). So from at least one big customer perspective we see this as useful.

Does your reference to "present deployment" mean you are running this code now?

Indeed, my understanding is that the DOE uses an Open MPI device (I think its called PTE) which is implemented directly over libibverbs and hence no path queries are issued at all, if this is indeed the case, for them its more of a "for-the-future" thing.

Or.

#27 lore raw

Re: Further 2.6.23 merge plans...

In reply to Roland Dreier
Quoted 10 lines

Quoting Roland Dreier <[email protected]>:
Subject: Re: Further 2.6.23 merge plans...

Quoted 6 lines

- Take a look at Sean's local SA caching patches. I merged everything else from Sean's tree, but I'm still undecided about these. I haven't read them carefully yet, but even aside from that I don't have a good feeling about whether there's consensus about this yet. Any opinions about merging, for or against, would be appreciated here.

Quoted 4 lines

Does anyone other than Sean have an opinion here? If you want this feature, if you've tested it, if you don't think it's ready yet, whatever, please speak up -- I don't feel comfortable making a decision on my own here (although I will if I have to).

We have the patches applied in ofed 1.2.c with default module parameter set to caching disabled (ofed 1.2 had a different version of the patches, but caching is disabled by default there, too). At least in this configuration (caching disabled), all issues I've seen seem to be fixed now, and tests seem to be running smoothly.

So I think it's safe to merge it up if the module parameter is set to cache disabled by default.
No idea what happens if it's enabled though :)

Signature · 1 line
MST
#28 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 6 lines · Michael S. Tsirkin

We have the patches applied in ofed 1.2.c with default module parameter set to caching disabled (ofed 1.2 had a different version of the patches, but caching is disabled by default there, too). At least in this configuration (caching disabled), all issues I've seen seem to be fixed now, and tests seem to be running smoothly.

As far as I know Intel run with SA cache enabled on large clusters with Intel MPI

Tziporet

#29 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 4 lines

Quoting Tziporet Koren <[email protected]>:
Subject: Re: [ofa-general] Re: Further 2.6.23 merge plans...

Michael S. Tsirkin wrote:

Quoted 8 lines
Quoted 5 lines

We have the patches applied in ofed 1.2.c with default module parameter set to caching disabled (ofed 1.2 had a different version of the patches, but caching is disabled by default there, too). At least
in this configuration (caching disabled), all issues I've seen seem to be fixed now, and tests seem to be running smoothly.

As far as I know Intel run with SA cache enabled on large clusters with Intel MPI

With OFED 1.2 version of the code, right?

Signature · 1 line
MST
#30 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 7 lines · Michael S. Tsirkin
Quoted 3 lines

As far as I know Intel run with SA cache enabled on large clusters with Intel MPI

With OFED 1.2 version of the code, right?

Yes.
But maybe they also used the new module - Sean?

#31 lore raw

RE: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 5 lines
Quoted 3 lines

With OFED 1.2 version of the code, right?

Yes.
But maybe they also used the new module - Sean?

We actually use the OFED 1.2 version. So, this feature is in use, but not this specific implementation.

- Sean

#32 lore raw

Re: [ofa-general] Re: Further 2.6.23 merge plans...

Quoted 2 lines

We actually use the OFED 1.2 version. So, this feature is in use, but not this specific implementation.

Hmm... how much testing has the implementation being proposed for merging actually had?

It might still be OK if the answer is that it hasn't been tested at scale but that the basic code works and should behave the same as the code that was tested because the underlying design is the same... is at least that much true?