Skip to contentSkip to conversation

InfiniBand 2.6.17 merge plans

DISCUSSION Roland Dreier 8 participants 22 messages Mar 27, 2006 – Apr 3, 2006 InfiniBand

lore
Share & tools
lore
#0 lore raw

As we enter the second week of the 2.6.17 merge window, I thought it might be helpful to give an update on what I'm thinking about the remaining pending pieces. This is really to stimulate discussion -- if there's something that you think really should (or shouldn't) be merged, let me know.

Of course bug fixes are always welcome and can be merged pretty much any time -- that's the whole point of the stabilization period after the merge window.

* PathScale ipath driver. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git ipath

There is still ia64 build breakage and a lot of sparse endian annotations to clean up before it's mergeable. Also, there are two issues I still see:

- Working around the IB midlayer SMA (subnet management agent) / implementation with a character device when ib_mad isn't loaded. Maybe I'm off-base here objecting to this. Hal and Sean, as the ib_mad guys, I'd be especially interested in your opinion of this.

- The driver depends on 64BIT && PCI_MSI, and is basically x86_64-only for practical purposes. I think this is OK as far as a merge goes, but it would be nice to be able to use a PCIe device on any system with PCIe slots...

* RDMA CM. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm

I think this is OK to merge, but I don't see much pull to get it in right now. There are three consumers on the horizon:

- userspace RDMA CM, which exports the abstraction to userspace. The feeling is that this interface needs more time to mature.

- NFS/RDMA. Not ready to merge right now.

- iSER. Maybe ready to merge -- I haven't heard anything recently.

My feeling is that without someone making a case for why this should go in now, I'm going to hold off.

* IPoIB tunables. This is probably OK but I haven't seen any patches yet, so we're running out time.

* SRP FMRs. I have a patch that I'm pretty happy with now, so I think this will go in. This will need a lot of testing, or we may have to turn it off by default for 2.6.17 final

So if you care about any of this, let me know what you think. And if there's something not on this list and not in

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-2.6.17

please make sure I know about it, or it won't get merged.

Thanks,
Roland

#1 lore raw
Quoted 4 lines · Roland Dreier

- Working around the IB midlayer SMA (subnet management agent) / implementation with a character device when ib_mad isn't loaded. Maybe I'm off-base here objecting to this. Hal and Sean, as the ib_mad guys, I'd be especially interested in your opinion of this.

I will need to spend some time reading through the PathScale code on this. I've
followed most of the discussion, though not in detail. Right now, I don't see
what's being gained by not loading ib_mad. For instance, how does the driver handle loading ib_mad later, say when another IB device is added to the system that requires it?

Quoted 7 lines

* RDMA CM. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm

I think this is OK to merge, but I don't see much pull to get it in right now. There are three consumers on the horizon:

Quoted 6 lines

- userspace RDMA CM, which exports the abstraction to userspace. The feeling is that this interface needs more time to mature.

- NFS/RDMA. Not ready to merge right now.

- iSER. Maybe ready to merge -- I haven't heard anything recently.

I agree that we need to let the userspace interface mature. And even the kernel interface could benefit from having some real users. The code was added to the -mm branch, correct?

The main drawback that we have not merging the RDMA CM with userspace support, is that OpenIB release 1.0 will have very poor connection management support. There's no easy way for userspace applications to obtain path records.

- Sean

#2 lore raw

Sean> I agree that we need to let the userspace interface mature. Sean> And even the kernel interface could benefit from having some Sean> real users. The code was added to the -mm branch, correct?

Yes, I've put both the rdma_cm and ipath branches into my for-mm branch, so Andrew will pick them up automatically.

- R.

#3 lore raw
In reply to Sean Hefty

BTW, what do you think of changing rdma_wq to be a GPL export, to give a hint that this is an internal symbol not really for general use?

In other words:

drivers/infiniband/core/addr.c +1 -1
index 6ebd1d1..cdf4888 100644
@@ -56,7 +56,7 @@ static DEFINE_MUTEX(lock);
 static LIST_HEAD(req_list);
 static DECLARE_WORK(work, process_req, NULL);
 struct workqueue_struct *rdma_wq;
-EXPORT_SYMBOL(rdma_wq);
+EXPORT_SYMBOL_GPL(rdma_wq);
 
 static int copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
 		     unsigned char *dst_dev_addr)
#4 lore raw
Quoted 2 lines · Roland Dreier

BTW, what do you think of changing rdma_wq to be a GPL export, to give a hint that this is an internal symbol not really for general use?

I'm actually testing a patch set now that moves rdma_wq internal to ib_addr, and
gives the RDMA CM its own WQ.

- Sean

#5 lore raw

Sean> I'm actually testing a patch set now that moves rdma_wq Sean> internal to ib_addr, and gives the RDMA CM its own WQ.

OK, that's better still.

BTW ib_local_sa also uses rdma_wq ... it might be worth fixing that, because it actually makes ib_local_sa depend on CONFIG_NET right now (since ib_addr can't build without CONFIG_NET).

Not that anyone is building kernels without CONFIG_NET...

- R.

#6 lore raw
Quoted 3 lines

BTW ib_local_sa also uses rdma_wq ... it might be worth fixing that, because it actually makes ib_local_sa depend on CONFIG_NET right now (since ib_addr can't build without CONFIG_NET).

Yes - I've fixed that as well.

- Sean

#7 lore raw
In reply to Roland Dreier

Roland wrote,

Quoted 2 lines

- userspace RDMA CM, which exports the abstraction to userspace. The feeling is that this interface needs more time to mature.

The userspace RDMA-CM is currently being used by uDAPL and seems to work fine for that application. Our testing so far shows that the code is stable. However, uDAPL also has a socket based mechanism that can used until the CMA
is merged. If others think it still needs more time to mature, then uDAPL has an interim solution.

my 2 cents on this one.

woody

#8 lore raw
In reply to Roland Dreier
Quoted 5 lines · Roland Dreier

As we enter the second week of the 2.6.17 merge window, I thought it might be helpful to give an update on what I'm thinking about the remaining pending pieces. This is really to stimulate discussion -- if there's something that you think really should (or shouldn't) be merged, let me know.

<snip>

Quoted 2 lines

- iSER. Maybe ready to merge -- I haven't heard anything recently.

iSER needs to complete the removal of the dummy socket provider and change the open-iscsi user-space interface.
We are coordinating this effort with open-iscsi, and rely on some open-iscsi changes that are planned to be pushed to 2.6.18.

Therefore the plan for iSER is to push it into 2.6.18

Dan

#9 lore raw

Dan> Therefore the plan for iSER is to push it into 2.6.18

OK, thanks.

Based on all of this I'm thinking that it's better to hold back the RDMA CM as well, since there will be no code using it ready to merge for 2.6.17.

- R.

#12 lore raw
In reply to Roland Dreier
Quoted 6 lines · Roland Dreier

* PathScale ipath driver. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git ipath

There is still ia64 build breakage and a lot of sparse endian annotations to clean up before it's mergeable.

These are all fixed now.

Quoted 2 lines

- Working around the IB midlayer SMA (subnet management agent) / implementation with a character device when ib_mad isn't loaded.

We dropped this in the round of changes I just submitted.

Quoted 4 lines

- The driver depends on 64BIT && PCI_MSI, and is basically x86_64-only for practical purposes. I think this is OK as far as a merge goes, but it would be nice to be able to use a PCIe device on any system with PCIe slots...

We'll see what we can do about this at some point, but I don't think it should block the merge.

	<b
#13 lore raw

updated InfiniBand 2.6.17 merge plans

In reply to Roland Dreier

OK, here's a quick update on 2.6.17 merge plans:

* PathScale ipath driver. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git ipath

The new version looks good to me. I'll merge it unless I hear an objection to the latest code.

* RDMA CM. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm

None of the users of this code look are to merge, and it looks like there's some changes in the design happening now. Seems like this can and should wait for 2.6.18.

* IPoIB tunable ring sizes. Still no patch yet.

* SRP FMRs. I have a patch that I like, but it's not totally stable. I may be hitting target bugs (ie in someone else's code). On the other hand I don't have any numbers showing a benefit, so I'm not sure if it's worth merging this.

* (new since last time) Improved static rate handling. I will get this in.

As before, if you care about any of this, let me know what you think. And if there's something not on this list and not in

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-2.6.17

please make sure I know about it, or it won't get merged.

Thanks,
Roland

#14 lore raw

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

Oh yeah, one other thing I plan to merge unless someone objects:

* Get rid of option for building IPoIB and mthca debug output unless EMBEDDED=y

- R.

#17 lore raw

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

In reply to Roland Dreier
Quoted 7 lines · Roland Dreier

* RDMA CM. In my git tree at

git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm

None of the users of this code look are to merge, and it looks like there's some changes in the design happening now. Seems like this can and should wait for 2.6.18.

Does it make sense to merge the first two patches of that patch series?

1. Provide common handling for marshalling data between userspace clients and kernel mode Infiniband drivers.

2. Extend the Infiniband CM to include private data comparisons as part of its connection request matching process.

This would make it easier to keep the openib tree in sync with the kernel.

- Sean

#15 lore raw

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

In reply to Roland Dreier
Quoted 3 lines · Roland Dreier

Oh yeah, one other thing I plan to merge unless someone objects:

* Get rid of option for building IPoIB and mthca debug output unless EMBEDDED=y

NACK. Just add a FOO_DEBUG config option, this has no�hing to do with EMBEDDED.

#16 lore raw

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

Roland> * Get rid of option for building IPoIB and mthca debug Roland> output unless EMBEDDED=y

Christoph> NACK. Just add a FOO_DEBUG config option, this has Christoph> noþhing to do with EMBEDDED.

I think you misunderstood (probably because what I wrote was very unclear). What I meant to say was that I wanted to make CONFIG_INFINIBAND_IPOIB_DEBUG=y, without the option to disable it, unless CONFIG_EMBEDDED is set. In Kconfig language, change it to:

config INFINIBAND_IPOIB_DEBUG
bool "IP-over-InfiniBand debugging" if EMBEDDED depends on INFINIBAND_IPOIB
default y

Do you still have a problem with that? The rationale is that we want the debugging output compiled in (it still needs to be enabled at runtime) unless the user really really knows they don't want it, because by the time we know we want the output, it's too late to recompile to kernel.

- R.

#18 lore raw

Re: updated InfiniBand 2.6.17 merge plans

In reply to Roland Dreier

Quoting r. Roland Dreier <[email protected]>:

Quoted 3 lines

Subject: updated InfiniBand 2.6.17 merge plans

OK, here's a quick update on 2.6.17 merge plans:

These are just the features, right? There's as the usual bugfixes which aren't listed here.

Signature · 2 lines
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies
#19 lore raw

Re: updated InfiniBand 2.6.17 merge plans

Michael> These are just the features, right? There's as the usual Michael> bugfixes which aren't listed here.

Yes, of course. We can always merge bugfixes. Although I don't know of any pending bugfixes that are not at least in my for-2.6.17 branch...

- R.

#10 lore raw
In reply to Roland Dreier

On the other hand, there are user-space consumers for CMA, such as udapl, so although ISER will attempt only at 2.6.18, maybe it is a good idea to get CMA into 2.6.17

Dan

Quoted 10 lines · Roland Dreier

Dan> Therefore the plan for iSER is to push it into 2.6.18

OK, thanks.

Based on all of this I'm thinking that it's better to hold back the RDMA CM as well, since there will be no code using it ready to merge for 2.6.17.

- R.

#11 lore raw

Dan> On the other hand, there are user-space consumers for CMA, Dan> such as udapl, so although ISER will attempt only at 2.6.18, Dan> maybe it is a good idea to get CMA into 2.6.17

Well, too late now anyway.

But Sean and I agreed that the userspace interface to CMA was not mature enough to merge for 2.6.17.

- R.

#20 lore raw

Re: updated InfiniBand 2.6.17 merge plans

In reply to Roland Dreier

Quoting r. Roland Dreier <[email protected]>:

Quoted 7 lines

Subject: Re: updated InfiniBand 2.6.17 merge plans

Michael> These are just the features, right? There's as the usual Michael> bugfixes which aren't listed here.

Yes, of course. We can always merge bugfixes. Although I don't know of any pending bugfixes that are not at least in my for-2.6.17 branch...

One thing I wanted to do was kill the global mutexes in ipoib, replacing all uses with priv->lock: we never do anything blocking there. Is this 2.6.17 material?

Signature · 2 lines
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies
#21 lore raw

Re: updated InfiniBand 2.6.17 merge plans

Michael> One thing I wanted to do was kill the global mutexes in Michael> ipoib, replacing all uses with priv->lock: we never do Michael> anything blocking there. Is this 2.6.17 material?

I don't have a strong feeling either way. It doesn't fix anything, but on the other hand it's a small low-risk cleanup.

- R.