Skip to contentSkip to conversation

Move InfiniBand .h files

RFC Roland Dreier 9 participants 19 messages Aug 4, 2005 – Aug 5, 2005

lore
Share & tools
lore
#0 lore raw

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

The advantages of doing this are:

- The headers become more easily accessible to other parts of the tree that might want to use IB support. For example, an NFS/RDMA client probably wants to live under fs/
- It makes it easier to build IB modules outside the tree, since include/linux gets put in /lib/modules/<ver>/build. I realize that we don't really care about out-of-tree modules, but it is convenient to be able to develop and distribute new drivers that build against someone's existing kernels.
- We can kill off the ugly

EXTRA_CFLAGS += -Idrivers/infiniband/include

lines in our Makefiles.

The disadvantages are:

- It's churn with little technical merit.
- It makes it a little harder to pull the OpenIB svn tree into a kernel tree, since one would have to link both drivers/infiniband and include/linux/rdma instead of just drivers/infiniband. This problem goes away if/when OpenIB shifts over to a new source code control system.

Thanks,
Roland

#1 lore raw
Quoted 4 lines · Roland Dreier

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

I think it is a great idea.

Quoted 12 lines

The advantages of doing this are:

- The headers become more easily accessible to other parts of the tree that might want to use IB support. For example, an NFS/RDMA client probably wants to live under fs/
- It makes it easier to build IB modules outside the tree, since include/linux gets put in /lib/modules/<ver>/build. I realize that we don't really care about out-of-tree modules, but it is convenient to be able to develop and distribute new drivers that build against someone's existing kernels.
- We can kill off the ugly

Quoted 3 lines

EXTRA_CFLAGS += -Idrivers/infiniband/include

lines in our Makefiles.

One more advantage:

- It shows our willingness to push past just infiniband.

-tduffy

#2 lore raw
In reply to Roland Dreier
Quoted 4 lines · Roland Dreier

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

please only put userspace clean headers here; the rest is more or less private headers for your subsystem.

At minimum the headers should be split in separate files for shared-userspace and kernel (eg no overlap at all), but I'd vote for keeping the headers in your own dir.

#3 lore raw
Quoted 8 lines · Arjan van de Ven
Quoted 4 lines · Roland Dreier

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

please only put userspace clean headers here; the rest is more or less private headers for your subsystem.

Sorry...this smells like a rathole...but does this mean linus agrees the kernel subsystems should export headers suitable for both user space and kernel driver modules?

Historical, I thought glibc and other user space libs were expected to maintain their own set of header files. Maybe I'm just confused...

thanks,
grant

#4 lore raw
Quoted 10 lines · Grant Grundler
Quoted 8 lines · Arjan van de Ven
Quoted 4 lines · Roland Dreier

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

please only put userspace clean headers here; the rest is more or less private headers for your subsystem.

Quoted 6 lines

Sorry...this smells like a rathole...but does this mean linus agrees the kernel subsystems should export headers suitable for both user space and kernel driver modules?

Historical, I thought glibc and other user space libs were expected to maintain their own set of header files. Maybe I'm just confused...

there is a definite requirement for the kernel to expose SOME things to userspace. Well for SOMETHING to expose them. Right now most distros ship a hacked up version of the kernel headers (eg removed of all the kernel specific stuff and all the gpl inline code etc). A good part of making such an external project possible is to make a clean separation between userspace shared stuff and pure kernel internals.

#5 lore raw
In reply to Arjan van de Ven

Arjan> At minimum the headers should be split in separate files Arjan> for shared-userspace and kernel (eg no overlap at all), but Arjan> I'd vote for keeping the headers in your own dir.

This is already done -- the userspace ABI is defined in ib_user_mad.h, ib_user_verbs.h, etc.

The problem with keeping subsystem headers under drivers/infiniband is that it's ugly for, say, fs/nfs/Makefile to have to add -Idrivers/infiniband/include to its CFLAGS just because it's implementing NFS/RDMA.

Also, drivers/infiniband/include doesn't get put into the /lib/modules/<ver>/build directory, so it's a pain for people developing new drivers (this is a real complaint that came to me from a vendor developing a driver for a new piece of IB hardware).

Thanks,
Roland

#6 lore raw
Quoted 3 lines

Also, drivers/infiniband/include doesn't get put into the /lib/modules/<ver>/build directory,

that is a symlink not a directory, and a symlink to the full source...

#7 lore raw

Roland> Also, drivers/infiniband/include doesn't get put into the Roland> /lib/modules/<ver>/build directory,

Arjan> that is a symlink not a directory, and a symlink to the Arjan> full source...

Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their "support kernel builds" package. And if I use an external build directory (ie "O=") then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/

- R.

#8 lore raw
Quoted 11 lines · Roland Dreier

Roland> Also, drivers/infiniband/include doesn't get put into the Roland> /lib/modules/<ver>/build directory,

Arjan> that is a symlink not a directory, and a symlink to the Arjan> full source...

Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their "support kernel builds" package. And if I use an external build directory (ie "O=") then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/

If the external module uses a Kbuild file as explained in Documentation/kbuild/makefiles.txt and then uses both O= and M= when compiling the module there is no issue.

With respect to moving the .h files - please do so. drivers/infiniband should only include header used in that same directory. Not header files potentially uased by fs/.

Sam

#9 lore raw
Quoted 9 lines · Sam Ravnborg
Quoted 5 lines · Roland Dreier

Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their "support kernel builds" package. And if I use an external build directory (ie "O=") then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/

Quoted 7 lines

If the external module uses a Kbuild file as explained in Documentation/kbuild/makefiles.txt and then uses both O= and M= when compiling the module there is no issue.

With respect to moving the .h files - please do so. drivers/infiniband should only include header used in that same directory. Not header files potentially uased by fs/.

I think Roland was talking about the case where the running kernel was built with "O=", in which case the /lib/modules.../build symlink points to the build directory rather than the original source tree.

Does Kbuild handle this case properly?

Chris

#10 lore raw
Quoted 18 lines · Christopher Friesen
Quoted 9 lines · Sam Ravnborg
Quoted 5 lines · Roland Dreier

Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their "support kernel builds" package. And if I use an external build directory (ie "O=") then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/

Quoted 7 lines

If the external module uses a Kbuild file as explained in Documentation/kbuild/makefiles.txt and then uses both O= and M= when compiling the module there is no issue.

With respect to moving the .h files - please do so. drivers/infiniband should only include header used in that same directory. Not header files potentially uased by fs/.

Quoted 5 lines

I think Roland was talking about the case where the running kernel was built with "O=", in which case the /lib/modules.../build symlink points to the build directory rather than the original source tree.

Does Kbuild handle this case properly?

Yes it does.
/lib/modules/.../ contains two symlinks these days:

build -> always point to the directory containing the output of the build source -> always point to the kernel source

In the 'make' case where the kernel is built without using O= they point to the same directory.
In the 'make O=' case they point to different directories.

SUSE does ship with a make O= build kernel these days. Fedora IIRC has done an ugly hack and just copied over a number of files so a compile works in most cases - but then also use both symlink.

It has never been easier to build a module if the target is only the running kernel. Only when you adds backwards compatibility it gets messy :-(

Sam

#11 lore raw
In reply to Roland Dreier
Quoted 9 lines · Roland Dreier

Roland> Also, drivers/infiniband/include doesn't get put into the Roland> /lib/modules/<ver>/build directory,

Arjan> that is a symlink not a directory, and a symlink to the Arjan> full source...

Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their "support kernel builds" package.

so what makes you think they will ship include/infiniband ?

#12 lore raw
In reply to Roland Dreier

Seems reasonable to me...

Steve.

Quoted 47 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Roland Dreier
Sent: Thursday, August 04, 2005 12:32 PM
To: [email protected]; [email protected]
Subject: [openib-general] [RFC] Move InfiniBand .h files

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

The advantages of doing this are:

- The headers become more easily accessible to other parts of the tree that might want to use IB support. For example, an NFS/RDMA client probably wants to live under fs/
- It makes it easier to build IB modules outside the tree, since include/linux gets put in /lib/modules/<ver>/build. I realize that we don't really care about out-of-tree modules, but it is convenient to be able to develop and distribute new drivers that build against someone's existing kernels.
- We can kill off the ugly

EXTRA_CFLAGS += -Idrivers/infiniband/include

lines in our Makefiles.

The disadvantages are:

- It's churn with little technical merit.
- It makes it a little harder to pull the OpenIB svn tree into a kernel tree, since one would have to link both drivers/infiniband and include/linux/rdma instead of just drivers/infiniband. This problem goes away if/when OpenIB shifts over to a new source code control system.

Thanks,
Roland
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general

#13 lore raw
In reply to Roland Dreier

On Thu, Aug 04, 2005 at 10:32:14AM -0700, Roland Dreier wrote:
...
I agree with the rename/relocation of the header files for the reasons you mentioned.

Quoted 5 lines

- It makes it a little harder to pull the OpenIB svn tree into a kernel tree, since one would have to link both drivers/infiniband and include/linux/rdma instead of just drivers/infiniband. This problem goes away if/when OpenIB shifts over to a new source code control system.

Any thoughts on renaming drivers/infiniband to drivers/rdma at the same time?

If you are going to churn...don't be shy about it :^)

grant

#14 lore raw

Grant> Any thoughts on renaming drivers/infiniband to drivers/rdma Grant> at the same time?

Grant> If you are going to churn...don't be shy about it :^)

Well, I'd rather avoid churn for purely political reasons. The main point of my proposal is to move the includes from drivers/ to include/, but while we're at it me might as well pick a more neutral directory name.

Moving drivers/infiniband to drivers/rdma has no technical merit right now, so I'd rather wait and see how it makes sense to organize the code we end up with.

- R.

#15 lore raw
In reply to Roland Dreier
Quoted 4 lines · Roland Dreier

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

I think it is a good idea.

Quoted 5 lines

The advantages of doing this are:

- The headers become more easily accessible to other parts of the tree that might want to use IB support. For example, an NFS/RDMA client probably wants to live under fs/

net/sunrpc/ has also been proposed.

#16 lore raw
In reply to Roland Dreier
Quoted 4 lines · Roland Dreier

I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts.

include/rmda, please. not need for the linux/ component.

Quoted 3 lines

The advantages of doing this are:

#17 lore raw

Christoph> include/rmda, please. not need for the linux/ component.

OK, fair enough. Any objection to this?

- R.

#18 lore raw
Quoted 3 lines · Roland Dreier

Christoph> include/rmda, please. not need for the linux/ component.

OK, fair enough. Any objection to this?

No, makes sense.

Sam