Skip to contentSkip to conversation

IB/core: export struct ib_port

PATCH In review Ralph Campbell 4 participants 14 messages Nov 11, 2009 – Nov 12, 2009 12 messages not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 IB/core: export struct ib_port

This patch moves the definition of struct ib_port from sysfs.c to ib_verbs.h so that HCAs can create files in /sys/class/infiniband/<hca>/ports/<N>/

---

drivers/infiniband/core/sysfs.c +0 -8
index 158a214..e01f3e7 100644
@@ -39,14 +39,6 @@
 
 #include <rdma/ib_mad.h>
 
-struct ib_port {
-	struct kobject         kobj;
-	struct ib_device      *ibdev;
-	struct attribute_group gid_group;
-	struct attribute_group pkey_group;
-	u8                     port_num;
-};
-
 struct port_attribute {
 	struct attribute attr;
 	ssize_t (*show)(struct ib_port *, struct port_attribute *, char *buf);
include/rdma/ib_verbs.h +8 -0
index c179318..5d23957 100644
@@ -1154,6 +1154,14 @@ struct ib_device {
 	u8                           phys_port_cnt;
 };
 
+struct ib_port {
+	struct kobject         kobj;
+	struct ib_device      *ibdev;
+	struct attribute_group gid_group;
+	struct attribute_group pkey_group;
+	u8                     port_num;
+};
+
 struct ib_client {
 	char  *name;
 	void (*add)   (struct ib_device *);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1257966478.992.300.camel-/[email protected]

#2 lore raw
Quoted 4 lines

This patch moves the definition of struct ib_port from sysfs.c to ib_verbs.h so that HCAs can create files in /sys/class/infiniband/<hca>/ports/<N>/

um, maybe, but we need to see how it gets used first. How do you get the to struct ib_port in driver code? Maybe it would make more sense to add a way for low-level drivers to pass in port attributes that get added when the port structure gets created?

By the way, any plans to resume working on the upstream driver for qlogic HCAs? Do you still plan to deprecate ib_ipath and add a new driver for new devices?

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adafx8kx41h.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#4 lore raw
Quoted 8 lines · Roland Dreier
Quoted 3 lines

This patch moves the definition of struct ib_port from sysfs.c to ib_verbs.h so that HCAs can create files in /sys/class/infiniband/<hca>/ports/<N>/

um, maybe, but we need to see how it gets used first. How do you get the to struct ib_port in driver code? Maybe it would make more sense to add a way for low-level drivers to pass in port attributes that get added when the port structure gets created?

It is used by the new ib_qib driver to expose the SL to VL table since the user level MPI library (libpsm) constructs packets including the IB header. After the driver calls ib_register_device(), it calls device_create_file() to create the files in /sys/class/infiniband/qib0/. Then it uses struct ib_device->port_list to get the pointer to ib_port to add a directory similar to "gids" and "pkeys" for each SL.

Quoted 3 lines

By the way, any plans to resume working on the upstream driver for qlogic HCAs? Do you still plan to deprecate ib_ipath and add a new driver for new devices?

Yes, this is what I'm working on.
The patch is the only change outside of the hw/qib/ directory. Do you want to see a preview of the sysfs code?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1257970050.992.317.camel-/[email protected]

#6 lore raw
Quoted 8 lines

It is used by the new ib_qib driver to expose the SL to VL table since the user level MPI library (libpsm) constructs packets including the IB header. After the driver calls ib_register_device(), it calls device_create_file() to create the files in /sys/class/infiniband/qib0/. Then it uses struct ib_device->port_list to get the pointer to ib_port to add a directory similar to "gids" and "pkeys" for each SL.

Hmm, maybe we should just add a vls directory with sl0 ... sl15 or something like that in generic code? I don't see why this needs to be driver-specific code.

Quoted 3 lines

Yes, this is what I'm working on.
The patch is the only change outside of the hw/qib/ directory. Do you want to see a preview of the sysfs code?

I think the earlier you can post the whole driver, the sooner you'll get it upstream.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaaayswzrh.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#8 lore raw

On Wed, 11 Nov 2009, Roland Dreier wrote:

|  > It is used by the new ib_qib driver to expose the SL to VL table
|  > since the user level MPI library (libpsm) constructs packets including
|  > the IB header. After the driver calls ib_register_device(),
|  > it calls device_create_file() to create the files in
|  > /sys/class/infiniband/qib0/. Then it uses struct ib_device->port_list
|  > to get the pointer to ib_port to add a directory similar to "gids"
|  > and "pkeys" for each SL.
| 
| Hmm, maybe we should just add a vls directory with sl0 ... sl15 or
| something like that in generic code?  I don't see why this needs to be
| driver-specific code.

No particular reason, it just didn't seem likely to be useful on other
HCA drivers. I can redo the patches that way, if people think it's
the right thing to do.

|  > Yes, this is what I'm working on.
|  > The patch is the only change outside of the hw/qib/ directory.
|  > Do you want to see a preview of the sysfs code?
| 
| I think the earlier you can post the whole driver, the sooner you'll get
| it upstream.

Yeah, we know... Schedule and people conflicts have slowed this down.
We want to do it, and soon.

And yes, the ib_ipath is being fully deprecated. The "full set" of patches that adds ib_qib upstream will include a subset that drops
ib_ipath. All the bug fixes and feature work have been done for ib_qib

Dave Olson
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): alpine.LFD.1.10.0911111303080.25952-vxnkQ4oxbxUi9g6yJnKVd0EOCMrvLtNR@public.gmane.org

#10 lore raw
Quoted 8 lines
| Hmm, maybe we should just add a vls directory with sl0 ... sl15 or
| something like that in generic code?  I don't see why this needs to be
| driver-specific code.

No particular reason, it just didn't seem likely to be useful on other
HCA drivers. I can redo the patches that way, if people think it's
the right thing to do.

To me it does seem like something generic. SLtoVL table is required of all CAs, so we might as well create it for all IB devices... as I see it the advantages of having it core code are:

- no need to expose internals of sysfs code port structure to low level drivers (we could also avoid this layering violation by giving a generic way for low-level drivers to add port attributes)
- IB-specified info is available for all IB devices with the same format etc. It may not be important for non-qlogic devices but there is some utility in SL mapping for debugging etc.

the only disadvantage I see is that it adds the overhead of having those sysfs attributes for all systems with an RDMA devices, even if the qlogic driver is never loaded. But that overhead is pretty much just a small amount of extra code that will never be run and a few sysfs structures that will never be touched, so it just takes up a little bit of memory. For RDMA-using systems, I can't imagine it matters.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaaaysvf69.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#12 lore raw
Quoted 19 lines · Roland Dreier
Quoted 7 lines
| Hmm, maybe we should just add a vls directory with sl0 ... sl15 or
| something like that in generic code?  I don't see why this needs to be
| driver-specific code.

No particular reason, it just didn't seem likely to be useful on other
HCA drivers. I can redo the patches that way, if people think it's
the right thing to do.

To me it does seem like something generic. SLtoVL table is required of all CAs, so we might as well create it for all IB devices... as I see it the advantages of having it core code are:

- no need to expose internals of sysfs code port structure to low level drivers (we could also avoid this layering violation by giving a generic way for low-level drivers to add port attributes)
- IB-specified info is available for all IB devices with the same format etc. It may not be important for non-qlogic devices but there is some utility in SL mapping for debugging etc.

Quoted 8 lines

the only disadvantage I see is that it adds the overhead of having those sysfs attributes for all systems with an RDMA devices, even if the qlogic driver is never loaded. But that overhead is pretty much just a small amount of extra code that will never be run and a few sysfs structures that will never be touched, so it just takes up a little bit of memory. For RDMA-using systems, I can't imagine it matters.

- R.

While this is true for SLtoVL, we create other files which are device specific under the port directory too.
It seems like we might need to introduce a callback into the driver to create the port specific sysfs files.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1257981770.992.336.camel-/[email protected]

#14 lore raw
Quoted 5 lines

While this is true for SLtoVL, we create other files which are device specific under the port directory too.
It seems like we might need to introduce a callback into the driver to create the port specific sysfs files.

Umm, you could have said there were other things initially!

Anyway, rather than a callback, I guess we could just add a place to attach a set of port attributes to the structure that gets passed into ib_register_device() maybe?

And maybe we could clean up the existing code that does device_create_file() to use a list of device attributes also...

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaws1wtywk.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#16 lore raw

On Wed, 11 Nov 2009, Roland Dreier wrote:

| 
|  > While this is true for SLtoVL, we create other files which are
|  > device specific under the port directory too.
|  > It seems like we might need to introduce a callback into the driver to
|  > create the port specific sysfs files.
| 
| Umm, you could have said there were other things initially!

Those have been there "forever" in qib without requiring the change in the core sysfs code. It's only sysfs group entries that require the patch to expose ib_port.

| Anyway, rather than a callback, I guess we could just add a place to
| attach a set of port attributes to the structure that gets passed into
| ib_register_device() maybe?

Seems like major overkill to have callbacks, when all we need is to get the structure that "owns" (is the parent kobject of) the directory.

| And maybe we could clean up the existing code that does | device_create_file() to use a list of device attributes also...

Seems to be a rather different issue, to me.

Dave Olson
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): alpine.LFD.1.10.0911112136110.24052-vxnkQ4oxbxUi9g6yJnKVd0EOCMrvLtNR@public.gmane.org

#18 lore raw
Quoted 11 lines
|  > While this is true for SLtoVL, we create other files which are
|  > device specific under the port directory too.
|  > It seems like we might need to introduce a callback into the driver to
|  > create the port specific sysfs files.
| 
| Umm, you could have said there were other things initially!

Those have been there "forever" in qib without requiring the change in the core sysfs code. It's only sysfs group entries that require the patch to expose ib_port.

Quoted 3 lines · I'm confused. Ralph originally

It [struct ib_port] is used by the new ib_qib driver to expose the SL to VL table since the user level MPI library (libpsm) constructs packets including the IB header.

So if that's the only use, then I'd be in favor of just exposing the standard, generic SL-to-VL table info for all IB devices. If there are other per-port device-specific things, then let's give a clean way for devices to add per-port attributes without having to know the internals of how the RDMA core implements sysfs stuff.

Quoted 6 lines
| Anyway, rather than a callback, I guess we could just add a place to
| attach a set of port attributes to the structure that gets passed into
| ib_register_device() maybe?

Seems like major overkill to have callbacks, when all we need is to get the structure that "owns" (is the parent kobject of) the directory.

Yes, I agree that callbacks aren't really the best way. I was suggesting passing in the per-port attributes as part of the ib_device structure in ib_register_device().

Quoted 4 lines

| And maybe we could clean up the existing code that does | device_create_file() to use a list of device attributes also...

Seems to be a rather different issue, to me.

Yes, it's independent. But if we're passing in per-port attributes, we might as well take the opportunity to make the API rational and pass in per-device attributes too.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): ada1vk3ftp4.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#20 lore raw

On Thu, 12 Nov 2009, Roland Dreier wrote:

| 
|  > |  > While this is true for SLtoVL, we create other files which are
|  > |  > device specific under the port directory too.
|  > |  > It seems like we might need to introduce a callback into the driver to
|  > |  > create the port specific sysfs files.
|  > | 
|  > | Umm, you could have said there were other things initially!
|  > 
|  > Those have been there "forever" in qib without requiring the change
|  > in the core sysfs code.  It's only sysfs group entries that require
|  > the patch to expose ib_port.
| 
| OK, I'm confused.  Ralph originally said:
| 
|  > It [struct ib_port] is used by the new ib_qib driver to expose the SL
|  > to VL table since the user level MPI library (libpsm) constructs
|  > packets including the IB header.
| 
| So if that's the only use, then I'd be in favor of just exposing the
| standard, generic SL-to-VL table info for all IB devices.  If there are
| other per-port device-specific things, then let's give a clean way for
| devices to add per-port attributes without having to know the internals
| of how the RDMA core implements sysfs stuff.

The other per-port stuff doesn't require ib_port, because it's not
using the same parent kobject. They are separate directories with
separate files, not groups attached to the port kobject.

Ralph is correct, but the answer is somewhat misleading, in his response to your reply. The new SL2VL table is the only thing in the HCA driver that needs ib_port to be exposed so far, and "likely" to be the only thing ever.

| Yes, I agree that callbacks aren't really the best way.  I was
| suggesting passing in the per-port attributes as part of the ib_device
| structure in ib_register_device().

For us, that would be inconvenient, although we could probably make
it work. And it's not just an attribute list, we'd then have to
export functions somehow (via a pointer table, presumably). It all seems pretty inflexible, and a lot more work, for no real gain.

Dave Olson
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw
In reply to a message not in the archive
Quoted 4 lines · Ralph Campbell

While this is true for SLtoVL, we create other files which are device specific under the port directory too.
It seems like we might need to introduce a callback into the driver to create the port specific sysfs files.

Maybe give some thought to using a syscall interface through uverbs for some of this?

IMHO, sysfs is getting out of hand for rdma:

$ find /sys/class/infiniband/mlx4_0 -type f | wc -l
660
$ strace -o /tmp/t /opt/ofa-1.5/sbin/perfquery ; grep sys/ /tmp/t | wc -l
289

That is alot of syscalls just to send two SMPs.

It just seems to me there are not that many examples of APIs that require so much trundling through sysfs to do common every day application tasks.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20091111234744.GA1966-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#23 lore raw
Quoted 3 lines

Maybe give some thought to using a syscall interface through uverbs for some of this?

Actually I think for exposing SL-to-VL and other things like that, sysfs is pretty good. Having something usable from both scripts and programs seems pretty useful, and having an opaque uverbs interface isn't really an improvement (especially when we have to design something extensible that device-specific stuff can be put into).

Quoted 1 line

IMHO, sysfs is getting out of hand for rdma:

I'm not sure how much of a problem this really is...

Quoted 2 lines
$ find /sys/class/infiniband/mlx4_0 -type f | wc -l
660

and presumably 512 of those are gid and pkey table entries?

Quoted 2 lines
$ strace -o /tmp/t /opt/ofa-1.5/sbin/perfquery ; grep sys/ /tmp/t | wc -l
289

That seems a little crazy, but maybe it's an app that's doing silly stuff? If I do ibv_rc_pingpong, the only /sys related things I see are:

open("/sys/class/infiniband_verbs/abi_version", O_RDONLY) = 3 open("/sys/class/infiniband_verbs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 stat("/sys/class/infiniband_verbs/abi_version", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 stat("/sys/class/infiniband_verbs/uverbs0", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 open("/sys/class/infiniband_verbs/uverbs0/ibdev", O_RDONLY) = 4 open("/sys/class/infiniband_verbs/uverbs0/abi_version", O_RDONLY) = 4 open("/sys/class/infiniband_verbs/uverbs0/device/vendor", O_RDONLY) = 3 open("/sys/class/infiniband_verbs/uverbs0/device/device", O_RDONLY) = 3 open("/sys/class/infiniband/mlx4_0/node_type", O_RDONLY) = 3

which is reasonable I think.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaljictxqd.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#25 lore raw
Quoted 9 lines · Roland Dreier
Quoted 3 lines

Maybe give some thought to using a syscall interface through uverbs for some of this?

Actually I think for exposing SL-to-VL and other things like that, sysfs is pretty good. Having something usable from both scripts and programs seems pretty useful, and having an opaque uverbs interface isn't really an improvement (especially when we have to design something extensible that device-specific stuff can be put into).

I guess it depends on the purpose, a noticable problem with sysfs is that there is no good way to be notified when the data changes. PKey, SL2VL, GID tables, sm_lid etc are all SM dynamic information and many cases that are using them should probably have code to know when the SM changes them and make appropriate adjustments.

For instance a long running SMP using program has no way to be notified when the sm_lid changes, or the GID table changes - but it can pick up an IB async event for the pkey table changes.. What should new things do?

It also means we can never have something like ifrename for IB - too racey with sysfs.

Quoted 3 lines
Quoted 1 line

IMHO, sysfs is getting out of hand for rdma:

I'm not sure how much of a problem this really is...

Neither am I.. But I've seen the various eternal lkml arguments about sysfs, netlink, syscall, etc and it does seem like the preferred option is a little bit of all them. It does seem worth asking from time to time if the rdma stuff in sysfs is appropriate.

Quoted 4 lines
Quoted 2 lines
$ find /sys/class/infiniband/mlx4_0 -type f | wc -l
660

and presumably 512 of those are gid and pkey table entries?

Probably. TBH, those are the ones I find most un-sysfs-like..

Quoted 5 lines
Quoted 2 lines
$ strace -o /tmp/t /opt/ofa-1.5/sbin/perfquery ; grep sys/ /tmp/t | wc -l
289

That seems a little crazy, but maybe it's an app that's doing silly stuff? If I do ibv_rc_pingpong, the only /sys related things I see are:

It is reading the pkey and gid tables for some reason. There is no other way to get that data except by trundling through sysfs.. Which I guess really is my point - it isn't so much that the stuff is in sysfs that is strange, but that it is *only* in sysfs.

Quoted 11 lines

open("/sys/class/infiniband_verbs/abi_version", O_RDONLY) = 3 open("/sys/class/infiniband_verbs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 stat("/sys/class/infiniband_verbs/abi_version", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 stat("/sys/class/infiniband_verbs/uverbs0", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 open("/sys/class/infiniband_verbs/uverbs0/ibdev", O_RDONLY) = 4 open("/sys/class/infiniband_verbs/uverbs0/abi_version", O_RDONLY) = 4 open("/sys/class/infiniband_verbs/uverbs0/device/vendor", O_RDONLY) = 3 open("/sys/class/infiniband_verbs/uverbs0/device/device", O_RDONLY) = 3 open("/sys/class/infiniband/mlx4_0/node_type", O_RDONLY) = 3

which is reasonable I think.

Yes, I also think that is pretty much fine.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html