Support RAW Ethernet when RoCE is disabled

Quoted 2 lines

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device,

port_num)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+	return device->port_immutable[port_num].core_cap_flags &

RDMA_CORE_CAP_PROT_RAW_PACKET;

Quoted 1 line

+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

I keep talking about how we need to get rid of the port_num in these sorts of places because it makes no sense...

Quoted 4 lines · Leon Romanovsky

Quoted 2 lines

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device,

u8

Quoted 9 lines

port_num)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+	return device->port_immutable[port_num].core_cap_flags &

RDMA_CORE_CAP_PROT_RAW_PACKET;

Quoted 1 line

+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

Quoted 3 lines

I keep talking about how we need to get rid of the port_num in these sorts of places because it makes no sense...

I agree. Requiring the port number has implications that ripple up into the
rdma-rw api as well...

Quoted 14 lines · Steve Wise

Quoted 4 lines · Leon Romanovsky

Quoted 2 lines

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device,

u8

Quoted 9 lines

port_num)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+	return device->port_immutable[port_num].core_cap_flags &

RDMA_CORE_CAP_PROT_RAW_PACKET;

Quoted 1 line

+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

Quoted 8 lines

Quoted 3 lines

I keep talking about how we need to get rid of the port_num in these sorts of places because it makes no sense...

I agree. Requiring the port number has implications that ripple up into the
rdma-rw api as well...

In all fairness, there is no requirement that any two ports on the same device be the same link layer, or if the link layer is Ethernet, there is no requirement that they can't support both iWARP and RoCE. The idea that the parent device defined the supported protocols for all ports of a device became wrong with the first mlx4 device that could do both IB and Ethernet. And I think I've heard rumblings of a combined RoCE/iWARP device possibly in the future from someone else.

Quoted 26 lines · Doug Ledford

Quoted 15 lines · Steve Wise

Quoted 5 lines · Leon Romanovsky

Quoted 3 lines

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device,

u8

Quoted 9 lines

port_num)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+    return device->port_immutable[port_num].core_cap_flags &

RDMA_CORE_CAP_PROT_RAW_PACKET;

Quoted 1 line

+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

Quoted 9 lines

Quoted 3 lines

I keep talking about how we need to get rid of the port_num in these sorts of places because it makes no sense...

I agree. Requiring the port number has implications that ripple up
into the
rdma-rw api as well...

Quoted 7 lines

In all fairness, there is no requirement that any two ports on the same device be the same link layer, or if the link layer is Ethernet, there is no requirement that they can't support both iWARP and RoCE. The idea that the parent device defined the supported protocols for all ports of a device became wrong with the first mlx4 device that could do both IB and Ethernet. And I think I've heard rumblings of a combined RoCE/iWARP device possibly in the future from someone else.

This one for instance?

http://www.qlogic.com/Resources/Documents/DataSheets/Adapters/DataSheet_QLE45211HL_QLE45212HL.pdf

I'd love to see any such device support protocol choice per connection, not just per port. That of course would have implications on the rdma commection manager api.

From: [email protected] [mailto:linux-rdma- [email protected]] On Behalf Of Tom Talpey

Quoted 8 lines

In all fairness, there is no requirement that any two ports on the same device be the same link layer, or if the link layer is Ethernet, there is no requirement that they can't support both iWARP and RoCE. The idea that the parent device defined the supported protocols for all ports of a device became wrong with the first mlx4 device that could do both IB and Ethernet. And I think I've heard rumblings of a combined RoCE/iWARP device possibly in the future from someone else.

This one for instance?

http://www.qlogic.com/Resources/Documents/DataSheets/Adapters/DataShee
t_QLE45211HL_QLE45212HL.pdf

I'd love to see any such device support protocol choice per connection, not just per port. That of course would have implications on the rdma commection manager api.

Quoted 3 lines

I'd love to see any such device support protocol choice per connection, not just per port. That of course would have implications on the rdma commection manager api.

Exactly. If/when such devices appear, we would need to extend connection management to specify the protocol, rather than infer it from the port space.

Rethinking about the uAPI, maybe we should report a protocol bit-mask similar to the kernel's, instead of QP types? This would provide all the required information (e.g., any combination of RoCEv1/v2, iWARP, and Raw Ethernet for Ethernet links) for today's use-cases as well as tomorrow's combined RoCE/iWARP devices.

Maybe we should dump this uapi stuff until Matan's patches are done. The introspection possible with Matan's work is flexible enough to cope with more cases..

From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]

Quoted 3 lines

Exactly. If/when such devices appear, we would need to extend connection management to specify the protocol, rather than infer it from the port space.

We support that perfectly today as long as the port creates two 'struct ib_devices'. Anything else will require some kind of changes to libibverb's API to specify the AH style.

Quoted 5 lines

Rethinking about the uAPI, maybe we should report a protocol bit-mask similar to the kernel's, instead of QP types? This would provide all the required information (e.g., any combination of RoCEv1/v2, iWARP, and Raw Ethernet for Ethernet links) for today's use-cases as well as tomorrow's combined RoCE/iWARP devices.

Maybe we should dump this uapi stuff until Matan's patches are done. The introspection possible with Matan's work is flexible enough to cope with more cases..

Jason

Quoted 7 lines

From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]

Quoted 3 lines

Exactly. If/when such devices appear, we would need to extend connection management to specify the protocol, rather than infer it from the port space.

Quoted 4 lines

We support that perfectly today as long as the port creates two 'struct ib_devices'. Anything else will require some kind of changes to libibverb's API to specify the AH style.

rdmacm would still have to choose between these ib_devices somehow

- this doesn't change anything. Also, AHs are already port-specific. So, I don't see any issue in this regard.

In any case, we have millions of multi-port devices that can use different link types deployed. This is the specification, and more such devices could appear in the future. We cannot change the device model.

No doubt that the new ABI would be a lot more flexible and self-describing. But it would take a while until we port everything to use it. So, generally, I don't see any problem using the current extensibility capabilities to support useful semantics.

Quoted 2 lines

- this doesn't change anything. Also, AHs are already port-specific. So, I don't see any issue in this regard.

The current scheme infers the protocol of the AH from the current configuration of the port, which is a crazy API when the port's protocol can change on the fly.

Quoted 7 lines

Quoted 3 lines

- this doesn't change anything. Also, AHs are already port-specific. So, I don't see any issue in this regard.

The current scheme infers the protocol of the AH from the current configuration of the port, which is a crazy API when the port's protocol can change on the fly.

Maybe the solution is to make the protocol selection explicit throughout the

and associate it with a QP, rather than attempting to list all transport

a port can support.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

- this doesn't change anything. Also, AHs are already port-specific. So, I don't see any issue in this regard.

The current scheme infers the protocol of the AH from the current configuration of the port, which is a crazy API when the port's protocol can change on the fly.

Maybe the solution is to make the protocol selection explicit

throughout the
APIs

Quoted 1 line

and associate it with a QP, rather than attempting to list all

transport
protocols that

Quoted 1 line

a port can support.

Do you mean requiring the application to pick the protocol?

Quoted 10 lines

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

- this doesn't change anything. Also, AHs are already port-specific. So, I don't see any issue in this regard.

The current scheme infers the protocol of the AH from the current configuration of the port, which is a crazy API when the port's protocol can change on the fly.

Maybe the solution is to make the protocol selection explicit

throughout the
APIs

Quoted 6 lines

Quoted 1 line

and associate it with a QP, rather than attempting to list all

transport
protocols that

Quoted 1 line

a port can support.

Do you mean requiring the application to pick the protocol?

Yes - it seems necessary to support devices with RoCE and iWarp running on the same port.

I'd agree that the application picks the protocol, but I'd also argue that the API should have a "wildcard" mode which allows the current behavior. It could be a simple extension to just add a selector parameter.

The tricky part is what to do with passive endpoints. If a wildcard is specified, should a multiprotocol adapter create listens on all protocols? Also, what of protocols which auto-negotiate? It's my understanding that some RoCE adapters will attempt to autodetect whether their peer is RoCEv1 or RoCEv2 capable, and adjust their protocol to suit.

From: [email protected] [mailto:linux-rdma- [email protected]] On Behalf Of Hefty, Sean

Quoted 4 lines

I'd agree that the application picks the protocol, but I'd also argue that the API should have a "wildcard" mode which allows the current behavior. It could be a simple extension to just add a selector parameter.

I agree. The QP type can be viewed as the socket type, we only need to add a protocol field to go along with it. I.e. the protocol should be a qp attribute provided on input to qp create.

Quoted 6 lines

The tricky part is what to do with passive endpoints. If a wildcard is specified, should a multiprotocol adapter create listens on all protocols? Also, what of protocols which auto-negotiate? It's my understanding that some RoCE adapters will attempt to autodetect whether their peer is RoCEv1 or RoCEv2 capable, and adjust their protocol to suit.

We could include the protocol selection, also with wildcard support, here as well. The rdma cm may be able to handle this, so that drivers won't need to deal with a wildcard endpoint. Assuming that we don't end up with some devices that require they implement wildcard support, while others can't...

Anyway, returning to the initial matter at hand: I would like to start with each port reporting a bit mask of the supported protocols on that link (RoCE v1/v2, Raw Ethernet, iWARP, etc.) It will be used for reporting device capabilities in general for tools, as well as by applications that don't use rdmacm.

From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]

Quoted 5 lines

Anyway, returning to the initial matter at hand: I would like to start with each port reporting a bit mask of the supported protocols on that link (RoCE v1/v2, Raw Ethernet, iWARP, etc.) It will be used for reporting device capabilities in general for tools, as well as by applications that don't use rdmacm.

Why don't we start by defining how it is supposed to even work and how to fix the RDMA CM before adding even more random capability bits?

Jason

Quoted 7 lines

From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]

Quoted 5 lines

Anyway, returning to the initial matter at hand: I would like to start with each port reporting a bit mask of the supported protocols on that link (RoCE v1/v2, Raw Ethernet, iWARP, etc.) It will be used for reporting device capabilities in general for tools, as well as by applications that don't use rdmacm.

Quoted 2 lines

Why don't we start by defining how it is supposed to even work and how to fix the RDMA CM before adding even more random capability bits?

Extensions to RDMA CM is an important topic, which we can continue to discuss as devices that require it are introduced.

Protocol capabilities are needed for information purposes as well as for applications that do not use RDMA CM. They are not "random", but depict what a device can do.

Do you want the change along what was proposed by Liran (allow user-space to query the supported protocols per port) or you are okay

I've already proposed disallowing multiprotocol struct ib_devices.

Quoted 1 line

I've already proposed disallowing multiprotocol struct ib_devices.

My preference is to discontinue attempts at associating a protocol with the device. A device could implement a dozen protocols in software. Transports belong to QPs or cm ids, not devices. Each rdma_cm_id should be associated with a specific cm/transport directly, rather than indirectly selecting one based on the bound <device, port>.

If an app wants a specific transport type for a QP, why doesn't it just try to open one and see if the call fails?

Quoted 5 lines · Sean

Quoted 1 line

I've already proposed disallowing multiprotocol struct ib_devices.

My preference is to discontinue attempts at associating a protocol with the device. A device could implement a dozen protocols in software. Transports belong to QPs or cm ids, not devices. Each rdma_cm_id should be associated with a specific cm/transport directly, rather than indirectly selecting one based on the bound <device, port>.

If an app wants a specific transport type for a QP, why doesn't it just try to open one and see if the call fails?

I tend to agree with Sean on this. And to answer Or's question from a couple emails back, I'm inclined to just take the last three patches for now while we work out a better idea of how everything here should be done on the first seven patches.

From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Thursday, December 08, 2016 12:41 AM

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

My preference is to discontinue attempts at associating a protocol with the

device. A device could implement a dozen protocols in software. Transports belong to QPs or cm ids, not devices. Each rdma_cm_id should be associated with a specific cm/transport directly, rather than indirectly selecting one based on the bound <device, port>.

Quoted 2 lines

Quoted 2 lines

If an app wants a specific transport type for a QP, why doesn't it just try to

open one and see if the call fails?

Quoted 5 lines

I tend to agree with Sean on this. And to answer Or's question from a couple emails back, I'm inclined to just take the last three patches for now while we work out a better idea of how everything here should be done on the first seven patches.

Well in that case, we should work toward getting rid of 'struct ib_device' - it is the cause of so much of this trouble. A port-focused model like the netstack uses is overall saner..

Quoted 4 lines

Well in that case, we should work toward getting rid of 'struct ib_device' - it is the cause of so much of this trouble. A port-focused model like the netstack uses is overall saner..

We are discussing 2 separate issues:
(1) Whether a device can support multiple protocols
(2) Multi-port devices, optionally with different link types.

I think that we all agree that (1) will be needed for future devices,

and that different QPs might be running different protocols. This means that supported protocols need to be device capabilities,

have today in the kernel. There are several devices that implement (2), which follows directly from the specification. The model for these devices is not going to change.

kernel). Doing so at the device level seems to be more aligned with how most future devices would be implemented. However, I won't rule out multi-port devices so soon. When the transport is offloaded to HW, it makes sense to conduct HA between ports within the domain of a single device.

In any case, this has nothing to do with 'struct ib_device', which represents a device that handles packet processing internally and exposes transport endpoints to applications.

Quoted 6 lines

Quoted 2 lines

- this doesn't change anything. Also, AHs are already port-specific. So, I don't see any issue in this regard.

The current scheme infers the protocol of the AH from the current configuration of the port, which is a crazy API when the port's protocol can change on the fly.

Maybe the solution is to make the protocol selection explicit throughout the APIs and associate it with a QP, rather than attempting to list all transport protocols that a port can support.

Quoted 3 lines

Maybe the solution is to make the protocol selection explicit throughout the APIs and associate it with a QP, rather than attempting to list all transport protocols that a port can support.

AH's are linked to a PD, not a QP..

If we had to do it again, a PD centric approach would be more sensible:

// Create a PD on 'port' using ah format 'protocol' pd = ibv_pd_create(port, enum ah_type protocol);

// Enable APM or resource sharing on the PD across two ports ibv_pd_add_port(pd, alt_port);

And get rid of the multi-port ib_device concept entirely.

Quoted 5 lines

Quoted 3 lines

Maybe the solution is to make the protocol selection explicit throughout the APIs and associate it with a QP, rather than attempting to list all transport protocols that a port can support.

AH's are linked to a PD, not a QP..

But protocols are associated with QPs. I'm not sure *why* an AH is linked to a PD. Conceptually the association seems unnecessary.

Quoted 2 lines

But protocols are associated with QPs. I'm not sure *why* an AH is linked to a PD. Conceptually the association seems unnecessary.

The AH has to be linked to the PD because the PD specifies the hardware target and the AH is a hardware object. All objects must be traced back to a PD. It is linked to a PD not a QP because the AH can be shared across all QPs, which is useful for UD applications..

The AH is really similar to the ethernet layer in an IP stack, and the QP
is akin the TCP layer.

Quoted 11 lines

Quoted 2 lines

But protocols are associated with QPs. I'm not sure *why* an AH is linked to a PD. Conceptually the association seems unnecessary.

The AH has to be linked to the PD because the PD specifies the hardware target and the AH is a hardware object. All objects must be traced back to a PD. It is linked to a PD not a QP because the AH can be shared across all QPs, which is useful for UD applications..

The AH is really similar to the ethernet layer in an IP stack, and the QP
is akin the TCP layer.

I always viewed the AH as specifying the route to the destination port. Locally, I still don't see why it couldn't work across the device, like the CQ does. Associating it with a PD just seems to limit which QPs, and by association the memory buffers, it can be used with. So, I'm still not seeing the reason for the linkage...

In all fairness, there is no requirement that any two ports on the same device be the same link layer, or if the link layer is Ethernet, there is no requirement that they can't support both iWARP and RoCE. The idea
that the parent device defined the supported protocols for all ports of a device became wrong with the first mlx4 device that could do both IB and Ethernet. And I think I've heard rumblings of a combined RoCE/iWARP
device possibly in the future from someone else.

Quoted 18 lines · Steve Wise

Quoted 4 lines · Leon Romanovsky

Quoted 2 lines

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device,

u8

Quoted 12 lines

port_num)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+	return device->port_immutable[port_num].core_cap_flags &

RDMA_CORE_CAP_PROT_RAW_PACKET;

Quoted 1 line

+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

I keep talking about how we need to get rid of the port_num in these sorts of places because it makes no sense...

Quoted 4 lines

I agree. Requiring the port number has implications that ripple up into the
rdma-rw api as well...

In all fairness, there is no requirement that any two ports on the same device be the same link layer, or if the link layer is Ethernet, there is no requirement that they can't support both iWARP and RoCE.

The idea that the parent device defined the supported protocols for all ports of a device became wrong with the first mlx4 device that

could do both IB and Ethernet. And I think I've heard rumblings of a combined RoCE/iWARP device possibly in the future from someone else.

Quoted 19 lines · Doug Ledford

Quoted 18 lines · Steve Wise

Quoted 4 lines · Leon Romanovsky

Quoted 2 lines

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device,

u8

Quoted 12 lines

port_num)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+	return device->port_immutable[port_num].core_cap_flags &

RDMA_CORE_CAP_PROT_RAW_PACKET;

Quoted 1 line

+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

I keep talking about how we need to get rid of the port_num in these sorts of places because it makes no sense...

Quoted 8 lines

Quoted 4 lines

I agree. Requiring the port number has implications that ripple up into the
rdma-rw api as well...

In all fairness, there is no requirement that any two ports on the same device be the same link layer, or if the link layer is Ethernet, there is no requirement that they can't support both iWARP and RoCE.

There actually is a requirement. The RDMA CM hard requires all ports be iWARP or !iWARP at least. I'm sure there are other subtle things floating around.

There are also things that become very confusing for user space, and we don't have the infrastructure to support, if ports can switch configurations on the fly.

The simplest, approach, most in line with how verbs was designed, is to require each ib_device to have a single kind of AH.

Quoted 2 lines

The idea that the parent device defined the supported protocols for all ports of a device became wrong with the first mlx4 device that

Arguably it was sort of OK for roceev1, is less OK for v2, but shouldn't have been done anyhow.

The uapi question here is do we want to double down and try and make this work (and what does that even *mean*) or admit mlx4 was an error and stop doing that going forward..

Or do something else? eg Specifying the AH type when creating the PD could potentially solve some of the problems...

Quoted 3 lines

could do both IB and Ethernet. And I think I've heard rumblings of a combined RoCE/iWARP device possibly in the future from someone else.

Two struct ib_devices for the same port then... Certainly ugly.

I'm not entirely sure I agree....I would have to think about it more, but the underlying problem I'm concerned with is exactly what I point out above: what restrictions would we be placing on the hardware that are artificial and a result of our stack that the hardware itself does not share? Could the hardware support automatic migration from iWARP to
RoCE or vice versa if both endpoints supported it? Would that work if we required two separate ib_devices? Things like that.

include/rdma/ib_verbs.h static inline bool rdma_ib_or_roce(const struct ib_device *device, u8 port_num) · line 2295

Quoted 4 lines · Leon Romanovsky

include/rdma/ib_verbs.h +4 -0

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device, u8 port_num)
+{
+     return device->port_immutable[port_num].core_cap_flags & RDMA_CORE_CAP_PROT_RAW_PACKET;
+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

Quoted 9 lines · Jason Gunthorpe

include/rdma/ib_verbs.h static inline bool rdma_ib_or_roce(const struct ib_device *device, u8 port_num) · line 2295

Quoted 4 lines · Leon Romanovsky

include/rdma/ib_verbs.h +4 -0

+static inline bool rdma_protocol_raw_packet(const struct ib_device *device, u8 port_num)
+{
+     return device->port_immutable[port_num].core_cap_flags & RDMA_CORE_CAP_PROT_RAW_PACKET;
+}

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

mlx4 yeah (practically for the last ~10 years) for instance Eth ports don't support SMI -- this goes back to the fact that mlx4 devices are single PCI function with potentially two ports and each port can be

set to different link layer. But this no more holds for mlx5, these devices are function-per-port and hence IB device per port.

Quoted 5 lines · Or Gerlitz

Quoted 2 lines · Jason Gunthorpe

Does the mlx drivers really register ports with different capabilities as the same ib_device? I'm not sure that should be allowed.

Quoted 3 lines

mlx4 yeah (practically for the last ~10 years) for instance Eth ports don't support SMI -- this goes back to the fact that mlx4 devices are single PCI function with potentially two ports and each port can be

struct ib_device is not linked to a PCI device. AFAIK mlx4 created one ib_device for rocee ports and one for IB,

or at least it should or things are already broken.

Quoted 2 lines

set to different link layer. But this no more holds for mlx5, these devices are function-per-port and hence IB device per port.

Since it has nothing to do with pci devices, please do this properly.

Quoted 1 line

or at least it should or things are already broken.

Can you elaborate what it broken with mlx4? I suspect we
even have down there some functionality which depends on that,
but again, I 1st would like to hear if/what is broken - I copied the maintainer.

Jason, patches 8-10 which carry the functional change I want to introduce (allow mlx5 IB devices to be created when RoCE is not supported) stand for themselves.

a re-write of the 10y old IB device-ing of things done by mlx4 just to be able and introduce this reduced functionally (raw packet qp only) of mlx5 devices.

From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add qp_type_cap port attribute which is a bit field representation of the ib_qp_type enum. This will allow applications to query what QP types are supported by an IB device instance on a specific port.

The qp_type_cap port attribute is set by the core according to the protocol supported for the device port. This holds for all the providers with the exception of two RoCE drivers that don't implement UD and UC. To handle that, they (hns and qedr) are patched to remove these QPs from what's the core has set for them as supported.

Signed-off-by Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

Reviewed-by Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/device.c          | 28 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/hns/hns_roce_main.c |  3 +++
 drivers/infiniband/hw/qedr/verbs.c        |  2 ++
 include/rdma/ib_verbs.h                   |  1 +
 4 files changed, 34 insertions(+)

drivers/infiniband/core/device.c +0 -0

index 760ef60..f7abde2 100644
@@ -646,6 +646,31 @@ void ib_dispatch_event(struct ib_event *event)
 }

drivers/infiniband/core/device.c +20 -0

+
+	if (rdma_ib_or_roce(device, port_num)) {
+		port_attr->qp_type_cap |= (BIT(IB_QPT_RC) | BIT(IB_QPT_UD) | BIT(IB_QPT_UC));
+		if (device->attrs.device_cap_flags & IB_DEVICE_XRC)
+			port_attr->qp_type_cap |= (BIT(IB_QPT_XRC_INI) | BIT(IB_QPT_XRC_TGT));
+	}
+
+	if (rdma_protocol_iwarp(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_RC);
+
+	if (rdma_protocol_raw_packet(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_RAW_PACKET);
+
+	if (rdma_protocol_usnic(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_UD);
+}
+
 /**
  * ib_query_port - Query IB port attributes
  * @device:Device to query

@@ -666,6 +691,9 @@ int ib_query_port(struct ib_device *device,
 		return -EINVAL;
 
 	memset(port_attr, 0, sizeof(*port_attr));
+
+	get_port_qp_types(device, port_num, port_attr);
+
 	err = device->query_port(device, port_num, port_attr);
 	if (err || port_attr->subnet_prefix)
 		return err;

drivers/infiniband/hw/hns/hns_roce_main.c +3 -0

index c6b5779..22de534 100644
@@ -430,6 +430,9 @@ static int hns_roce_query_port(struct ib_device *ib_dev, u8 port_num,
 			IB_PORT_ACTIVE : IB_PORT_DOWN;
 	props->phys_state = (props->state == IB_PORT_ACTIVE) ? 5 : 3;
 
+	/* mark that UD and UC aren't supported */
+	props->qp_type_cap &= ~(BIT(IB_QPT_UD) | BIT(IB_QPT_UC));
+
 	spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
 
 	return 0;

drivers/infiniband/hw/qedr/verbs.c +2 -0

index 53207ff..33d0219 100644
@@ -263,6 +263,8 @@ int qedr_query_port(struct ib_device *ibdev, u8 port, struct ib_port_attr *attr)
 	attr->max_msg_sz = rdma_port->max_msg_size;
 	attr->max_vl_num = 4;
 
+	/* mark that UD and UC aren't supported */
+	attr->qp_type_cap &= ~(BIT(IB_QPT_UD) | BIT(IB_QPT_UC));
 	return 0;
 }
 

include/rdma/ib_verbs.h +1 -0

index 485b725..0b839e4 100644
@@ -536,6 +536,7 @@ struct ib_port_attr {
 	u8			active_speed;
 	u8                      phys_state;
 	bool			grh_required;
+	u16			qp_type_cap;
 };
 
 enum ib_device_modify_flags {

Signature · 6 lines

2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 30 lines · Leon Romanovsky

From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add qp_type_cap port attribute which is a bit field representation of the ib_qp_type enum. This will allow applications to query what QP types are supported by an IB device instance on a specific port.

The qp_type_cap port attribute is set by the core according to the protocol supported for the device port. This holds for all the providers with the exception of two RoCE drivers that don't implement UD and UC. To handle that, they (hns and qedr) are patched to remove these QPs from what's the core has set for them as supported.

Signed-off-by Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

Reviewed-by Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/device.c          | 28 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/hns/hns_roce_main.c |  3 +++
 drivers/infiniband/hw/qedr/verbs.c        |  2 ++
 include/rdma/ib_verbs.h                   |  1 +
 4 files changed, 34 insertions(+)

drivers/infiniband/core/device.c +0 -0

index 760ef60..f7abde2 100644
@@ -646,6 +646,31 @@ void ib_dispatch_event(struct ib_event *event)
 }
 EXPORT_SYMBOL(ib_dispatch_event);
 

drivers/infiniband/core/device.c void ib_dispatch_event(struct ib_event *event) · line 656

Quoted 8 lines

drivers/infiniband/core/device.c +8 -0

+static void get_port_qp_types(const struct ib_device *device, u8 port_num,
+			      struct ib_port_attr *port_attr)
+{
+	if (rdma_cap_ib_smi(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_SMI);
+
+	if (rdma_cap_ib_cm(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_GSI);

This is not accurate. The IB CM is not the same as having QP1 supported.

Ira

Quoted 72 lines

drivers/infiniband/core/device.c +20 -0

+
+	if (rdma_ib_or_roce(device, port_num)) {
+		port_attr->qp_type_cap |= (BIT(IB_QPT_RC) | BIT(IB_QPT_UD) | BIT(IB_QPT_UC));
+		if (device->attrs.device_cap_flags & IB_DEVICE_XRC)
+			port_attr->qp_type_cap |= (BIT(IB_QPT_XRC_INI) | BIT(IB_QPT_XRC_TGT));
+	}
+
+	if (rdma_protocol_iwarp(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_RC);
+
+	if (rdma_protocol_raw_packet(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_RAW_PACKET);
+
+	if (rdma_protocol_usnic(device, port_num))
+		port_attr->qp_type_cap |= BIT(IB_QPT_UD);
+}
+
 /**
  * ib_query_port - Query IB port attributes
  * @device:Device to query

@@ -666,6 +691,9 @@ int ib_query_port(struct ib_device *device,
 		return -EINVAL;
 
 	memset(port_attr, 0, sizeof(*port_attr));
+
+	get_port_qp_types(device, port_num, port_attr);
+
 	err = device->query_port(device, port_num, port_attr);
 	if (err || port_attr->subnet_prefix)
 		return err;

drivers/infiniband/hw/hns/hns_roce_main.c +3 -0

index c6b5779..22de534 100644
@@ -430,6 +430,9 @@ static int hns_roce_query_port(struct ib_device *ib_dev, u8 port_num,
 			IB_PORT_ACTIVE : IB_PORT_DOWN;
 	props->phys_state = (props->state == IB_PORT_ACTIVE) ? 5 : 3;
 
+	/* mark that UD and UC aren't supported */
+	props->qp_type_cap &= ~(BIT(IB_QPT_UD) | BIT(IB_QPT_UC));
+
 	spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
 
 	return 0;

drivers/infiniband/hw/qedr/verbs.c +2 -0

index 53207ff..33d0219 100644
@@ -263,6 +263,8 @@ int qedr_query_port(struct ib_device *ibdev, u8 port, struct ib_port_attr *attr)
 	attr->max_msg_sz = rdma_port->max_msg_size;
 	attr->max_vl_num = 4;
 
+	/* mark that UD and UC aren't supported */
+	attr->qp_type_cap &= ~(BIT(IB_QPT_UD) | BIT(IB_QPT_UC));
 	return 0;
 }
 

include/rdma/ib_verbs.h +1 -1

index 485b725..0b839e4 100644
@@ -536,6 +536,7 @@ struct ib_port_attr {
 	u8			active_speed;
 	u8                      phys_state;
 	bool			grh_required;
+	u16			qp_type_cap;
 };
 
 enum ib_device_modify_flags {
--

2.7.4

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 30 lines · Leon Romanovsky

From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add qp_type_cap port attribute which is a bit field representation of the ib_qp_type enum. This will allow applications to query what QP types are supported by an IB device instance on a specific port.

The qp_type_cap port attribute is set by the core according to the protocol supported for the device port. This holds for all the providers with the exception of two RoCE drivers that don't implement UD and UC. To handle that, they (hns and qedr) are patched to remove these QPs from what's the core has set for them as supported.

Signed-off-by Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

Reviewed-by Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/device.c          | 28 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/hns/hns_roce_main.c |  3 +++
 drivers/infiniband/hw/qedr/verbs.c        |  2 ++
 include/rdma/ib_verbs.h                   |  1 +
 4 files changed, 34 insertions(+)

drivers/infiniband/core/device.c +0 -0

index 760ef60..f7abde2 100644
@@ -646,6 +646,31 @@ void ib_dispatch_event(struct ib_event *event)
 }
 EXPORT_SYMBOL(ib_dispatch_event);
 

drivers/infiniband/core/device.c void ib_dispatch_event(struct ib_event *event) · line 656

Quoted 8 lines

drivers/infiniband/core/device.c +8 -0

+static void get_port_qp_types(const struct ib_device *device, u8 port_num,
+                           struct ib_port_attr *port_attr)
+{
+     if (rdma_cap_ib_smi(device, port_num))
+             port_attr->qp_type_cap |= BIT(IB_QPT_SMI);
+
+     if (rdma_cap_ib_cm(device, port_num))
+             port_attr->qp_type_cap |= BIT(IB_QPT_GSI);

This is not accurate. The IB CM is not the same as having QP1 supported.

Hi Doug,

Please find below the patch set from Or. I didn't add version notation to this patch set, because it was enriched extensively over initial version [1], from one mlx5-only patch to native IB core support.

It is tested against v4.9-rc6.

From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

On some environments, such as certain SRIOV VF configurations, RoCE is not supported for mlx5 Ethernet ports. Currently, the driver will not open IB device on that port.

This is problematic, since we do want user-space RAW Ethernet (RAW_PACKET QPs) functionality to remain in place. For that end, we change the relevant driver flows such that an IB device instance is created in that case as well.

Following the previous post [1], Doug wanted us to enable a way for applications to query what QP types are actually supported on a device. This series adds that functionality on device/port granularity, since some drivers (mlx4) could support different protocols per port and hence different QP types per port.

QP types are basically derived from the protocol/s supported on the port. We added two protocols (raw packet and usnic) to have a protocol set which is consistent with what is currently upstream.

Patches 1-7 deal with the new query and patches 8-10 with the mlx5 specific changes.