Skip to contentSkip to conversation

Another round of exported properties for RDMAtool

PATCH In review Leon Romanovsky 2 participants 15 messages Jun 28, 2017 11 messages not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

cover Another round of exported properties for RDMAtool

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>

hi Doug,

This patch set is based on previously sent RDMA netlink work.

!!! Important to note, that LIDs and SM_LIDS are exported as u32, to support OmniPath natively.

Thanks

Leon Romanovsky (4):
RDMA/netlink: Export guid and sys_image_guid
RDMA/netlink: Advertise IB subnet prefix
RDMA/netink: Export lids and sm_lids
RDMA/netlink: Export LID mask counter (LMC)

 drivers/infiniband/core/nldev.c  | 21 ++++++++++++++++++++-
 include/uapi/rdma/rdma_netlink.h | 31 +++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 1 deletion(-)

--
2.13.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628133445.16550-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

#2 lore raw

PATCH 1/4 RDMA/netlink: Export guid and sys_image_guid

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add GUID and system image GUID to the device properties exported by RDMA netlink, to be used by RDMAtool.

---
 drivers/infiniband/core/nldev.c  |  7 +++++++
 include/uapi/rdma/rdma_netlink.h | 13 +++++++++++++
 2 files changed, 20 insertions(+)
drivers/infiniband/core/nldev.c +7 -0
index 888eadced3bd..56500810c2c0 100644
@@ -42,6 +42,8 @@ static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
 	[RDMA_NLDEV_ATTR_PORT_INDEX]	= { .type = NLA_U32 },
 	[RDMA_NLDEV_ATTR_FW_VERSION]	= { .type = NLA_NUL_STRING,
 					    .len = IB_FW_VERSION_NAME_MAX - 1},
+	[RDMA_NLDEV_ATTR_GUID]		= { .type = NLA_U64 },
+	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = { .type = NLA_U64 },
 };
 
 static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
@@ -64,6 +66,11 @@ static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
 	if (nla_put_string(msg, RDMA_NLDEV_ATTR_FW_VERSION, fw))
 		return -EMSGSIZE;
 
+	if (nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_GUID, device->node_guid, 0))
+		return -EMSGSIZE;
+	if (nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_SYS_IMAGE_GUID,
+			      device->attrs.sys_image_guid, 0))
+		return -EMSGSIZE;
 	return 0;
 }
 
include/uapi/rdma/rdma_netlink.h +13 -0
index cfcb2a4224fa..f46af029ace2 100644
@@ -266,6 +266,19 @@ enum rdma_nldev_attr {
 	 * Devices without FW will return empty string terminated by "\0"
 	 */
 	RDMA_NLDEV_ATTR_FW_VERSION,		/* string */
+
+	/*
+	 * GUID (in network byte order) associated with the RDMA device.
+	 */
+	RDMA_NLDEV_ATTR_GUID,			/* u64 */
+
+	/*
+	 * System image GUID (in network byte order) associated with
+	 * this RDMA device and other devices which are part of a
+	 * single system.
+	 */
+	RDMA_NLDEV_ATTR_SYS_IMAGE_GUID,		/* u64 */
+
 	RDMA_NLDEV_ATTR_MAX
 };
 #endif /* _UAPI_RDMA_NETLINK_H */
Signature · 6 lines
2.13.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628133445.16550-2-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

#4 lore raw

Re: [PATCH rdma-next 1/4] RDMA/netlink: Export guid and sys_image_guid

drivers/infiniband/core/nldev.c static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = { · line 45

Quoted 1 line · Leon Romanovsky
drivers/infiniband/core/nldev.c +1 -0
+	[RDMA_NLDEV_ATTR_GUID]		= { .type = NLA_U64 },

This should identify which guid it is. Port? Node? Device? Something else?

If it is the port guid should it be able to carry the entire table, not just the 0th entry?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628160025.GA14375-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#6 lore raw

Re: [PATCH rdma-next 1/4] RDMA/netlink: Export guid and sys_image_guid

Quoted 5 lines · Jason Gunthorpe

drivers/infiniband/core/nldev.c static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = { · line 45

Quoted 1 line · Leon Romanovsky
drivers/infiniband/core/nldev.c +1 -0
+	[RDMA_NLDEV_ATTR_GUID]		= { .type = NLA_U64 },

This should identify which guid it is. Port? Node? Device? Something else?

It is my mistake, I was under impression of sysfs. It is node_guid.

Quoted 5 lines

If it is the port guid should it be able to carry the entire table, not just the 0th entry?

Jason

#7 lore raw

PATCH 2/4 RDMA/netlink: Advertise IB subnet prefix

In reply to a message not in the archive

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add IB subnet prefix to the port properties exported by RDMA netlink.

---
 drivers/infiniband/core/nldev.c  | 4 ++++
 include/uapi/rdma/rdma_netlink.h | 5 +++++
 2 files changed, 9 insertions(+)
drivers/infiniband/core/nldev.c +4 -0
index 56500810c2c0..755b8167280b 100644
@@ -44,6 +44,7 @@ static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
 					    .len = IB_FW_VERSION_NAME_MAX - 1},
 	[RDMA_NLDEV_ATTR_GUID]		= { .type = NLA_U64 },
 	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = { .type = NLA_U64 },
+	[RDMA_NLDEV_ATTR_SUBNET_PREFIX]	= { .type = NLA_U64 },
 };
 
 static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
@@ -95,6 +96,9 @@ static int fill_port_info(struct sk_buff *msg,
 	if (nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_CAP_FLAGS,
 			      (u64)attr.port_cap_flags, 0))
 		return -EMSGSIZE;
+	if (nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_SUBNET_PREFIX,
+			      attr.subnet_prefix, 0))
+		return -EMSGSIZE;
 
 	return 0;
 }
include/uapi/rdma/rdma_netlink.h +5 -0
index f46af029ace2..5fec88038e6e 100644
@@ -279,6 +279,11 @@ enum rdma_nldev_attr {
 	 */
 	RDMA_NLDEV_ATTR_SYS_IMAGE_GUID,		/* u64 */
 
+	/*
+	 * Subnet prefix (in host byte order)
+	 */
+	RDMA_NLDEV_ATTR_SUBNET_PREFIX,		/* u64 */
+
 	RDMA_NLDEV_ATTR_MAX
 };
 #endif /* _UAPI_RDMA_NETLINK_H */
Signature · 6 lines
2.13.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628133445.16550-3-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

#9 lore raw

Re: [PATCH rdma-next 2/4] RDMA/netlink: Advertise IB subnet prefix

include/uapi/rdma/rdma_netlink.h enum rdma_nldev_attr { · line 285

Quoted 4 lines · Leon Romanovsky
include/uapi/rdma/rdma_netlink.h +4 -0
+	/*
+	 * Subnet prefix (in host byte order)
+	 */
+	RDMA_NLDEV_ATTR_SUBNET_PREFIX,		/* u64 */

I've always hated how random the be / host choices are in our API.

Since the GUID is in host order I recommend making the subnet_prefix in host order too.

At the very least, the comments beside anything not in host order should be 'be64' not u64

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628160251.GB14375-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#11 lore raw

Re: [PATCH rdma-next 2/4] RDMA/netlink: Advertise IB subnet prefix

Quoted 8 lines · Jason Gunthorpe

include/uapi/rdma/rdma_netlink.h enum rdma_nldev_attr { · line 285

Quoted 4 lines · Leon Romanovsky
include/uapi/rdma/rdma_netlink.h +4 -0
+	/*
+	 * Subnet prefix (in host byte order)
+	 */
+	RDMA_NLDEV_ATTR_SUBNET_PREFIX,		/* u64 */

I've always hated how random the be / host choices are in our API.

Agree

Quoted 6 lines

Since the GUID is in host order I recommend making the subnet_prefix in host order too.

At the very least, the comments beside anything not in host order should be 'be64' not u64

Those comments represent netlink types. It doesn't have be64.

Quoted 2 lines

Jason

A message that is not in the archive (sent off-list or not yet received): 20170628161020.GI1248-U/[email protected]

#13 lore raw

Re: [PATCH rdma-next 2/4] RDMA/netlink: Advertise IB subnet prefix

Quoted 7 lines · Leon Romanovsky
Quoted 5 lines

Since the GUID is in host order I recommend making the subnet_prefix in host order too.

At the very least, the comments beside anything not in host order should be 'be64' not u64

Those comments represent netlink types. It doesn't have be64.

Then don't send b64 down netlink as a netlink u64 type?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628161236.GC28839-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#15 lore raw

Re: [PATCH rdma-next 2/4] RDMA/netlink: Advertise IB subnet prefix

Quoted 10 lines · Jason Gunthorpe
Quoted 7 lines · Leon Romanovsky
Quoted 5 lines

Since the GUID is in host order I recommend making the subnet_prefix in host order too.

At the very least, the comments beside anything not in host order should be 'be64' not u64

Those comments represent netlink types. It doesn't have be64.

Then don't send b64 down netlink as a netlink u64 type?

Agree, conversion to u64 is the right way to do.

Quoted 2 lines

Jason

#16 lore raw

PATCH 3/4 RDMA/netink: Export lids and sm_lids

In reply to a message not in the archive

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>

According to the IB specification, the LID and SM_LID are 16-bit wide, but to support OmniPath users, export it as 32-bit value from the beginning.

---
 drivers/infiniband/core/nldev.c  | 7 ++++++-
 include/uapi/rdma/rdma_netlink.h | 8 ++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
drivers/infiniband/core/nldev.c +6 -1
index 755b8167280b..6b290dada589 100644
@@ -45,6 +45,8 @@ static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
 	[RDMA_NLDEV_ATTR_GUID]		= { .type = NLA_U64 },
 	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = { .type = NLA_U64 },
 	[RDMA_NLDEV_ATTR_SUBNET_PREFIX]	= { .type = NLA_U64 },
+	[RDMA_NLDEV_ATTR_LID]		= { .type = NLA_U32 },
+	[RDMA_NLDEV_ATTR_SM_LID]	= { .type = NLA_U32 },
 };
 
 static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
@@ -99,7 +101,10 @@ static int fill_port_info(struct sk_buff *msg,
 	if (nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_SUBNET_PREFIX,
 			      attr.subnet_prefix, 0))
 		return -EMSGSIZE;
-
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_LID, attr.lid))
+		return -EMSGSIZE;
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_SM_LID, attr.sm_lid))
+		return -EMSGSIZE;
 	return 0;
 }
 
include/uapi/rdma/rdma_netlink.h +8 -0
index 5fec88038e6e..22e4870f2e31 100644
@@ -284,6 +284,14 @@ enum rdma_nldev_attr {
 	 */
 	RDMA_NLDEV_ATTR_SUBNET_PREFIX,		/* u64 */
 
+	/*
+	 * Local Identifier (LID),
+	 * According to IB specification, It is 16-bit address assigned
+	 * by the Subnet Manager. Extended to be 32-bit for OmniPath users.
+	 */
+	RDMA_NLDEV_ATTR_LID,			/* u32 */
+	RDMA_NLDEV_ATTR_SM_LID,			/* u32 */
+
 	RDMA_NLDEV_ATTR_MAX
 };
 #endif /* _UAPI_RDMA_NETLINK_H */

--
2.13.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628133445.16550-4-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

#18 lore raw

Re: [PATCH rdma-next 3/4] RDMA/netink: Export lids and sm_lids

drivers/infiniband/core/nldev.c static int fill_port_info(struct sk_buff *msg, · line 107

Quoted 4 lines · Leon Romanovsky
drivers/infiniband/core/nldev.c +4 -0
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_LID, attr.lid))
+		return -EMSGSIZE;
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_SM_LID, attr.sm_lid))
+		return -EMSGSIZE;

Is this careful to only stuff attributes if they actuall exists? Eg these should not be present for iwarp/roce/etc

Same comment for all attributes. Only present if they are valid.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628160513.GC14375-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#20 lore raw

Re: [PATCH rdma-next 3/4] RDMA/netink: Export lids and sm_lids

Quoted 10 lines · Jason Gunthorpe

drivers/infiniband/core/nldev.c static int fill_port_info(struct sk_buff *msg, · line 107

Quoted 4 lines · Leon Romanovsky
drivers/infiniband/core/nldev.c +4 -0
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_LID, attr.lid))
+		return -EMSGSIZE;
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_SM_LID, attr.sm_lid))
+		return -EMSGSIZE;

Is this careful to only stuff attributes if they actuall exists? Eg these should not be present for iwarp/roce/etc

Same comment for all attributes. Only present if they are valid.

It will be zero if it is not exist, but you are right it is better to avoid to fill unneeded fields.

Quoted 2 lines

Jason

A message that is not in the archive (sent off-list or not yet received): 20170628161616.GK1248-U/[email protected]

#22 lore raw

Re: [PATCH rdma-next 3/4] RDMA/netink: Export lids and sm_lids

Quoted 10 lines · Leon Romanovsky
Quoted 9 lines · Jason Gunthorpe

drivers/infiniband/core/nldev.c static int fill_port_info(struct sk_buff *msg, · line 107

Quoted 4 lines · Leon Romanovsky
drivers/infiniband/core/nldev.c +4 -0
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_LID, attr.lid))
+		return -EMSGSIZE;
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_SM_LID, attr.sm_lid))
+		return -EMSGSIZE;

Is this careful to only stuff attributes if they actuall exists? Eg these should not be present for iwarp/roce/etc

Quoted 4 lines
Quoted 1 line

Same comment for all attributes. Only present if they are valid.

It will be zero if it is not exist, but you are right it is better to avoid to fill unneeded fields.

Not existing is the netlink way. Same comment for everything attributes. Many of the guids do not apply to iwarp, IIRC..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170628161903.GA28996-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#24 lore raw

Re: [PATCH rdma-next 3/4] RDMA/netink: Export lids and sm_lids

Quoted 11 lines · Jason Gunthorpe
Quoted 10 lines · Leon Romanovsky
Quoted 9 lines · Jason Gunthorpe

Review

Quoted 4 lines · Leon Romanovsky
(quoted diff) +4 -0
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_LID, attr.lid))
+		return -EMSGSIZE;
+	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_SM_LID, attr.sm_lid))
+		return -EMSGSIZE;

Is this careful to only stuff attributes if they actuall exists? Eg these should not be present for iwarp/roce/etc

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Same comment for all attributes. Only present if they are valid.

It will be zero if it is not exist, but you are right it is better to avoid to fill unneeded fields.

Not existing is the netlink way. Same comment for everything attributes. Many of the guids do not apply to iwarp, IIRC..

I got it, this is why I prefer to have basic RDMAtool with minimal prints (device name + caps) be accepted and only after that start work on every field in serial manner.

Thanks

Quoted 2 lines

Jason

#25 lore raw

PATCH 4/4 RDMA/netlink: Export LID mask counter (LMC)

In reply to a message not in the archive

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>

---
 drivers/infiniband/core/nldev.c  | 3 +++
 include/uapi/rdma/rdma_netlink.h | 5 +++++
 2 files changed, 8 insertions(+)
drivers/infiniband/core/nldev.c +3 -0
index 6b290dada589..8f037a2be508 100644
@@ -47,6 +47,7 @@ static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
 	[RDMA_NLDEV_ATTR_SUBNET_PREFIX]	= { .type = NLA_U64 },
 	[RDMA_NLDEV_ATTR_LID]		= { .type = NLA_U32 },
 	[RDMA_NLDEV_ATTR_SM_LID]	= { .type = NLA_U32 },
+	[RDMA_NLDEV_ATTR_LMC]		= { .type = NLA_U8 },
 };
 
 static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
@@ -105,6 +106,8 @@ static int fill_port_info(struct sk_buff *msg,
 		return -EMSGSIZE;
 	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_SM_LID, attr.sm_lid))
 		return -EMSGSIZE;
+	if (nla_put_u8(msg, RDMA_NLDEV_ATTR_LMC, attr.lmc))
+		return -EMSGSIZE;
 	return 0;
 }
 
include/uapi/rdma/rdma_netlink.h +5 -0
index 22e4870f2e31..9e9357e63b65 100644
@@ -292,6 +292,11 @@ enum rdma_nldev_attr {
 	RDMA_NLDEV_ATTR_LID,			/* u32 */
 	RDMA_NLDEV_ATTR_SM_LID,			/* u32 */
 
+	/*
+	 * LID mask count (LMC)
+	 */
+	RDMA_NLDEV_ATTR_LMC,			/* u8 */
+
 	RDMA_NLDEV_ATTR_MAX
 };
 #endif /* _UAPI_RDMA_NETLINK_H */
Signature · 6 lines
2.13.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html