RDMAtool
Subsystems (4)
cover RDMAtool
From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Hi,
This series is implementing the RDAMtool - the tool to configure RDMA devices. The initial proposal was sent as RFC [1] and was based on sysfs entries as POC.
The current series was rewritten completely to work with RDMA netlinks as a source of user<->kernel communications. In order to achieve that, the RDMA netlinks were extensively refactored and modernized [2, 3 and 4].
The changes from RFC:
1. Removed everything that is not implemented yet.
2. Abandoned sysfs interfaces in favor of netlink.
Available in the "topic/rdmatool-netlink" topic branch of this git repo:git://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git
Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/iproute2.git/log/?h=topic/rdmatool-netlink
Thanks
[1] https://www.spinics.net/lists/linux-rdma/msg49575.html
[2] https://patchwork.kernel.org/patch/9752865/
[3] https://www.spinics.net/lists/linux-rdma/msg50827.html
[4] https://www.spinics.net/lists/linux-rdma/msg51210.html
Leon Romanovsky (5):
rdma: Add basic infrastructure for RDMA tool
rdma: Add dev object
rdma: Add device capability parsing
rdma: Add link option and parsing
rdma: Add initial manual for the tool
Makefile | 2 +- man/man8/Makefile | 3 +- man/man8/rdma.8 | 82 +++++++++++++++ rdma/.gitignore | 1 + rdma/Makefile | 22 ++++ rdma/dev.c | 140 +++++++++++++++++++++++++ rdma/link.c | 201 +++++++++++++++++++++++++++++++++++ rdma/rdma.c | 112 ++++++++++++++++++++ rdma/rdma.h | 89 ++++++++++++++++ rdma/utils.c | 307 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 10 files changed, 957 insertions(+), 2 deletions(-) create mode 100644 man/man8/rdma.8 create mode 100644 rdma/.gitignore create mode 100644 rdma/Makefile create mode 100644 rdma/dev.c create mode 100644 rdma/link.c create mode 100644 rdma/rdma.c create mode 100644 rdma/rdma.h create mode 100644 rdma/utils.c
--
2.13.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170626182128.24964-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
PATCH 1/5 rdma: Add basic infrastructure for RDMA tool
From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
RDMA devices are cross-functional devices from one side, but very tailored for the specific markets from another.
Such diversity caused to spread of RDMA related configuration across various tools, e.g. devlink, ip, ethtool, ib specific and vendor specific solutions.
This patch adds ability to fill device and port information by reading RDMA netlink.
--- Makefile | 2 +- man/man8/Makefile | 3 +- rdma/.gitignore | 1 + rdma/Makefile | 22 +++++ rdma/rdma.c | 110 ++++++++++++++++++++++ rdma/rdma.h | 76 +++++++++++++++ rdma/utils.c | 270 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 482 insertions(+), 2 deletions(-) create mode 100644 rdma/.gitignore create mode 100644 rdma/Makefile create mode 100644 rdma/rdma.c create mode 100644 rdma/rdma.h create mode 100644 rdma/utils.c
Makefile +1 -1
@@ -52,7 +52,7 @@ WFLAGS += -Wmissing-declarations -Wold-style-definition -Wformat=2 CFLAGS := $(WFLAGS) $(CCOPTS) -I../include $(DEFINES) $(CFLAGS) YACCFLAGS = -d -t -v -SUBDIRS=lib ip tc bridge misc netem genl tipc devlink man +SUBDIRS=lib ip tc bridge misc netem genl tipc devlink rdma man LIBNETLINK=../lib/libnetlink.a ../lib/libutil.a LDLIBS += $(LIBNETLINK)
man/man8/Makefile +2 -1
@@ -19,7 +19,8 @@ MAN8PAGES = $(TARGETS) ip.8 arpd.8 lnstat.8 routel.8 rtacct.8 rtmon.8 rtpr.8 ss. tc-simple.8 tc-skbedit.8 tc-vlan.8 tc-xt.8 tc-skbmod.8 tc-ife.8 \ tc-tunnel_key.8 tc-sample.8 \ devlink.8 devlink-dev.8 devlink-monitor.8 devlink-port.8 devlink-sb.8 \ - ifstat.8 + ifstat.8 \ + rdma.8 all: $(TARGETS)
rdma/.gitignore new file +1 -0
@@ -0,0 +1 @@ +rdma
rdma/Makefile new file +22 -0
@@ -0,0 +1,22 @@ +include ../Config + +ifeq ($(HAVE_MNL),y) + +RDMA_OBJ = rdma.o utils.o + +TARGETS=rdma +CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags) +LDLIBS += $(shell $(PKG_CONFIG) libmnl --libs) + +endif + +all: $(TARGETS) $(LIBS) + +rdma: $(RDMA_OBJ) $(LIBS) + $(QUIET_LINK)$(CC) $^ $(LDFLAGS) $(LDLIBS) -o $@ + +install: all + install -m 0755 $(TARGETS) $(DESTDIR)$(SBINDIR) + +clean: + rm -f $(RDMA_OBJ) $(TARGETS)
rdma/rdma.c new file +110 -0
110 lines not on this page · raw message
rdma/rdma.h new file +76 -0
76 lines not on this page · raw message
rdma/utils.c new file +270 -0
270 lines not on this page · raw message
--
2.13.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 3/5 rdma: Add device capability parsing
In reply to a message not in the archiveFrom: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add parsing interface for the device capability flags
$ rdma dev show 1: mlx5_0: caps 0x1257e1c26 2: mlx5_1: caps 0x1257e1c26 3: mlx5_2: caps 0x1257e1c26 4: mlx5_3: caps 0x1257e1c26 5: mlx5_4: caps 0x1257e1c26
$ rdma dev show mlx5_4 5: mlx5_4: caps 0x1257e1c26
$ rdma dev show mlx5_4 caps 5: mlx5_4: caps 0x1257e1c26 Bit Description ------------------------------------------ 01 DEVICE_BAD_PKEY_CNTR 02 DEVICE_BAD_QKEY_CNTR 05 DEVICE_CHANGE_PHY_PORT 10 DEVICE_PORT_ACTIVE_EVENT 11 DEVICE_SYS_IMAGE_GUID 12 DEVICE_RC_RNR_NAK_GEN 17 DEVICE_MEM_WINDOW 18 DEVICE_UD_IP_CSUM 19 DEVICE_UD_TSO 20 DEVICE_XRC 21 DEVICE_MEM_MGT_EXTENSIONS 22 DEVICE_BLOCK_MULTICAST_LOOPBACK 24 DEVICE_MEM_WINDOW_TYPE_2B 26 DEVICE_RAW_IP_CSUM 29 DEVICE_SIGNATURE_HANDOVER 32 DEVICE_VIRTUAL_FUNCTION
$ rdma dev show mlx5_4 cap_flags Unknown parameter 'caps_flags'.
--- rdma/dev.c | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++++++----- rdma/rdma.h | 3 ++ rdma/utils.c | 3 +- 3 files changed, 93 insertions(+), 9 deletions(-)
rdma/dev.c +88 -8
@@ -14,6 +14,7 @@static int dev_help (struct rdma *rd) {pr_out ("Usage: %s dev show [DEV]\n" , rd->filename); +pr_out (" %s dev show DEV caps\n" , rd->filename);/* * Example of set command:
@@ -22,28 +23,107 @@ static int dev_help(struct rdma *rd)return 0 ; } -static void dev_one_show (const struct dev_map *dev_map) +static const char *dev_caps[64 ] = { +"DEVICE_RESIZE_MAX_WR" , +"DEVICE_BAD_PKEY_CNTR" , +"DEVICE_BAD_QKEY_CNTR" , +"DEVICE_RAW_MULTI" , +"DEVICE_AUTO_PATH_MIG" , +"DEVICE_CHANGE_PHY_PORT" , +"DEVICE_UD_AV_PORT_ENFORCE" , +"DEVICE_CURR_QP_STATE_MOD" , +"DEVICE_SHUTDOWN_PORT" , +"DEVICE_INIT_TYPE" , +"DEVICE_PORT_ACTIVE_EVENT" , +"DEVICE_SYS_IMAGE_GUID" , +"DEVICE_RC_RNR_NAK_GEN" , +"DEVICE_SRQ_RESIZE" , +"DEVICE_N_NOTIFY_CQ" , +"DEVICE_LOCAL_DMA_LKEY" , +"DEVICE_RESERVED" , +"DEVICE_MEM_WINDOW" , +"DEVICE_UD_IP_CSUM" , +"DEVICE_UD_TSO" , +"DEVICE_XRC" , +"DEVICE_MEM_MGT_EXTENSIONS" , +"DEVICE_BLOCK_MULTICAST_LOOPBACK" , +"DEVICE_MEM_WINDOW_TYPE_2A" , +"DEVICE_MEM_WINDOW_TYPE_2B" , +"DEVICE_RC_IP_CSUM" , +"DEVICE_RAW_IP_CSUM" , +"DEVICE_CROSS_CHANNEL" , +"DEVICE_MANAGED_FLOW_STEERING" , +"DEVICE_SIGNATURE_HANDOVER" , +"DEVICE_ON_DEMAND_PAGING" , +"DEVICE_SG_GAPS_REG" , +"DEVICE_VIRTUAL_FUNCTION" , +"DEVICE_RAW_SCATTER_FCS" , +"DEVICE_RDMA_NETDEV_OPA_VNIC" , +}; + +static int dev_print_caps (struct rdma *rd) +{ +struct dev_map *dev_map = rd->dev_map_curr; +uint64_t caps = dev_map->caps; +uint32_t idx; + +pr_out ("%u: %s: " , dev_map->idx, dev_map->dev_name); +pr_out ("caps 0x%" PRIx64"\n" , dev_map->caps); +pr_out ("Bit\tDescription\n" ); +pr_out ("------------------------------------------\n" ); +for (idx =0 ; idx <64 ; idx++) { +if (caps &0x1 ) +pr_out (" %02u\t%s\n" , idx, dev_caps[idx]?dev_caps[idx]:"UNKNONW" ); + caps >>=0x1 ; + } +return 0 ; +} + +static int dev_no_args (struct rdma *rd) +{ +struct dev_map *dev_map = rd->dev_map_curr; + +pr_out ("%u: %s: " , dev_map->idx, dev_map->dev_name); +pr_out ("caps 0x%" PRIx64"\n" , dev_map->caps); +return 0 ; +} + +static int dev_one_show (struct rdma *rd) { -pr_out ("%u: %s:\n" , dev_map->idx, dev_map->dev_name); +const struct rdma_cmd cmds[] = { + {NULL , dev_no_args}, + {"caps" , dev_print_caps}, + {0 } + }; + +return rdma_exec_cmd (rd, cmds,"parameter" ); + }static int dev_show (struct rdma *rd) {struct dev_map *dev_map; +int ret =0 ;if (rd_no_arg (rd)) { -list_for_each_entry (dev_map, &rd->dev_map_list, list) -dev_one_show (dev_map); +list_for_each_entry (dev_map, &rd->dev_map_list, list) { + rd->dev_map_curr = dev_map; + ret =dev_one_show (rd); +if (ret) +return ret; + } + }else { - dev_map =dev_map_lookup (rd, false); -if (!dev_map) { + rd->dev_map_curr =dev_map_lookup (rd, false); +if (!rd->dev_map_curr) {pr_err ("Wrong device name\n" );return -ENOENT ; } -dev_one_show (dev_map); +rd_arg_inc (rd); + ret =dev_one_show (rd); } -return 0 ; +return ret; }int cmd_dev (struct rdma *rd)
rdma/rdma.h +3 -0
@@ -34,6 +34,7 @@ struct dev_map {uint32_t num_ports;struct list_head port_map_list;uint32_t idx; +uint64_t caps; };struct rdma {
@@ -41,6 +42,7 @@ struct rdma {char **argv;char *filename;struct list_head dev_map_list; +struct dev_map *dev_map_curr;struct mnl_socket *nl;struct nlmsghdr *nlh;char *buff;
@@ -57,6 +59,7 @@ struct rdma_cmd {bool rd_no_arg (struct rdma *rd);bool rd_argv_match (struct rdma *rd,const char *pattern);void rd_arg_inc (struct rdma *rd); +char *rd_argv (struct rdma *rd);/* * Commands interface
rdma/utils.c +2 -1
@@ -28,7 +28,7 @@ static int rd_argc(struct rdma *rd)return rd->argc; } -static char *rd_argv (struct rdma *rd) +char *rd_argv (struct rdma *rd) {if (!rd_argc (rd))return NULL ;
@@ -128,6 +128,7 @@ static int port_map_alloc(struct dev_map *dev_map, uint32_t num_ports) }static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX ] = { + [RDMA_NLDEV_ATTR_DEV_INDEX ] =MNL_TYPE_U32 , [RDMA_NLDEV_ATTR_DEV_NAME ] =MNL_TYPE_NUL_STRING , [RDMA_NLDEV_ATTR_PORT_INDEX ] =MNL_TYPE_U32 , };
--
2.13.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170626182128.24964-4-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add parsing interface for the device capability flags
$ rdma dev show 1: mlx5_0: caps 0x1257e1c26
This seems very un ip-like. I wouldn't show an undecoded hex value like that, it isn't really useful.
Quoted 5 lines
$ rdma dev show mlx5_4 caps 5: mlx5_4: caps 0x1257e1c26 Bit Description 01 DEVICE_BAD_PKEY_CNTR 02 DEVICE_BAD_QKEY_CNTR
This table also seems un ip-like, the usual format is a list of words, I think.
Quoted 2 lines
$ rdma dev show mlx5_4 cap_flags Unknown parameter 'caps_flags'.
?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170626182924.GB16026-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add parsing interface for the device capability flags
$ rdma dev show 1: mlx5_0: caps 0x1257e1c26This seems very un ip-like. I wouldn't show an undecoded hex value like that, it isn't really useful.
It is first supported field, after new fields will be added, we will have very similar to ip interface.
1: mlx5_0: caps 0x1257e1c2 key_1 val_1 key_2 val_2 ....
The value are presented as is can be usable as an input for different scripts.
Quoted 9 lines
Quoted 6 lines
$ rdma dev show mlx5_4 caps 5: mlx5_4: caps 0x1257e1c26 Bit Description 01 DEVICE_BAD_PKEY_CNTR 02 DEVICE_BAD_QKEY_CNTRThis table also seems un ip-like, the usual format is a list of words, I think.
It is true for key<->value data, but it is less obvious for bit parsing. Internally, I tried to present them as list and it was ugly like hell without any chance (without extra parsing) to actual see if specific capability is present or no.
The question is: is such presentation usable and does it make sense?
Quoted 5 lines
Quoted 3 lines
$ rdma dev show mlx5_4 cap_flags Unknown parameter 'caps_flags'.?
An example of "wrong" parameter. There is no such caps_flags.
Quoted 2 lines
Jason
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 12 lines · Leon Romanovsky
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Add parsing interface for the device capability flags
$ rdma dev show 1: mlx5_0: caps 0x1257e1c26This seems very un ip-like. I wouldn't show an undecoded hex value like that, it isn't really useful.
Quoted 6 lines
It is first supported field, after new fields will be added, we will have very similar to ip interface.
1:
mlx5_0: caps0x1257e1c2key_1val_1key_2val_2....The value are presented as is can be usable as an input for different scripts.
I still wouldn't show an undecoded hex value.. It isn't useful.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
$ rdma dev show mlx5_4 caps 5: mlx5_4: caps 0x1257e1c26 Bit Description 01 DEVICE_BAD_PKEY_CNTR 02 DEVICE_BAD_QKEY_CNTRThis table also seems un ip-like, the usual format is a list of words, I think.
It is true for key<->value data, but it is less obvious for bit parsing.
Several of the word decodes are from bit fields..
Quoted 3 lines
Internally, I tried to present them as list and it was ugly like hell without any chance (without extra parsing) to actual see if specific capability is present or no.
lspci seems to have no problem being readable while doing this..
Jason
A message that is not in the archive (sent off-list or not yet received): 20170626203610.GB17892-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 34 lines · Jason Gunthorpe
Quoted 12 lines · Leon Romanovsky
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add parsing interface for the device capability flags
$ rdma dev show 1: mlx5_0: caps 0x1257e1c26This seems very un ip-like. I wouldn't show an undecoded hex value like that, it isn't really useful.
Quoted 6 lines
It is first supported field, after new fields will be added, we will have very similar to ip interface.
1:
mlx5_0: caps0x1257e1c2key_1val_1key_2val_2....The values are presented as is can be usable as an input for different scripts.
I still wouldn't show an undecoded hex value.. It isn't useful.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
$ rdma dev show mlx5_4 caps 5: mlx5_4: caps 0x1257e1c26 Bit Description 01 DEVICE_BAD_PKEY_CNTR 02 DEVICE_BAD_QKEY_CNTRThis table also seems un ip-like, the usual format is a list of words, I think.
It is true for key<->value data, but it is less obvious for bit parsing.
Quoted 7 lines
Several of the word decodes are from bit fields..
Quoted 3 lines
Internally, I tried to present them as list and it was ugly like hell without any chance (without extra parsing) to actual see if specific capability is present or no.
lspci seems to have no problem being readable while doing this..
No problem,
If i understand you correctly, you are suggesting to drop parsing of "caps" as a separate command and embed it into general show <devname>.
Can you help me and give an example of how would you present those caps? What will be the output of such command?
$ rdma dev show mlx5_4
Thanks
Quoted 2 lines
Jason
A message that is not in the archive (sent off-list or not yet received): 20170627040604.GI1248-U/[email protected]
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 43 lines · Leon Romanovsky
Quoted 34 lines · Jason Gunthorpe
Quoted 12 lines · Leon Romanovsky
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add parsing interface for the device capability flags
$ rdma dev show 1: mlx5_0: caps 0x1257e1c26This seems very un ip-like. I wouldn't show an undecoded hex value like that, it isn't really useful.
Quoted 6 lines
It is first supported field, after new fields will be added, we will have very similar to ip interface.
1:
mlx5_0: caps0x1257e1c2key_1val_1key_2val_2....The values are presented as is can be usable as an input for different scripts.
I still wouldn't show an undecoded hex value.. It isn't useful.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
$ rdma dev show mlx5_4 caps 5: mlx5_4: caps 0x1257e1c26 Bit Description 01 DEVICE_BAD_PKEY_CNTR 02 DEVICE_BAD_QKEY_CNTRThis table also seems un ip-like, the usual format is a list of words, I think.
It is true for key<->value data, but it is less obvious for bit parsing.
Quoted 7 lines
Several of the word decodes are from bit fields..
Quoted 3 lines
Internally, I tried to present them as list and it was ugly like hell without any chance (without extra parsing) to actual see if specific capability is present or no.
lspci seems to have no problem being readable while doing this..
Quoted 7 lines
No problem,
If i understand you correctly, you are suggesting to drop parsing of "caps" as a separate command and embed it into general show <devname>.Can you help me and give an example of how would you present those caps? What will be the output of such command?
$ rdma dev show mlx5_4
ip-like style:
$ rdma dev show mlx5_4
5: mlx5_4:
caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>
$ rdma link show mlx5_3
4/1: mlx5_3/1:
caps: <CM, IP_BASED_GIDS>Thanks
Quoted 5 lines
Thanks
Quoted 2 lines
Jason
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 11 lines · Leon Romanovsky
Quoted 2 lines
What will be the output of such command?
$ rdma dev show mlx5_4ip-like style:
$ rdma dev show mlx5_4 5: mlx5_4: caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION> $ rdma link show mlx5_3 4/1: mlx5_3/1: caps: <CM, IP_BASED_GIDS>
I think that is better, maybe it should only show under some kind of verbose mode, I don't know, it depends what other stuff ends up being displayed..
Are you going to dump the gid table and pkey table too in one of these commands?
Jason
A message that is not in the archive (sent off-list or not yet received): 20170627164150.GA4288-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Leon Romanovsky
Quoted 2 lines
What will be the output of such command?
$ rdma dev show mlx5_4ip-like style:
$ rdma dev show mlx5_4 5: mlx5_4: caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION> $ rdma link show mlx5_3 4/1: mlx5_3/1: caps: <CM, IP_BASED_GIDS>
Quoted 5 lines
I think that is better, maybe it should only show under some kind of verbose mode, I don't know, it depends what other stuff ends up being displayed..
Are you going to dump the gid table and pkey table too in one of these commands?
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show mlx5_4 caps
So for large dumps, I'm going to use that technique again and maybe print summary as a default. For example, for gids, we can print utilization as a summary while whole
table if someone really wants it: $ rdma link show mlx5_4/1 gids <index>
Something like that.
Thanks
Quoted 2 lines
Jason
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 2 lines · Leon Romanovsky
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4caps
I think you should have a useful summary display similar to 'ip a' and other commands.
guid(s), subnet prefix or default gid for IB, lid/lmc, link state, speed, mtu, pkeys protocol(s)
A show gid table command makes sense for rocee where it can show the gid and the IP binding for it, rocee mode, etc..
Jason
A message that is not in the archive (sent off-list or not yet received): 20170627173735.GA5162-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 10 lines · Jason Gunthorpe
Quoted 2 lines · Leon Romanovsky
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4capsI think you should have a useful summary display similar to 'ip a' and other commands.
guid(s), subnet prefix or default gid for IB, lid/lmc, link state, speed, mtu, pkeys protocol(s)
It will, but before I would like to see this tool be a part of iproute2, so other people will be able to extend it in addition to me.
Are you fine with the proposed code?
Quoted 5 lines
A show gid table command makes sense for rocee where it can show the gid and the IP binding for it, rocee mode, etc..
Jason
A message that is not in the archive (sent off-list or not yet received): 20170627174615.GV1248-U/[email protected]
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 9 lines · On Tue, 27 Jun 2017 20:46:15 +0300 Leon Romanovsky…
Quoted 8 lines · Jason Gunthorpe
Quoted 2 lines · Leon Romanovsky
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4capsI think you should have a useful summary display similar to 'ip a' and other commands.
Quoted 9 lines
Quoted 2 lines
guid(s), subnet prefix or default gid for IB, lid/lmc, link state, speed, mtu, pkeys protocol(s)
It will, but before I would like to see this tool be a part of iproute2, so other people will be able to extend it in addition to me.
Are you fine with the proposed code?
Output formats need to be nailed down. The output of iproute2 commands is almost like an ABI. Users build scripts to parse it (whether that is a great idea or not is debateable, it mostly shows the weakness in programatic API's). Therefore fully changing output formats in later revisions is likely to get users upset.
The first version doesn't have to be perfect, just close to the overall goal of what is planned.
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 22 lines · Stephen Hemminger
Quoted 9 lines · On Tue, 27 Jun 2017 20:46:15 +0300 Leon Romanovsky…
Quoted 8 lines · Jason Gunthorpe
Quoted 2 lines · Leon Romanovsky
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4capsI think you should have a useful summary display similar to 'ip a' and other commands.
Quoted 9 lines
Quoted 2 lines
guid(s), subnet prefix or default gid for IB, lid/lmc, link state, speed, mtu, pkeys protocol(s)
It will, but before I would like to see this tool be a part of iproute2, so other people will be able to extend it in addition to me.
Are you fine with the proposed code?
Quoted 7 lines
Output formats need to be nailed down. The output of iproute2 commands is almost like an ABI. Users build scripts to parse it (whether that is a great idea or not is debateable, it mostly shows the weakness in programatic API's). Therefore fully changing output formats in later revisions is likely to get users upset.
The first version doesn't have to be perfect, just close to the overall goal of what is planned.
In this version, I'm going to use arrays without indexes, because I prefer to expose the bare minimum from the kernel, which is RDMA netlink. After everything else is settled, I'll move those defines to UAPI and reuse them in rdmatool.
I'll send new version with -d/--details flag and enrich minimal summary. It won't include anything related to tables gids, pkey yet.
Thanks
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
In reply to Stephen HemmingerQuoted 19 lines · Stephen Hemminger
Quoted 16 lines · On Tue, 27 Jun 2017 20:46:15 +0300 Leon Romanovsky…
Quoted 10 lines · Jason Gunthorpe
Quoted 2 lines · Leon Romanovsky
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4capsI think you should have a useful summary display similar to 'ip a' and other commands.
guid(s), subnet prefix or default gid for IB, lid/lmc, link state, speed, mtu, pkeys protocol(s)
It will, but before I would like to see this tool be a part of iproute2, so other people will be able to extend it in addition to me.
Quoted 7 lines
Quoted 2 lines
Are you fine with the proposed code?
Output formats need to be nailed down. The output of iproute2 commands is almost like an ABI. Users build scripts to parse it (whether that is a great idea or not is debateable, it mostly shows the weakness in programatic API's). Therefore fully changing output formats in later revisions is likely to get users upset.
It would be nice to see an example of what the completed command should output to make judgements on the format.. Going bit by bit doesn't really give a full picture, IHO.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 23 lines · Jason Gunthorpe
Quoted 22 lines · Stephen Hemminger
Quoted 18 lines · On Tue, 27 Jun 2017 20:46:15 +0300 Leon Romanovsky <[email protected]> wrote:
Quoted 10 lines · Jason Gunthorpe
Quoted 2 lines · Leon Romanovsky
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4capsI think you should have a useful summary display similar to 'ip a' and other commands.
guid(s), subnet prefix or default gid for IB, lid/lmc, link state, speed, mtu, pkeys protocol(s)
It will, but before I would like to see this tool be a part of iproute2, so other people will be able to extend it in addition to me.
Are you fine with the proposed code?
Quoted 8 lines
Quoted 4 lines
Output formats need to be nailed down. The output of iproute2 commands is almost like an ABI. Users build scripts to parse it (whether that is a great idea or not is debateable, it mostly shows the weakness in programatic API's). Therefore fully changing output formats in later revisions is likely to get users upset.
It would be nice to see an example of what the completed command should output to make judgements on the format.. Going bit by bit doesn't really give a full picture, IHO.
Bit by bit expansion allow easily control of what is needed. Mostly, those full examples have nothing close to real use case.
Quoted 2 lines
Jason
A message that is not in the archive (sent off-list or not yet received): 20170627173301.GS1248-U/[email protected]
Re: [PATCH iproute2 3/5] rdma: Add device capability parsing
Quoted 23 lines · On Tue, 27 Jun 2017 20:33:01 +0300 Leon Romanovsky…
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Leon Romanovsky
Quoted 2 lines
What will be the output of such command?
$ rdma dev show mlx5_4ip-like style:
$ rdma dev show mlx5_4 5: mlx5_4: caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION> $ rdma link show mlx5_3 4/1: mlx5_3/1: caps: <CM, IP_BASED_GIDS>Quoted 5 lines
I think that is better, maybe it should only show under some kind of verbose mode, I don't know, it depends what other stuff ends up being displayed..
Are you going to dump the gid table and pkey table too in one of these commands?
My initial plan was to put all parsers under their respective names, in the similar way as I did for caps: $ rdma dev show
mlx5_4caps
Quoted 10 lines
So for large dumps, I'm going to use that technique again and maybe print summary as a default. For example, for gids, we can print utilization as a summary while whole
table if someone really wants it: $ rdma link show mlx5_4/1 gids <index>Something like that.
Thanks
Quoted 2 lines
Jason
Agree with discussion so far.
For iproute2 style commands, the show and set commands need to have similar arguments. Ideally, everything after the colon in the show would be parameters to set command.
Please consider having a concise form for normal users and a detail form (with -d) for configuration and setup cases. The caps should not need to be displayed in normal show output.
PATCH 2/5 rdma: Add dev object
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Device (dev) object represents struct ib_device to user space.
The supported commands are show, set and help, but it doesn't print anything except device name at this stage. The downstream patches will fill this object with subcommands.
Print all devices:
# rdma dev 1: mlx5_0: 2: mlx5_1: 3: mlx5_2:
Print specific device:
# rdma dev show mlx5_1 2: mlx5_1:
--- rdma/Makefile | 2 +- rdma/dev.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/rdma.c | 3 ++- rdma/rdma.h | 6 ++++++ rdma/utils.c | 37 +++++++++++++++++++++++++++++++----- 5 files changed, 101 insertions(+), 7 deletions(-) create mode 100644 rdma/dev.c
rdma/Makefile +1 -1
@@ -2,7 +2,7 @@ include ../Config ifeq ($(HAVE_MNL),y) -RDMA_OBJ = rdma.o utils.o +RDMA_OBJ = rdma.o utils.o dev.o TARGETS=rdma CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
rdma/dev.c new file +60 -0
60 lines not on this page · raw message
rdma/rdma.c +2 -1
@@ -18,7 +18,7 @@static void help (char *name) {pr_out ("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n" -"where OBJECT := { help }\n" +"where OBJECT := { dev | help }\n" " OPTIONS := { -V[ersion] }\n" , name); }
@@ -33,6 +33,7 @@ static int rd_cmd(struct rdma *rd)const struct rdma_cmd cmds[] = { {NULL , cmd_help }, {"help" , cmd_help }, + {"dev" , cmd_dev }, {0 } };
rdma/rdma.h +6 -0
@@ -58,12 +58,18 @@ bool rd_no_arg(struct rdma *rd);bool rd_argv_match (struct rdma *rd,const char *pattern);void rd_arg_inc (struct rdma *rd); +/* +* Commands interface +*/ +int cmd_dev (struct rdma *rd);int rdma_exec_cmd (struct rdma *rd,const struct rdma_cmd *c,const char *str);/* * Device manipulation */ void rdma_free_devmap (struct rdma *rd); +struct dev_map *dev_map_lookup (struct rdma *rd,bool allow_port_index); +struct dev_map *_dev_map_lookup (struct rdma *rd,const char *dev_name);/* * Netlink
rdma/utils.c +32 -5
@@ -156,10 +156,9 @@ int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data)struct rdma *rd = data;const char *dev_name;uint32_t num_ports; -static int i =1 ;mnl_attr_parse (nlh,0 , rd_attr_cb, tb); -if (!tb[RDMA_NLDEV_ATTR_DEV_NAME ]) +if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX ] || !tb[RDMA_NLDEV_ATTR_DEV_NAME ])return MNL_CB_ERROR ;if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX ]) {pr_err ("This tool doesn't support switches yet\n" );
@@ -167,18 +166,18 @@ int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data) } dev_name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]); - num_ports =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_PORT_INDEX ]); - dev_map =dev_map_alloc (dev_name);if (!dev_map)/* The main function will cleanup the allocations */ return MNL_CB_ERROR ;list_add_tail (&dev_map->list, &rd->dev_map_list); - dev_map->idx = i++; + num_ports =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_PORT_INDEX ]);if (port_map_alloc (dev_map, num_ports))return MNL_CB_ERROR ; dev_map->num_ports = num_ports; + dev_map->idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); + dev_map->caps =mnl_attr_get_u64 (tb[RDMA_NLDEV_ATTR_CAP_FLAGS ]);return MNL_CB_OK ; }
@@ -268,3 +267,31 @@ int rdma_recv_msg(struct rdma *rd, mnl_cb_t callback, void *data, unsigned int smnl_socket_close (rd->nl);return ret; } + +struct dev_map *_dev_map_lookup (struct rdma *rd,const char *dev_name) +{ +struct dev_map *dev_map; + +list_for_each_entry (dev_map, &rd->dev_map_list, list) +if (strcmp (dev_name, dev_map->dev_name) ==0 ) +return dev_map; + +return NULL ; +} +struct dev_map *dev_map_lookup (struct rdma *rd,bool allow_port_index) +{ +struct dev_map *dev_map; +char *dev_name; +char *slash; + + dev_name =strdup (rd_argv (rd)); +if (allow_port_index) { + slash =strrchr (dev_name,'/' ); +if (slash) + *slash ='\0' ; + } + + dev_map =_dev_map_lookup (rd, dev_name); +free (dev_name); +return dev_map; +}
PATCH 4/5 rdma: Add link option and parsing
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Add link object interface together with port capability parsing command
$ rdma link 1/1: mlx5_0/1: caps 0x20 2/1: mlx5_1/1: caps 0x20 3/1: mlx5_2/1: caps 0x20 4/1: mlx5_3/1: caps 0x4010000 5/1: mlx5_4/1: caps 0x20
$ rdma link show mlx5_4 5/1: mlx5_4/1: caps 0x20
$ rdma link show mlx5_3 caps 4/1: mlx5_3/1: caps 0x4010000 Bit Description ------------------------------------------ 16 PORT_CM 26 PORT_IP_BASED_GIDS
$ rdma link show mlx5_4/1 caps 5/1: mlx5_4/1: caps 0x20 Bit Description ------------------------------------------ 05 PORT_AUTO_MIGR
$ rdma link show mlx5_4/1 caps_flags Unknown parameter 'caps_flags'
--- rdma/Makefile | 2 +- rdma/link.c | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ rdma/rdma.c | 1 + rdma/rdma.h | 4 ++ rdma/utils.c | 9 +++ 5 files changed, 216 insertions(+), 1 deletion(-) create mode 100644 rdma/link.c
rdma/Makefile +1 -1
@@ -2,7 +2,7 @@ include ../Config ifeq ($(HAVE_MNL),y) -RDMA_OBJ = rdma.o utils.o dev.o +RDMA_OBJ = rdma.o utils.o dev.o link.o TARGETS=rdma CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
rdma/link.c new file +201 -0
201 lines not on this page · raw message
rdma/rdma.c +1 -0
@@ -34,6 +34,7 @@ static int rd_cmd(struct rdma *rd) { NULL, cmd_help }, { "help", cmd_help }, { "dev", cmd_dev }, + { "link", cmd_link }, { 0 } };
rdma/rdma.h +4 -0
@@ -26,6 +26,7 @@struct port_map {struct list_head list;uint32_t idx; +uint64_t caps; };struct dev_map {
@@ -43,6 +44,7 @@ struct rdma {char *filename;struct list_head dev_map_list;struct dev_map *dev_map_curr; +struct port_map *port_map_curr;struct mnl_socket *nl;struct nlmsghdr *nlh;char *buff;
@@ -60,11 +62,13 @@ bool rd_no_arg(struct rdma *rd);bool rd_argv_match (struct rdma *rd,const char *pattern);void rd_arg_inc (struct rdma *rd);char *rd_argv (struct rdma *rd); +uint32_t get_port_from_argv (struct rdma *rd);/* * Commands interface */ int cmd_dev (struct rdma *rd); +int cmd_link (struct rdma *rd);int rdma_exec_cmd (struct rdma *rd,const struct rdma_cmd *c,const char *str);/*
rdma/utils.c +9 -0
@@ -62,6 +62,15 @@ bool rd_no_arg(struct rdma *rd)return rd_argc (rd) ==0 ; } +uint32_t get_port_from_argv (struct rdma *rd) +{ +char *slash; + + slash =strchr (rd_argv (rd),'/' ); +/* if no port found, return 0 */ +return (slash) ? (atoi (slash +1 )):0 ; +} +static struct dev_map *dev_map_alloc (const char *dev_name) {struct dev_map *dev_map;
PATCH 5/5 rdma: Add initial manual for the tool
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
--- man/man8/rdma.8 | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 man/man8/rdma.8
man/man8/rdma.8 new file +82 -0
@@ -0,0 +1,82 @@ +.TH RDMA 8 "28 Mar 2017" "iproute2" "Linux" +.SH NAME +rdma \- RDMA tool +.SH SYNOPSIS +.sp +.ad l +.in +8 +.ti -8 +.B rdma +.RI "[ " OPTIONS " ] " OBJECT " { " COMMAND " | " +.BR help " }" +.sp + +.ti -8 +.IR OBJECT " := { " +.BR dev " | " link " }" +.sp + +.ti -8 +.IR OPTIONS " := { " +\fB\-V\fR[\fIersion\fR] } + +.SH OPTIONS + +.TP +.BR "\-V" , " -Version" +Print the version of the +.B rdma +tool and exit. + +.SS +.I OBJECT + +.TP +.B dev +- RDMA device. + +.TP +.B link +- RDMA port related. + +.PP +The names of all objects may be written in full or +abbreviated form, for example +.B stats +can be abbreviated as +.B stat +or just +.B s. + +.SS +.I COMMAND + +Specifies the action to perform on the object. +The set of possible actions depends on the object type. +As a rule, it is possible to +.B show +(or +.B list +) objects, but some objects do not allow all of these operations +or have some additional commands. The +.B help +command is available for all objects. It prints +out a list of available commands and argument syntax conventions. +.sp +If no command is given, some default command is assumed. +Usually it is +.B list +or, if the objects of this class cannot be listed, +.BR "help" . + +.SH EXIT STATUS +Exit status is 0 if command was successful or a positive integer upon failure. + +.SH REPORTING BUGS +Report any bugs to the Linux RDMA mailing list +.B <[email protected]> +where the development and maintenance is primarily done. +You do not have to be subscribed to the list to send a message there. + +.SH AUTHOR +Leon Romanovsky <[email protected]>