dumping queue state

Hey,

I have a need to provide tools for customers to gather runtime state for an
rdma device. Say, when an application is stuck waiting for some completion
or other rdma event. This includes hw/fw state of course, and equally as
important, rdma object sw state. Is debugfs the correct way to export this sw state? The data is quite large potentially; each QP, its structures, the dma queue memory, etc. Ditto for CQs. Also MR state, etc etc. It seems that would be overloading debugfs to me. Currently the hw/fw state is being gathered via ethtool dump commands (--get-dump, --register-dump, --eeprom-dump). I am considering using the ethtool --get-dump method for the low level driver to also include dumping the rdma queue state for the
device. Is that a reasonable approach?

Any thoughts/suggestions?

Thanks in advance,

Steve.

---
This email has been checked for viruses by AVG.
http://www.avg.com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

T24gV2VkLCAyMDE3LTEyLTIwIGF0IDA5OjUzIC0wNjAwLCBTdGV2ZSBXaXNlIHdyb3RlOg0KPiBJ IGhhdmUgYSBuZWVkIHRvIHByb3ZpZGUgdG9vbHMgZm9yIGN1c3RvbWVycyB0byBnYXRoZXIgcnVu dGltZSBzdGF0ZSBmb3IgYW4NCj4gcmRtYSBkZXZpY2UuICAgU2F5LCB3aGVuIGFuIGFwcGxpY2F0 aW9uIGlzIHN0dWNrIHdhaXRpbmcgZm9yIHNvbWUgY29tcGxldGlvbg0KPiBvciBvdGhlciByZG1h IGV2ZW50LiAgIFRoaXMgaW5jbHVkZXMgaHcvZncgc3RhdGUgb2YgY291cnNlLCBhbmQgZXF1YWxs eSBhcw0KPiBpbXBvcnRhbnQsIHJkbWEgb2JqZWN0IHN3IHN0YXRlLiAgSXMgZGVidWdmcyB0aGUg Y29ycmVjdCB3YXkgdG8gZXhwb3J0IHRoaXMNCj4gc3cgc3RhdGU/ICBUaGUgZGF0YSBpcyBxdWl0 ZSBsYXJnZSBwb3RlbnRpYWxseTsgZWFjaCBRUCwgaXRzIHN0cnVjdHVyZXMsIHRoZQ0KPiBkbWEg cXVldWUgbWVtb3J5LCBldGMuICBEaXR0byBmb3IgQ1FzLiAgQWxzbyBNUiBzdGF0ZSwgZXRjIGV0 Yy4gIEl0IHNlZW1zDQo+IHRoYXQgd291bGQgYmUgb3ZlcmxvYWRpbmcgZGVidWdmcyB0byBtZS4g IEN1cnJlbnRseSB0aGUgaHcvZncgc3RhdGUgaXMgYmVpbmcNCj4gZ2F0aGVyZWQgdmlhIGV0aHRv b2wgZHVtcCBjb21tYW5kcyAoLS1nZXQtZHVtcCwgLS1yZWdpc3Rlci1kdW1wLA0KPiAtLWVlcHJv bS1kdW1wKS4gIEkgYW0gY29uc2lkZXJpbmcgdXNpbmcgdGhlIGV0aHRvb2wgLS1nZXQtZHVtcCBt ZXRob2QgZm9yDQo+IHRoZSBsb3cgbGV2ZWwgZHJpdmVyIHRvIGFsc28gaW5jbHVkZSBkdW1waW5n IHRoZSByZG1hIHF1ZXVlIHN0YXRlIGZvciB0aGUNCj4gZGV2aWNlLiAgIElzIHRoYXQgYSByZWFz b25hYmxlIGFwcHJvYWNoPw0KPiANCj4gQW55IHRob3VnaHRzL3N1Z2dlc3Rpb25zPyANCg0KSGVs bG8gU3RldmUsDQoNCkknbSBub3Qgc3VyZSB3aGF0IHRoZSBiZXN0IHNvbHV0aW9uIGlzIGZvciBS RE1BIHF1ZXVlcy4gQnV0IGlmIHlvdSB3YW50IHlvdSBjYW4NCmhhdmUgYSBsb29rIGF0IGJsb2Nr L2Jsay1tcS1kZWJ1Z2ZzLmMuIFRoYXQgY29kZSBoYXMgYmVlbiBhZGRlZCByZWNlbnRseSBhbmQg aGFzDQpiZWVuIHZlcnkgdmFsdWFibGUgZm9yIGRlYnVnZ2luZyBibG9jayBsYXllciAvIFNDU0kg cXVldWUgbG9ja3Vwcy4NCg0KQmFydC4=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

In this cycle, I'm going to submit RDMA resource tracking feature, which will give an infrastructure to do it in RDMA:

The kernel code, it is based on RCU and not final, because my call to synchronize_rcu is not effective.
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/restrack-rcu

There is supplementary part in RDMAtool, which presents global information, QP information with options to filter.

It is initial stage.

+ /mnt/iproute2/rdma/rdma res
1: mlx5_0: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144 2: mlx5_1: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144 3: mlx5_2: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144 4: mlx5_3: curr/max: pd 2/16777216 cq 3/16777216 qp 2/262144 5: mlx5_4: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
+ /mnt/iproute2/rdma/rdma res show mlx5_4
5: mlx5_4: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4

DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/-  8          UD    RESET  0          [ipoib-verbs]
mlx5_4/1  7          UD    RTS    0          [mlx5-gsi]
mlx5_4/1  1          GSI   RTS    0          [rdma-mad]
mlx5_4/1  0          SMI   RTS    0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/-  8          UD    RESET  0          [ipoib-verbs]
mlx5_4/1  7          UD    RTS    0          [mlx5-gsi]
mlx5_4/1  1          GSI   RTS    0          [rdma-mad]
mlx5_4/1  0          SMI   RTS    0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/0
Wrong device name
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/1  7          UD    RTS    0          [mlx5-gsi]
mlx5_4/1  1          GSI   RTS    0          [rdma-mad]
mlx5_4/1  0          SMI   RTS    0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/-
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/-  8          UD    RESET  0          [ipoib-verbs]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/- -d
DEV/PORT  LQPN       RQPN       TYPE  STATE  PID        COMM            SQ-PSN     RQ-PSN     PATH-MIG
mlx5_4/-  8          ---        UD    RESET  0          [ipoib-verbs]   0          ---        ---
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1 display pid,lqpn,comm
DEV/PORT  LQPN       PID        COMM
mlx5_4/1  7          0          [mlx5-gsi]
mlx5_4/1  1          0          [rdma-mad]
mlx5_4/1  0          0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1 display pid,lqpn,comm -d
DEV/PORT  LQPN       PID        COMM
mlx5_4/1  7          0          [mlx5-gsi]
mlx5_4/1  1          0          [rdma-mad]
mlx5_4/1  0          0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1 display pid,lqpn,comm pid 0-2000
DEV/PORT  LQPN       PID        COMM
mlx5_4/1  7          0          [mlx5-gsi]
mlx5_4/1  1          0          [rdma-mad]
mlx5_4/1  0          0          [rdma-mad]

Care to try?

Sure, I'm building the kernel now. Where do I get the rdma command that will work with this?

Is there a way to extend commands with provider-specific data?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I uploaded iproute2 part.
https://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git/log/?h=topic/restrack

WIP: no manuals, no review yet and not massively tested - but works for me :

In future yes, rdmatool provides "-d" option (detailed output) and IMHO all vendor specific will got there.

Thanks

:

Cool.

Looksie:

[root@stevo1 iproute2]# ./rdma/rdma res
1: mlx4_0: curr/max: pd 4/32764 cq 17/65408 qp 3/163768 2: cxgb4_0: curr/max: pd 2/65536 cq 16/24576 qp 2/12288 2: cxgb4_0: curr/max: pd 2/65536 cq 16/24576 qp 2/12288

[root@stevo1 iproute2]# ./rdma/rdma res show qp link cxgb4_0

DEV/PORT  LQPN       TYPE  STATE  PID        COMM
cxgb4_0/- 1028       RC    RTS    0          [rdma-cm]
cxgb4_0/- 1026       RC    RTS    0          [rdma-cm]

I'm not sure why the port is not specified on the above cxgb4 QPs which are connected. They should have a port number. But perhaps it isn't filled in correctly in the cm_id or something?

[root@stevo1 iproute2]# ./rdma/rdma res show qp link mlx4_0

DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx4_0/1  537        RC    RTS    0          [rdma-cm]
mlx4_0/1  536        RC    RTS    0          [rdma-cm]
mlx4_0/1  1          GSI   RTS    0          [rdma-mad]

[root@stevo1 iproute2]#

rdma should probably understand enough transport-specific details to not display ib-specific stuff on an iwarp qp. Here PSNs and PATH-MIG are all IB-specific:

[root@stevo1 iproute2]# ./rdma/rdma res show qp link cxgb4_0 -d
DEV/PORT LQPN RQPN TYPE STATE PID COMM
SQ-PSN RQ-PSN PATH-MIG
cxgb4_0/- 1028 0 RC RTS 0 [rdma-cm] 0
0 MIGRATED
cxgb4_0/- 1026 0 RC RTS 0 [rdma-cm] 0
0 MIGRATED

[root@stevo1 iproute2]# ./rdma/rdma res show qp link mlx4_0 -d
DEV/PORT LQPN RQPN TYPE STATE PID COMM
SQ-PSN RQ-PSN PATH-MIG
mlx4_0/1 537 537 RC RTS 0 [rdma-cm]
13145128 12441844 MIGRATED
mlx4_0/1 536 536 RC RTS 0 [rdma-cm]
14218681 1982803 MIGRATED
mlx4_0/1 1 --- GSI RTS 0 [rdma-mad] 0
--- ---

Looking good though!

Why did you prefer netlink vs debugfs for this stuff? (I'm sure it was discussed on the list but I missed it).

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I'm not using CM and take this information from ib_query_qp
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/tree/drivers/infiniband/core/nldev.c?h=topic/restrack-rcu#n236
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/tree/drivers/infiniband/core/nldev.c?h=topic/restrack-rcu#n249

Most probably it returns port = 0 and it is not valid for the RDMAtool.

Right, it is bug.

Check filters options too, to filters by numbers and display columns.

Three reasons:
1. We wanted rdmatool to be part of iproute2.
2. Debugfs requires root access and netlink doesn't
3. Better extensibility, if the device doesn't support the netlink won't provide attribute and all neltink libraries know how to deal with it.

[root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link cxgb4_0/- display lqpn,type,state,pid lqpn 1000-1030

DEV/PORT  LQPN       TYPE  STATE  PID
cxgb4_0/- 1027       RC    RTS    0
[root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link cxgb4_0/-
display lqpn,type,state,pid lqpn 1000-1033
DEV/PORT  LQPN       TYPE  STATE  PID
cxgb4_0/- 1027       RC    RTS    0
cxgb4_0/- 1033       RC    RTS    0

[root@stevo1 ~]#

Working!

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks,
It supports various combinations with "," and "-", like 990,1000-1030,1031-1032,1033

While the rdma tool easily allows poking around various resources, it would be sort of painful to do a full state dump. Perhaps a fastpath command like 'rdma dumpall' to just dump everything in detail?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

It can be an option after we add more objects, right now, I added only QP and this one object is "full state" for now. To dump all QPs in detailed format simply run "rdma res show qp -d" and add "-j" to output it in JSON format.

Thanks

By the way, it will be very useful to dump cm_ids, or at least include the ip address/port numbers for each connected qp...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Will do, we need it for RoCE too.