Skip to contentSkip to conversation

Is it ok to use debugfs to dump some ucontext-level driver-defined info?

DISCUSSION liweihang 2 participants 6 messages Dec 29, 2020 – Jan 5, 2021 InfiniBand

lore
Share & tools
lore

liweihang

author
#0 lore raw

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0

Or should it be achieved in rdmatool?

Thanks
Weihang

#1 lore raw
Quoted 11 lines · liweihang

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

Quoted 8 lines

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Quoted 3 lines

Thanks
Weihang

author
#2 lore raw
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0
Quoted 6 lines
Quoted 1 line

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may looks like:

1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:

$ rdma res show ucontext
uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

2) We want to know which ucontext a QP belongs to:

$ rdma res show qp
link hns_0/1 lqpn 1 type GSI ... uctx_id 1
link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.

Thanks
Weihang

#3 lore raw
Quoted 32 lines · liweihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0
Quoted 6 lines
Quoted 1 line

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may looks like:

Quoted 7 lines

1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:

$ rdma res show ucontext
uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.

Quoted 9 lines

2) We want to know which ucontext a QP belongs to:

$ rdma res show qp
link hns_0/1 lqpn 1 type GSI ... uctx_id 1
link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.

We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.

[leonro@vm ~]$ ibv_rc_pingpong &
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong [leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong

                                      ^^^^^^

Thanks

Quoted 4 lines

Thanks
Weihang

author
#4 lore raw
Quoted 65 lines · Leon Romanovsky
Quoted 32 lines · liweihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0
Quoted 6 lines
Quoted 1 line

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may looks like:

Quoted 7 lines

1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:

$ rdma res show ucontext
uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.

Quoted 9 lines

2) We want to know which ucontext a QP belongs to:

$ rdma res show qp
link hns_0/1 lqpn 1 type GSI ... uctx_id 1
link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.

We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.

[leonro@vm ~]$ ibv_rc_pingpong &
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong [leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong

Quoted 4 lines
                                      ^^^^^^

Thanks

OK, thank you, we will try to use rdmatool to achieve our goal.

By the way, can we use debugfs to add some trace functions? Debugfs can quickly get a large amount of information at a time, it's more convenient than netlink in such situation, especially for some vendor-defined funtions.

Thanks
Weihang

#5 lore raw
Quoted 71 lines · liweihang
Quoted 65 lines · Leon Romanovsky
Quoted 32 lines · liweihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0
Quoted 6 lines
Quoted 1 line

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may looks like:

Quoted 7 lines

1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:

$ rdma res show ucontext
uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.

Quoted 9 lines

2) We want to know which ucontext a QP belongs to:

$ rdma res show qp
link hns_0/1 lqpn 1 type GSI ... uctx_id 1
link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.

We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.

[leonro@vm ~]$ ibv_rc_pingpong &
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong [leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong

Quoted 4 lines
                                      ^^^^^^

Thanks

Quoted 6 lines

OK, thank you, we will try to use rdmatool to achieve our goal.

By the way, can we use debugfs to add some trace functions? Debugfs can quickly get a large amount of information at a time, it's more convenient than netlink in such situation, especially for some vendor-defined funtions.

If you are talking about connecting your driver to ftrace, so sure go for it. However for the data, please use rdmatool, we have raw option (-r) in the rdmatool that dumps in binary format your specific object.

Thanks

Quoted 3 lines

Thanks
Weihang