Is it ok to use debugfs to dump some ucontext-level driver-defined info?
Hi all,
We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.
rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?
This may looks like:
$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0
Or should it be achieved in rdmatool?
Thanks
Weihang
Quoted 11 lines · liweihang
Hi all,
We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.
rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?
Quoted 8 lines
This may looks like:
$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0Or should it be achieved in rdmatool?
I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?
Thanks
Quoted 3 lines
Thanks
Weihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang
Hi all,
We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.
rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?
This may looks like:
$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0
Quoted 6 lines
Quoted 1 line
Or should it be achieved in rdmatool?
I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?
Thanks
Hi Leon,
Thanks for your response. If we can achieve it in rdmatool, it may looks like:
1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:
$ rdma res show ucontext uctx_id 1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096 uctx_id 2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048
2) We want to know which ucontext a QP belongs to:
$ rdma res show qp link hns_0/1 lqpn 1 type GSI ... uctx_id 1 link hns_0/1 lqpn 2 type RC ... uctx_id 1
So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.
Thanks
Weihang
Quoted 32 lines · liweihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang
Hi all,
We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.
rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?
This may looks like:
$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0Quoted 6 lines
Quoted 1 line
Or should it be achieved in rdmatool?
I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?
Thanks
Hi Leon,
Thanks for your response. If we can achieve it in rdmatool, it may looks like:
Quoted 7 lines
1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:
$ rdma res show ucontext uctx_id 1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096 uctx_id 2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048
I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.
Quoted 9 lines
2) We want to know which ucontext a QP belongs to:
$ rdma res show qp link hns_0/1 lqpn 1 type GSI ... uctx_id 1 link hns_0/1 lqpn 2 type RC ... uctx_id 1So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.
We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.
[leonro@vm ~]$ ibv_rc_pingpong &
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong [leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong
^^^^^^
Thanks
Quoted 4 lines
Thanks
Weihang
Quoted 65 lines · Leon Romanovsky
Quoted 32 lines · liweihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang
Hi all,
We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.
rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?
This may looks like:
$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0Quoted 6 lines
Quoted 1 line
Or should it be achieved in rdmatool?
I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?
Thanks
Hi Leon,
Thanks for your response. If we can achieve it in rdmatool, it may looks like:
Quoted 7 lines
1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:
$ rdma res show ucontext uctx_id 1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096 uctx_id 2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.
Quoted 9 lines
2) We want to know which ucontext a QP belongs to:
$ rdma res show qp link hns_0/1 lqpn 1 type GSI ... uctx_id 1 link hns_0/1 lqpn 2 type RC ... uctx_id 1So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.
We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.
[leonro@vm ~]$
ibv_rc_pingpong&
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4cm_id0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 commibv_rc_pingpong[leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 commibv_rc_pingpong
Quoted 4 lines
^^^^^^Thanks
OK, thank you, we will try to use rdmatool to achieve our goal.
By the way, can we use debugfs to add some trace functions? Debugfs can quickly get a large amount of information at a time, it's more convenient than netlink in such situation, especially for some vendor-defined funtions.
Thanks
Weihang
Quoted 71 lines · liweihang
Quoted 65 lines · Leon Romanovsky
Quoted 32 lines · liweihang
Quoted 19 lines · Leon Romanovsky
Quoted 18 lines · liweihang
Hi all,
We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.
rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?
This may looks like:
$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0Quoted 6 lines
Quoted 1 line
Or should it be achieved in rdmatool?
I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?
Thanks
Hi Leon,
Thanks for your response. If we can achieve it in rdmatool, it may looks like:
Quoted 7 lines
1) We want to get some information of a ucontext (assuming that each ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:
$ rdma res show ucontext uctx_id 1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096 uctx_id 2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.
Quoted 9 lines
2) We want to know which ucontext a QP belongs to:
$ rdma res show qp link hns_0/1 lqpn 1 type GSI ... uctx_id 1 link hns_0/1 lqpn 2 type RC ... uctx_id 1So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.
We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.
[leonro@vm ~]$
ibv_rc_pingpong&
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4cm_id0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 commibv_rc_pingpong[leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 commibv_rc_pingpongQuoted 4 lines
^^^^^^Thanks
Quoted 6 lines
OK, thank you, we will try to use rdmatool to achieve our goal.
By the way, can we use debugfs to add some trace functions? Debugfs can quickly get a large amount of information at a time, it's more convenient than netlink in such situation, especially for some vendor-defined funtions.
If you are talking about connecting your driver to ftrace, so sure go for it. However for the data, please use rdmatool, we have raw option (-r) in the rdmatool that dumps in binary format your specific object.
Thanks
Quoted 3 lines
Thanks
Weihang