question about the completion tasklet in the rxe driver
I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion handler. This has been in there since the driver went upstream so the history of why it is there is lost.
I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion handler can't sleep anyway.
As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is slightly better with the completion handler called inline rather than deferred to another tasklet. If we can eliminate this there won't be anymore tasklets in the rxe driver.
Does anyone know why the tasklet was put in in the first place?
Thanks,
Bob
Quoted 24 lines
-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Wednesday, 15 March 2023 22:56
To: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]; Bernard Metzler <[email protected]>
Subject: [EXTERNAL] question about the completion tasklet in the rxe driverI have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion
handler can't sleep anyway.
Quoted 4 lines
As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.
That is what I would suggest to do. Why would you leave receive processing or failing send processing w/o creating the CQE and kicking the CQ handler, if you are in a context with all information available to build a CQE, signal its availability to the consumer and kick a user handler if registered and armed?
Only exception I see: If you process the SQ in post_send() user context and a failure results in immediate CQE creation, direct CQ handler calling is not allowed - see Documentation/infiniband/core_locking.rst Not sure though, if rxe allows for direct SQ processing out of user context.
Cheers,
Bernard.
Quoted 7 lines
If we can eliminate this there won't be anymore tasklets in the rxe driver.
Does anyone know why the tasklet was put in in the first place?
Thanks,
Bob
Quoted 35 lines · Bernard Metzler
Quoted 24 lines
-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Wednesday, 15 March 2023 22:56
To: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]; Bernard Metzler <[email protected]>
Subject: [EXTERNAL] question about the completion tasklet in the rxe driverI have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion
handler can't sleep anyway.Quoted 4 lines
As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.That is what I would suggest to do. Why would you leave receive processing or failing send processing w/o creating the CQE and kicking the CQ handler, if you are in a context with all information available to build a CQE, signal its availability to the consumer and kick a user handler if registered and armed?
Quoted 8 lines
Only exception I see: If you process the SQ in
post_send()user context and a failure results in immediate CQE creation, direct CQ handler calling is not allowed - seeDocumentation/infiniband/core_locking.rstNot sure though, if rxe allows for direct SQ processing out of user context.Cheers,
Bernard.
And you did. I am not sure why the mlx5 driver defers the completion handler call to a tasklet. I could be that it gets called in a hard interrupt and completion handling is deferred to a soft interrupt context. But for rxe the completion is always already in a soft interrupt context or a process context.
Bob
Quoted 8 lines
Quoted 8 lines
If we can eliminate this there won't be anymore tasklets in the rxe driver.
Does anyone know why the tasklet was put in in the first place?
Thanks,
Bob
Quoted 60 lines
-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Thursday, 16 March 2023 18:15
To: Bernard Metzler <[email protected]>; Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]
Subject: [EXTERNAL] Re: question about the completion tasklet in the rxe driverQuoted 8 lines · Bernard Metzler
Quoted 8 lines
-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Wednesday, 15 March 2023 22:56
To: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]; Bernard Metzler <[email protected]>
Subject: [EXTERNAL] question about the completion tasklet in the rxedriver
Quoted 4 lines
Quoted 4 lines
I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the taskletthat
Quoted 7 lines
Quoted 7 lines
defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for therxe
Quoted 2 lines
Quoted 2 lines
driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet isa
Quoted 8 lines
Quoted 7 lines
soft interrupt so the completion
handler can't sleep anyway.As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.Quoted 8 lines
That is what I would suggest to do. Why would you leave receive processing or failing send processing w/o creating the CQE and kicking the CQ handler, if you are in a context with all information available to build a CQE, signal its availability to the consumer and kick a user handler if registered and armed?
Only exception I see: If you process the SQ in
post_send()user context and a failure results in immediate CQE creation, direct CQ handlercalling
Quoted 6 lines
is not allowed - see
Documentation/infiniband/core_locking.rstNot sure though, if rxe allows for direct SQ processing out of user context.Cheers,
Bernard.
Quoted 8 lines
And you did. I am not sure why the mlx5 driver defers the completion handler call
to a tasklet. I could be that it gets called in a hard interrupt and completion
handling is deferred to a soft interrupt context. But for rxe the completion
is always already in a soft interrupt context or a process context.
right, go for it ;)
Quoted 11 lines
Bob
Quoted 2 lines
Quoted 2 lines
If we can eliminate this there won't be anymore tasklets in the rxe
driver.
Quoted 6 lines
Quoted 6 lines
Does anyone know why the tasklet was put in in the first place?
Thanks,
Bob
Jason Gunthorpe
maintainerQuoted 4 lines · Bob Pearson
And you did. I am not sure why the mlx5 driver defers the completion handler call to a tasklet. I could be that it gets called in a hard interrupt and completion handling is deferred to a soft interrupt context. But for rxe the completion is always already in a soft interrupt context or a process context.
mlx5_eq_comp_int() is a hard IRQ context
mlx5_cq_tasklet_cb() is in a tasklet context
It has some logic that it only does a certain amount of work per tasklet call and then it reschedules itself to run again.
mlx5_ib_cq_comp() is in a tasklet context
Jason
Quoted 1 line · Bob Pearson
I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
If work queue is used, is it possible to distribute the work queue tasks to different cpus evenly in the same numa (if numa is consider)? ^_^
Zhu Yanjun
Quoted 17 lines
three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion handler. This has been in there since the driver went upstream so the history of why it is there is lost.
I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion handler can't sleep anyway.
As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is slightly better with the completion handler called inline rather than deferred to another tasklet. If we can eliminate this there won't be anymore tasklets in the rxe driver.
Does anyone know why the tasklet was put in in the first place?
Thanks,
Bob