Skip to contentSkip to conversation

question about the completion tasklet in the rxe driver

DISCUSSION Bob Pearson 4 participants 6 messages Mar 15, 2023 – Mar 21, 2023 InfiniBand

lore
Share & tools
lore
#0 lore raw

I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion handler. This has been in there since the driver went upstream so the history of why it is there is lost.

I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion handler can't sleep anyway.

As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is slightly better with the completion handler called inline rather than deferred to another tasklet. If we can eliminate this there won't be anymore tasklets in the rxe driver.

Does anyone know why the tasklet was put in in the first place?

Thanks,

Bob

#1 lore raw
Quoted 24 lines

-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Wednesday, 15 March 2023 22:56
To: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]; Bernard Metzler <[email protected]>
Subject: [EXTERNAL] question about the completion tasklet in the rxe driver

I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.

I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion
handler can't sleep anyway.

Quoted 4 lines

As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.

That is what I would suggest to do. Why would you leave receive processing or failing send processing w/o creating the CQE and kicking the CQ handler, if you are in a context with all information available to build a CQE, signal its availability to the consumer and kick a user handler if registered and armed?

Only exception I see: If you process the SQ in post_send() user context and a failure results in immediate CQE creation, direct CQ handler calling is not allowed - see Documentation/infiniband/core_locking.rst Not sure though, if rxe allows for direct SQ processing out of user context.

Cheers,
Bernard.

Quoted 7 lines

If we can eliminate this there won't be anymore tasklets in the rxe driver.

Does anyone know why the tasklet was put in in the first place?

Thanks,

Bob

#2 lore raw
Quoted 35 lines · Bernard Metzler
Quoted 24 lines

-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Wednesday, 15 March 2023 22:56
To: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]; Bernard Metzler <[email protected]>
Subject: [EXTERNAL] question about the completion tasklet in the rxe driver

I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.

I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion
handler can't sleep anyway.

Quoted 4 lines

As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.

That is what I would suggest to do. Why would you leave receive processing or failing send processing w/o creating the CQE and kicking the CQ handler, if you are in a context with all information available to build a CQE, signal its availability to the consumer and kick a user handler if registered and armed?

Quoted 8 lines

Only exception I see: If you process the SQ in post_send() user context and a failure results in immediate CQE creation, direct CQ handler calling is not allowed - see Documentation/infiniband/core_locking.rst Not sure though, if rxe allows for direct SQ processing out of user context.

Cheers,
Bernard.

And you did. I am not sure why the mlx5 driver defers the completion handler call to a tasklet. I could be that it gets called in a hard interrupt and completion handling is deferred to a soft interrupt context. But for rxe the completion is always already in a soft interrupt context or a process context.

Bob

Quoted 8 lines
Quoted 8 lines

If we can eliminate this there won't be anymore tasklets in the rxe driver.

Does anyone know why the tasklet was put in in the first place?

Thanks,

Bob

#3 lore raw
Quoted 60 lines

-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Thursday, 16 March 2023 18:15
To: Bernard Metzler <[email protected]>; Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]
Subject: [EXTERNAL] Re: question about the completion tasklet in the rxe driver

Quoted 8 lines · Bernard Metzler
Quoted 8 lines

-----Original Message-----
From: Bob Pearson <[email protected]>
Sent: Wednesday, 15 March 2023 22:56
To: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; [email protected]; Bernard Metzler <[email protected]>
Subject: [EXTERNAL] question about the completion tasklet in the rxe

driver

Quoted 4 lines
Quoted 4 lines

I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the tasklet

that

Quoted 7 lines
Quoted 7 lines

defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.

I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for the

rxe

Quoted 2 lines
Quoted 2 lines

driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet is

a

Quoted 8 lines
Quoted 7 lines

soft interrupt so the completion
handler can't sleep anyway.

As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.

Quoted 8 lines

That is what I would suggest to do. Why would you leave receive processing or failing send processing w/o creating the CQE and kicking the CQ handler, if you are in a context with all information available to build a CQE, signal its availability to the consumer and kick a user handler if registered and armed?

Only exception I see: If you process the SQ in post_send() user context and a failure results in immediate CQE creation, direct CQ handler

calling

Quoted 6 lines

is not allowed - see Documentation/infiniband/core_locking.rst Not sure though, if rxe allows for direct SQ processing out of user context.

Cheers,
Bernard.

Quoted 8 lines

And you did. I am not sure why the mlx5 driver defers the completion handler call
to a tasklet. I could be that it gets called in a hard interrupt and completion
handling is deferred to a soft interrupt context. But for rxe the completion
is always already in a soft interrupt context or a process context.

right, go for it ;)

Quoted 11 lines

Bob

Quoted 2 lines
Quoted 2 lines

If we can eliminate this there won't be anymore tasklets in the rxe

driver.

Quoted 6 lines
Quoted 6 lines

Does anyone know why the tasklet was put in in the first place?

Thanks,

Bob

maintainer
#4 lore raw
In reply to Bob Pearson
Quoted 4 lines · Bob Pearson

And you did. I am not sure why the mlx5 driver defers the completion handler call to a tasklet. I could be that it gets called in a hard interrupt and completion handling is deferred to a soft interrupt context. But for rxe the completion is always already in a soft interrupt context or a process context.

mlx5_eq_comp_int() is a hard IRQ context

mlx5_cq_tasklet_cb() is in a tasklet context

It has some logic that it only does a certain amount of work per tasklet call and then it reschedules itself to run again.

mlx5_ib_cq_comp() is in a tasklet context

Jason

#5 lore raw
In reply to Bob Pearson
Quoted 1 line · Bob Pearson

I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the

If work queue is used, is it possible to distribute the work queue tasks to different cpus evenly in the same numa (if numa is consider)? ^_^

Zhu Yanjun

Quoted 17 lines

three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion handler. This has been in there since the driver went upstream so the history of why it is there is lost.

I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion handler can't sleep anyway.

As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is slightly better with the completion handler called inline rather than deferred to another tasklet. If we can eliminate this there won't be anymore tasklets in the rxe driver.

Does anyone know why the tasklet was put in in the first place?

Thanks,

Bob