graph: dispatch mode with libevent for control plane
Hi folks,
I had a look at the graph dispatch mode and wondered if it could be adapted to be used with event loops for "slow path" processing.
The use case I have in mind is having lcores dedicated for control plane operations which are not latency sensitive. It would be nice to allow data plane workers to send packets to these lcores while keeping the packets in the graph.
The problem I have with this solution for now, is that all CPUs processing the graph with rte_graph_walk_mcore_dispatch() are supposed to do busy polling. This is not compatible with select/epoll based event loops.
Would it be possible to have a way for busy-polling threads to signal non-busy-polling threads that there are packets waiting for them in the graph ring? In grout, we implemented something outside of the graph using pthread_cond_signal().
https://github.com/DPDK/grout/blob/main/modules/infra/datapath/control_output.c
https://github.com/DPDK/grout/blob/main/modules/infra/control/control_output.c
Unless I am mistaken, this does not perform any syscall and does not require acquiring any lock. Only the "slow" thread needs to acquire locks and wait for the data path threads to call pthread_cond_signal().
This solution can only be used for busy-polling -> epoll-based communication, for the other direction epoll-based -> busy-polling there would not need any specific signaling required. Only posting the packets to the graph ring would be enough.
What do you think? How would you integrate this into rte_graph? What would the API look like in your opinion?
Thanks!
Signature · 1 line
Robin
Quoted 24 lines
-----Original Message-----
From: Robin Jarry <[email protected]>
Sent: Monday, December 9, 2024 2:58 PM
To: Nitin Saxena <[email protected]>; [email protected]; [email protected]
Cc: Jerin Jacob <[email protected]>; David Marchand <[email protected]>; Christophe Fontaine <[email protected]>; Nitin Saxena <[email protected]>
Subject: [EXTERNAL] graph: dispatch mode with libevent for control planeHi folks, I had a look at the graph dispatch mode and wondered if it could be adapted to be used with event loops for "slow path" processing. The use case I have in mind is having lcores dedicated for control plane operations which are not latency
Hi folks,I had a look at the graph dispatch mode and wondered if it could be adapted to be used with event loops for "slow path" processing.
The use case I have in mind is having lcores dedicated for control plane operations which are not latency sensitive. It would be nice to allow data plane workers to send packets to these lcores while keeping the packets in the graph.
Quoted 8 lines
The problem I have with this solution for now, is that all CPUs processing the graph with
rte_graph_walk_mcore_dispatch()are supposed to do busy polling. This is not compatible with select/epoll based event loops.Would it be possible to have a way for busy-polling threads to signal non-busy- polling threads that there are packets waiting for them in the graph ring? In grout, we implemented something outside of the graph using
pthread_cond_signal().
Sorry for delay, I was on business travel.
Adding [email protected](mcore model maintainer).
If I understand it correctly, we need three operations. a)wait object creation.
b)wait object waiting instead of polling. c)signal wait object from fastpath to wake up (b)
I see two options.
Option A:
In application (without graph change)
1)call (a)
2)
while(1)
{
Call (b)
rte_graph_walk_mcore_dispatch()
}
3)Call (c) from fp nodes(Assuming it is out of tree nodes)
Option B
1)Call (a) in model create or so
2)Add an option in rte_graph_walk_mcore_dispatch() to call (b) 3)Update rte_.._enqueue() to call (c) after ring enqueue
Option B is need if we need inbuilt fp nodes.
Quoted 30 lines
https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_DPDK_grout_blob_main_modules_infra_datapath_control- 5Foutput.c&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uIT ozGOCa0s5f4wCNtTa4UUKvcsvI&m=sctQnRalWyQEGrI4o2UpcCXD8n8qveibIe4C clcMOUmS7Oi0S7MoVryikpo_ZDh1&s=Zm- i7a_D6MFxZO362gYHHe9PtKO_Kr6JeFNlmEHpMCg&e=
https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_DPDK_grout_blob_main_modules_infra_control_control- 5Foutput.c&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uIT ozGOCa0s5f4wCNtTa4UUKvcsvI&m=sctQnRalWyQEGrI4o2UpcCXD8n8qveibIe4C clcMOUmS7Oi0S7MoVryikpo_ZDh1&s=09ea7Sa47B5SPe7SjP5zBNMzqem0XzI3 HnlU-k73N2w&e=Unless I am mistaken, this does not perform any syscall and does not require acquiring any lock. Only the "slow" thread needs to acquire locks and wait for the data path threads to call
pthread_cond_signal().This solution can only be used for busy-polling -> epoll-based communication, for the other direction epoll-based -> busy-polling there would not need any specific signaling required. Only posting the packets to the graph ring would be enough.
What do you think? How would you integrate this into
rte_graph? What would the API look like in your opinion?Thanks!
--
Robin