locklessly protect left members in struct rps_dev_flow
Subsystems (2)
v3 cover locklessly protect left members in struct rps_dev_flow
From: Jason Xing <[email protected]>
Since Eric did a more complicated locklessly change to last_qtail member[1] in struct rps_dev_flow, the left members are easier to change as the same.
One thing important I would like to share by qooting Eric:
"rflow is located in rxqueue->rps_flow_table, it is thus private to current thread. Only one cpu can service an RX queue at a time."
So we only pay attention to the reader in the rps_may_expire_flow() and writer in the set_rps_cpu(). They are in the two different contexts.
[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=3b4cf29bdab
v3
1. adjust the protection in a right way (Eric)
v2
1. fix passing wrong type qtail.
Jason Xing (3):
net: rps: protect last_qtail with rps_input_queue_tail_save() helper
net: rps: protect filter locklessly
net: rps: locklessly access rflow->cpu
net/core/dev.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
Signature · 1 line
2.37.3
PATCH v3 1/3 net: rps: protect last_qtail with rps_input_queue_tail_save() helper
From: Jason Xing <[email protected]>
Removing one unnecessary reader protection and add another writer protection to finish the locklessly proctection job.
Note: the removed READ_ONCE() is not needed because we only have to protect the locklessly reader in the different context (rps_may_expire_flow()).
--- net/core/dev.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
net/core/dev.c +4 -4
@@ -4501,7 +4501,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,struct netdev_rx_queue *rxqueue;struct rps_dev_flow_table *flow_table;struct rps_dev_flow *old_rflow; -u32 flow_id; +u32 flow_id, head;u16 rxq_index;int rc;
@@ -4529,8 +4529,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, old_rflow->filter =RPS_NO_FILTER ; out:#endif - rflow->last_qtail = -READ_ONCE (per_cpu (softnet_data, next_cpu).input_queue_head); + head =READ_ONCE (per_cpu (softnet_data, next_cpu).input_queue_head); +rps_input_queue_tail_save (&rflow->last_qtail, head); } rflow->cpu = next_cpu;
@@ -4613,7 +4613,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,if (unlikely (tcpu != next_cpu) && (tcpu >= nr_cpu_ids || !cpu_online (tcpu) || ((int )(READ_ONCE (per_cpu (softnet_data, tcpu).input_queue_head) - -READ_ONCE (rflow->last_qtail))) >=0 )) { + rflow->last_qtail)) >=0 )) { tcpu = next_cpu; rflow =set_rps_cpu (dev, skb, rflow, next_cpu); }
Signature · 1 line
2.37.3
Re: [PATCH net-next v3 1/3] net: rps: protect last_qtail with rps_input_queue_tail_save() helper
Quoted 10 lines · Jason Xing
From: Jason Xing <[email protected]>
Removing one unnecessary reader protection and add another writer protection to finish the locklessly proctection job.
Note: the removed
READ_ONCE()is not needed because we only have to protect the locklessly reader in the different context (rps_may_expire_flow()).Signed-off-by Jason Xing <[email protected]>
compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?
In reply to Jason XingIf CONFIG_RFS_ACCEL is off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.
Re: compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?
Quoted 2 lines · John Sperbeck
If
CONFIG_RFS_ACCELis off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.
I assume you are speaking of this commit ?
commit 84b6823cd96b38c40b3b30beabbfa48d92990e1a
Author: Jason Xing <[email protected]> Date: Thu Apr 18 15:36:01 2024 +0800
net: rps: protect last_qtail with rps_input_queue_tail_save() helper
Re: compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?
Quoted 5 lines · Eric Dumazet
Quoted 2 lines · John Sperbeck
If
CONFIG_RFS_ACCELis off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.
Quoted 7 lines
I assume you are speaking of this commit ?
commit
84b6823cd96b38c40b3b30beabbfa48d92990e1aAuthor: Jason Xing <[email protected]> Date: Thu Apr 18 15:36:01 2024 +0800net: rps: protect
last_qtailwithrps_input_queue_tail_save()helper
Yes, I will fix it soon.
Thanks,
Jason
Re: compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?
In reply to John SperbeckHi John,
Quoted 2 lines · John Sperbeck
If
CONFIG_RFS_ACCELis off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.
Thanks for your report. I'll fix it today.
PATCH v3 2/3 net: rps: protect filter locklessly
In reply to Jason XingFrom: Jason Xing <[email protected]>
As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.
--- net/core/dev.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
net/core/dev.c +4 -4
@@ -4524,9 +4524,9 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, goto out; old_rflow = rflow; rflow = &flow_table->flows[flow_id]; - rflow->filter = rc; -if (old_rflow->filter == rflow->filter) - old_rflow->filter =RPS_NO_FILTER ; +WRITE_ONCE (rflow->filter, rc); +if (old_rflow->filter == rc) +WRITE_ONCE (old_rflow->filter,RPS_NO_FILTER ); out:#endif head =READ_ONCE (per_cpu (softnet_data, next_cpu).input_queue_head);
@@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,if (flow_table && flow_id <= flow_table->mask) { rflow = &flow_table->flows[flow_id]; cpu =READ_ONCE (rflow->cpu); -if (rflow->filter == filter_id && cpu < nr_cpu_ids && +if (READ_ONCE (rflow->filter) == filter_id && cpu < nr_cpu_ids && ((int )(READ_ONCE (per_cpu (softnet_data, cpu).input_queue_head) -READ_ONCE (rflow->last_qtail)) < (int )(10 * flow_table->mask)))
Signature · 1 line
2.37.3
Re: [PATCH net-next v3 2/3] net: rps: protect filter locklessly
Quoted 7 lines · Jason Xing
From: Jason Xing <[email protected]>
As we can see,
rflow->filtercan be written/read concurrently, so lockless access is needed.Signed-off-by Jason Xing <[email protected]>
PATCH v3 3/3 net: rps: locklessly access rflow->cpu
In reply to Jason XingFrom: Jason Xing <[email protected]>
This is the last member in struct rps_dev_flow which should be protected locklessly. So finish it.
--- net/core/dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/dev.c +1 -1
@@ -4533,7 +4533,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,rps_input_queue_tail_save (&rflow->last_qtail, head); } - rflow->cpu = next_cpu; +WRITE_ONCE (rflow->cpu, next_cpu);return rflow; }
Signature · 1 line
2.37.3
Re: [PATCH net-next v3 3/3] net: rps: locklessly access rflow->cpu
Quoted 7 lines · Jason Xing
From: Jason Xing <[email protected]>
This is the last member in struct
rps_dev_flowwhich should be protected locklessly. So finish it.Signed-off-by Jason Xing <[email protected]>
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Quoted 6 lines · On Thu, 18 Apr 2024 15:36:00 +0800 you wrote:
From: Jason Xing <[email protected]>
Since Eric did a more complicated locklessly change to
last_qtailmember[1] in structrps_dev_flow, the left members are easier to change as the same.
Quoted 7 lines
One thing important I would like to share by qooting Eric:
"rflow is located inrxqueue->rps_flow_table, it is thus private to current thread. Only one cpu can service an RX queue at a time."
So we only pay attention to the reader in therps_may_expire_flow()and writer in theset_rps_cpu(). They are in the two different contexts.[...]
Here is the summary with links:
- [net-next,v3,1/3] net: rps: protect last_qtail with rps_input_queue_tail_save() helper
https://git.kernel.org/netdev/net-next/c/84b6823cd96b - [net-next,v3,2/3] net: rps: protect filter locklessly https://git.kernel.org/netdev/net-next/c/f00bf5dc8320 - [net-next,v3,3/3] net: rps: locklessly access rflow->cpu https://git.kernel.org/netdev/net-next/c/f7b60cce8470
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html