Skip to contentSkip to conversation

locklessly protect left members in struct rps_dev_flow

PATCH Applied Jason Xing 4 participants 12 messages Apr 18, 2024 – May 30, 2024

Subsystems (2)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

v3 cover locklessly protect left members in struct rps_dev_flow

From: Jason Xing <[email protected]>

Since Eric did a more complicated locklessly change to last_qtail member[1] in struct rps_dev_flow, the left members are easier to change as the same.

One thing important I would like to share by qooting Eric:
"rflow is located in rxqueue->rps_flow_table, it is thus private to current thread. Only one cpu can service an RX queue at a time."
So we only pay attention to the reader in the rps_may_expire_flow() and writer in the set_rps_cpu(). They are in the two different contexts.

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=3b4cf29bdab

v3

1. adjust the protection in a right way (Eric)

v2
1. fix passing wrong type qtail.

Jason Xing (3):
net: rps: protect last_qtail with rps_input_queue_tail_save() helper
net: rps: protect filter locklessly
net: rps: locklessly access rflow->cpu

 net/core/dev.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
Signature · 1 line
2.37.3
author
#1 lore raw

PATCH v3 1/3 net: rps: protect last_qtail with rps_input_queue_tail_save() helper

From: Jason Xing <[email protected]>

Removing one unnecessary reader protection and add another writer protection to finish the locklessly proctection job.

Note: the removed READ_ONCE() is not needed because we only have to protect the locklessly reader in the different context (rps_may_expire_flow()).

---
 net/core/dev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
net/core/dev.c +4 -4
index 854a3a28a8d8..58e0da91bfef 100644
@@ -4501,7 +4501,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 		struct netdev_rx_queue *rxqueue;
 		struct rps_dev_flow_table *flow_table;
 		struct rps_dev_flow *old_rflow;
-		u32 flow_id;
+		u32 flow_id, head;
 		u16 rxq_index;
 		int rc;
 
@@ -4529,8 +4529,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 			old_rflow->filter = RPS_NO_FILTER;
 	out:
 #endif
-		rflow->last_qtail =
-			READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head);
+		head = READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head);
+		rps_input_queue_tail_save(&rflow->last_qtail, head);
 	}
 
 	rflow->cpu = next_cpu;
@@ -4613,7 +4613,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 		if (unlikely(tcpu != next_cpu) &&
 		    (tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
 		     ((int)(READ_ONCE(per_cpu(softnet_data, tcpu).input_queue_head) -
-		      READ_ONCE(rflow->last_qtail))) >= 0)) {
+		      rflow->last_qtail)) >= 0)) {
 			tcpu = next_cpu;
 			rflow = set_rps_cpu(dev, skb, rflow, next_cpu);
 		}
Signature · 1 line
2.37.3
author
#9 lore raw

PATCH v3 3/3 net: rps: locklessly access rflow->cpu

In reply to Jason Xing

From: Jason Xing <[email protected]>

This is the last member in struct rps_dev_flow which should be protected locklessly. So finish it.

---
 net/core/dev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/dev.c +1 -1
index ed6efef01582..8010036c07b6 100644
@@ -4533,7 +4533,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 		rps_input_queue_tail_save(&rflow->last_qtail, head);
 	}
 
-	rflow->cpu = next_cpu;
+	WRITE_ONCE(rflow->cpu, next_cpu);
 	return rflow;
 }
 
Signature · 1 line
2.37.3
author
#7 lore raw

PATCH v3 2/3 net: rps: protect filter locklessly

In reply to Jason Xing

From: Jason Xing <[email protected]>

As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.

---
 net/core/dev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
net/core/dev.c +4 -4
index 58e0da91bfef..ed6efef01582 100644
@@ -4524,9 +4524,9 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 			goto out;
 		old_rflow = rflow;
 		rflow = &flow_table->flows[flow_id];
-		rflow->filter = rc;
-		if (old_rflow->filter == rflow->filter)
-			old_rflow->filter = RPS_NO_FILTER;
+		WRITE_ONCE(rflow->filter, rc);
+		if (old_rflow->filter == rc)
+			WRITE_ONCE(old_rflow->filter, RPS_NO_FILTER);
 	out:
 #endif
 		head = READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head);
@@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
 	if (flow_table && flow_id <= flow_table->mask) {
 		rflow = &flow_table->flows[flow_id];
 		cpu = READ_ONCE(rflow->cpu);
-		if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
+		if (READ_ONCE(rflow->filter) == filter_id && cpu < nr_cpu_ids &&
 		    ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) -
 			   READ_ONCE(rflow->last_qtail)) <
 		     (int)(10 * flow_table->mask)))
Signature · 1 line
2.37.3
Reviewed-by on v3 1/3
#2 lore raw

Re: [PATCH net-next v3 1/3] net: rps: protect last_qtail with rps_input_queue_tail_save() helper

In reply to Jason Xing
Quoted 10 lines · Jason Xing

From: Jason Xing <[email protected]>

Removing one unnecessary reader protection and add another writer protection to finish the locklessly proctection job.

Note: the removed READ_ONCE() is not needed because we only have to protect the locklessly reader in the different context (rps_may_expire_flow()).

#11 lore raw
In reply to Jason Xing
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:

Quoted 6 lines · On Thu, 18 Apr 2024 15:36:00 +0800 you wrote:

From: Jason Xing <[email protected]>

Since Eric did a more complicated locklessly change to last_qtail member[1] in struct rps_dev_flow, the left members are easier to change as the same.

Quoted 7 lines

One thing important I would like to share by qooting Eric:
"rflow is located in rxqueue->rps_flow_table, it is thus private to current thread. Only one cpu can service an RX queue at a time."
So we only pay attention to the reader in the rps_may_expire_flow() and writer in the set_rps_cpu(). They are in the two different contexts.

[...]

Here is the summary with links:
- [net-next,v3,1/3] net: rps: protect last_qtail with rps_input_queue_tail_save() helper

    https://git.kernel.org/netdev/net-next/c/84b6823cd96b
  - [net-next,v3,2/3] net: rps: protect filter locklessly
    https://git.kernel.org/netdev/net-next/c/f00bf5dc8320
  - [net-next,v3,3/3] net: rps: locklessly access rflow->cpu
    https://git.kernel.org/netdev/net-next/c/f7b60cce8470

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#3 lore raw

compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?

In reply to Jason Xing

If CONFIG_RFS_ACCEL is off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.

#4 lore raw

Re: compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?

Quoted 2 lines · John Sperbeck

If CONFIG_RFS_ACCEL is off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.

I assume you are speaking of this commit ?

commit 84b6823cd96b38c40b3b30beabbfa48d92990e1a

Author: Jason Xing <[email protected]>
Date:   Thu Apr 18 15:36:01 2024 +0800

net: rps: protect last_qtail with rps_input_queue_tail_save() helper

author
#6 lore raw

Re: compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?

In reply to John Sperbeck

Hi John,

Quoted 2 lines · John Sperbeck

If CONFIG_RFS_ACCEL is off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.

Thanks for your report. I'll fix it today.

author
#5 lore raw

Re: compile error in set_rps_cpu() without CONFIG_RFS_ACCEL?

In reply to Eric Dumazet
Quoted 5 lines · Eric Dumazet
Quoted 2 lines · John Sperbeck

If CONFIG_RFS_ACCEL is off, then I think there will be a compile error with this change, since 'head' is used, but no longer defined.

Quoted 7 lines

I assume you are speaking of this commit ?

commit 84b6823cd96b38c40b3b30beabbfa48d92990e1a

Author: Jason Xing <[email protected]>
Date:   Thu Apr 18 15:36:01 2024 +0800

net: rps: protect last_qtail with rps_input_queue_tail_save() helper

Yes, I will fix it soon.

Thanks,
Jason