Skip to contentSkip to conversation

locklessly protect left members in struct rps_dev_flow

PATCH Superseded Jason Xing 2 participants 11 messages Apr 17, 2024

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jason Xing

author
#0 lore raw

v2 cover locklessly protect left members in struct rps_dev_flow

From: Jason Xing <[email protected]>

Since Eric did a more complicated locklessly change to last_qtail member[1] in struct rps_dev_flow, the left members are easier to change as the same.

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=3b4cf29bdab

v2
1. fix passing wrong type qtail.

Jason Xing (3):
net: rps: protect last_qtail with rps_input_queue_tail_save() helper
net: rps: protect filter locklessly
net: rps: locklessly access rflow->cpu

 net/core/dev.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
Signature · 1 line
2.37.3
author
#1 lore raw

PATCH v2 1/3 net: rps: protect last_qtail with rps_input_queue_tail_save() helper

From: Jason Xing <[email protected]>

Only one left place should be proctected locklessly. This patch made it.

---
 net/core/dev.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +3 -3
index 854a3a28a8d8..2003b9a61e40 100644
@@ -4501,7 +4501,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 		struct netdev_rx_queue *rxqueue;
 		struct rps_dev_flow_table *flow_table;
 		struct rps_dev_flow *old_rflow;
-		u32 flow_id;
+		u32 flow_id, head;
 		u16 rxq_index;
 		int rc;
 
@@ -4529,8 +4529,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 			old_rflow->filter = RPS_NO_FILTER;
 	out:
 #endif
-		rflow->last_qtail =
-			READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head);
+		head = READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head);
+		rps_input_queue_tail_save(&rflow->last_qtail, head);
 	}
 
 	rflow->cpu = next_cpu;
Signature · 1 line
2.37.3
author
#2 lore raw

PATCH v2 2/3 net: rps: protect filter locklessly

In reply to Jason Xing

From: Jason Xing <[email protected]>

As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.

---
I'm not very sure if the READ_ONCE in set_rps_cpu() is useful. I scaned/checked the codes and found no lock can prevent multiple threads from calling set_rps_cpu() and handling the same flow simultaneously. The same question still exists in patch [3/3].

---
 net/core/dev.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +3 -3
index 2003b9a61e40..40a535158e45 100644
@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 			goto out;
 		old_rflow = rflow;
 		rflow = &flow_table->flows[flow_id];
-		rflow->filter = rc;
-		if (old_rflow->filter == rflow->filter)
+		WRITE_ONCE(rflow->filter, rc);
+		if (old_rflow->filter == READ_ONCE(rflow->filter))
 			old_rflow->filter = RPS_NO_FILTER;
 	out:
 #endif
@@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
 	if (flow_table && flow_id <= flow_table->mask) {
 		rflow = &flow_table->flows[flow_id];
 		cpu = READ_ONCE(rflow->cpu);
-		if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
+		if (READ_ONCE(rflow->filter) == filter_id && cpu < nr_cpu_ids &&
 		    ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) -
 			   READ_ONCE(rflow->last_qtail)) <
 		     (int)(10 * flow_table->mask)))
Signature · 1 line
2.37.3
#3 lore raw

Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly

Quoted 21 lines · Jason Xing

From: Jason Xing <[email protected]>

As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.

---
I'm not very sure if the READ_ONCE in set_rps_cpu() is useful. I scaned/checked the codes and found no lock can prevent multiple threads from calling set_rps_cpu() and handling the same flow simultaneously. The same question still exists in patch [3/3].

---
 net/core/dev.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +0 -0
index 2003b9a61e40..40a535158e45 100644

net/core/dev.c set_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4528

Quoted 7 lines
net/core/dev.c +2 -2
@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
                        goto out;
                old_rflow = rflow;
                rflow = &flow_table->flows[flow_id];
-               rflow->filter = rc;
-               if (old_rflow->filter == rflow->filter)
+               WRITE_ONCE(rflow->filter, rc);
+               if (old_rflow->filter == READ_ONCE(rflow->filter))

You missed the obvious opportunity to use

               if (old_rflow->filter ==  rc)

Here your code is going to force the compiler to read the memory right after a prior write, adding a stall on some arches.

Quoted 15 lines
net/core/dev.c +1 -2
                        old_rflow->filter = RPS_NO_FILTER;
        out:
 #endif
@@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
        if (flow_table && flow_id <= flow_table->mask) {
                rflow = &flow_table->flows[flow_id];
                cpu = READ_ONCE(rflow->cpu);
-               if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
+               if (READ_ONCE(rflow->filter) == filter_id && cpu < nr_cpu_ids &&
                    ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) -
                           READ_ONCE(rflow->last_qtail)) <
                     (int)(10 * flow_table->mask)))
--

2.37.3

author
#4 lore raw

Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly

Quoted 33 lines · Eric Dumazet
Quoted 21 lines · Jason Xing

From: Jason Xing <[email protected]>

As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.

---
I'm not very sure if the READ_ONCE in set_rps_cpu() is useful. I scaned/checked the codes and found no lock can prevent multiple threads from calling set_rps_cpu() and handling the same flow simultaneously. The same question still exists in patch [3/3].

---
 net/core/dev.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +0 -0
index 2003b9a61e40..40a535158e45 100644

net/core/dev.c set_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4528

Quoted 7 lines
net/core/dev.c +2 -2
@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
                        goto out;
                old_rflow = rflow;
                rflow = &flow_table->flows[flow_id];
-               rflow->filter = rc;
-               if (old_rflow->filter == rflow->filter)
+               WRITE_ONCE(rflow->filter, rc);
+               if (old_rflow->filter == READ_ONCE(rflow->filter))

You missed the obvious opportunity to use

Quoted 4 lines
               if (old_rflow->filter ==  rc)

Here your code is going to force the compiler to read the memory right after a prior write, adding a stall on some arches.

Thanks. I see. I will remove READ_ONCE() and then reuse 'rc'.

I would like to ask one relational question: could multiple threads access the same rflow in set_rps_cpu() concurrently? Because I was thinking a lot about whether I should use the READ_ONCE() here to prevent another thread accessing/modifying this value concurrently. The answer is probably yes?

Thanks,
Jason

Quoted 16 lines
Quoted 16 lines
net/core/dev.c +1 -2
                        old_rflow->filter = RPS_NO_FILTER;
        out:
 #endif
@@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
        if (flow_table && flow_id <= flow_table->mask) {
                rflow = &flow_table->flows[flow_id];
                cpu = READ_ONCE(rflow->cpu);
-               if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
+               if (READ_ONCE(rflow->filter) == filter_id && cpu < nr_cpu_ids &&
                    ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) -
                           READ_ONCE(rflow->last_qtail)) <
                     (int)(10 * flow_table->mask)))
--

2.37.3

#5 lore raw

Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly

Quoted 40 lines · Jason Xing
Quoted 33 lines · Eric Dumazet
Quoted 21 lines · Jason Xing

From: Jason Xing <[email protected]>

As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.

---
I'm not very sure if the READ_ONCE in set_rps_cpu() is useful. I scaned/checked the codes and found no lock can prevent multiple threads from calling set_rps_cpu() and handling the same flow simultaneously. The same question still exists in patch [3/3].

---
 net/core/dev.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +0 -0
index 2003b9a61e40..40a535158e45 100644

net/core/dev.c set_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4528

Quoted 7 lines
net/core/dev.c +2 -2
@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
                        goto out;
                old_rflow = rflow;
                rflow = &flow_table->flows[flow_id];
-               rflow->filter = rc;
-               if (old_rflow->filter == rflow->filter)
+               WRITE_ONCE(rflow->filter, rc);
+               if (old_rflow->filter == READ_ONCE(rflow->filter))

You missed the obvious opportunity to use

Quoted 4 lines
               if (old_rflow->filter ==  rc)

Here your code is going to force the compiler to read the memory right after a prior write, adding a stall on some arches.

Quoted 6 lines

Thanks. I see. I will remove READ_ONCE() and then reuse 'rc'.

I would like to ask one relational question: could multiple threads access the same rflow in set_rps_cpu() concurrently? Because I was thinking a lot about whether I should use the READ_ONCE() here to prevent another thread accessing/modifying this value concurrently.

READ_ONCE() would not prevent this.

Quoted 1 line

The answer is probably yes?

I think the answer is no. rflow is located in rxqueue->rps_flow_table, it is thus private to current thread.

Only one cpu can service an RX queue at a time.

I think you can scrap the patch series.

I will instead remove the not needed annotations :

include/net/rps.h +0 -0
index a93401d23d66e45210acc73f0326087813b69d59..3f913464a2b321efe38a05dd107bf134fae6ad17

100644

include/net/rps.h +0 -0
@@ -134,7 +134,7 @@ static inline u32 rps_input_queue_tail_incr(struct

softnet_data *sd)
static inline void rps_input_queue_tail_save(u32 *dest, u32 tail) {
#ifdef CONFIG_RPS
- WRITE_ONCE(*dest, tail);
+ *dest = tail;
#endif
}

net/core/dev.c +0 -0
index 854a3a28a8d85b335a9158378ae0cca6dfbf8b36..d774e4009790c9af30d3c8f9a5eab83e9cf01bd8

100644

net/core/dev.c +0 -0
@@ -4613,7 +4613,7 @@ static int get_rps_cpu(struct net_device *dev,

struct sk_buff *skb,

                if (unlikely(tcpu != next_cpu) &&
                    (tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
                     ((int)(READ_ONCE(per_cpu(softnet_data,
tcpu).input_queue_head) -
-                     READ_ONCE(rflow->last_qtail))) >= 0)) {
+                     rflow->last_qtail)) >= 0)) {
                        tcpu = next_cpu;
                        rflow = set_rps_cpu(dev, skb, rflow, next_cpu);
                }
@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev,
u16 rxq_index,
                cpu = READ_ONCE(rflow->cpu);
                if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
                    ((int)(READ_ONCE(per_cpu(softnet_data,
cpu).input_queue_head) -
-                          READ_ONCE(rflow->last_qtail)) <
+                          rflow->last_qtail) <
                     (int)(10 * flow_table->mask)))
                        expire = false;
        }
#6 lore raw

Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly

Review

Quoted 12 lines · Eric Dumazet
(quoted diff) +0 -0
@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev,
 u16 rxq_index,
                cpu = READ_ONCE(rflow->cpu);
                if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
                    ((int)(READ_ONCE(per_cpu(softnet_data,

cpu).input_queue_head) -

(quoted diff) +1 -1
-                          READ_ONCE(rflow->last_qtail)) <
+                          rflow->last_qtail) <
                     (int)(10 * flow_table->mask)))
                        expire = false;
        }

Oh well, rps_may_expire_flow() might be called from other contexts, so only the READ_ONCE()
from get_rps_cpu() is not really necessary.

author
#7 lore raw

Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly

Quoted 18 lines · Eric Dumazet

Review

Quoted 12 lines · Eric Dumazet
(quoted diff) +0 -0
@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev,
 u16 rxq_index,
                cpu = READ_ONCE(rflow->cpu);
                if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
                    ((int)(READ_ONCE(per_cpu(softnet_data,

cpu).input_queue_head) -

(quoted diff) +1 -1
-                          READ_ONCE(rflow->last_qtail)) <
+                          rflow->last_qtail) <
                     (int)(10 * flow_table->mask)))
                        expire = false;
        }

Oh well, rps_may_expire_flow() might be called from other contexts, so only the READ_ONCE()
from get_rps_cpu() is not really necessary.

Thanks for telling me the access logic about qtail in the previous email.

Yes, I'm writing exactly what you're saying now :) I can keep protecting rflow->cpu and rflow->filter locklessly.

I can remove the unneeded annotations around qtail as you suggested with those two patches if I can, or you can submit it first. It's up to you :)

Thanks,
Jason

author
#8 lore raw

Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly

Quoted 21 lines · Jason Xing
Quoted 18 lines · Eric Dumazet

Review

Quoted 12 lines · Eric Dumazet
(quoted diff) +0 -0
@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev,
 u16 rxq_index,
                cpu = READ_ONCE(rflow->cpu);
                if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
                    ((int)(READ_ONCE(per_cpu(softnet_data,

cpu).input_queue_head) -

(quoted diff) +1 -1
-                          READ_ONCE(rflow->last_qtail)) <
+                          rflow->last_qtail) <
                     (int)(10 * flow_table->mask)))
                        expire = false;
        }

Oh well, rps_may_expire_flow() might be called from other contexts, so only the READ_ONCE()
from get_rps_cpu() is not really necessary.

Quoted 5 lines

Thanks for telling me the access logic about qtail in the previous email.

Yes, I'm writing exactly what you're saying now :) I can keep protecting rflow->cpu and rflow->filter locklessly.

I can keep these three patches just like now only without that READ_ONCE(), I have to update my statement.

[...]

Quoted 3 lines

I can remove the unneeded annotations around qtail as you suggested with those two patches if I can, or you can submit it first. It's up to you :)

The 'qtail' also needs protection. What I was saying is not true.

Quoted 3 lines

Thanks,
Jason

author
#9 lore raw

PATCH v2 3/3 net: rps: locklessly access rflow->cpu

In reply to Jason Xing

From: Jason Xing <[email protected]>

This is the last member in struct rps_dev_flow which should be protected locklessly. So finish it.

---
 net/core/dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/core/dev.c +2 -2
index 40a535158e45..aeb45025e2bc 100644
@@ -4533,7 +4533,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 		rps_input_queue_tail_save(&rflow->last_qtail, head);
 	}
 
-	rflow->cpu = next_cpu;
+	WRITE_ONCE(rflow->cpu, next_cpu);
 	return rflow;
 }
 
@@ -4597,7 +4597,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 		 * we can look at the local (per receive queue) flow table
 		 */
 		rflow = &flow_table->flows[hash & flow_table->mask];
-		tcpu = rflow->cpu;
+		tcpu = READ_ONCE(rflow->cpu);
 
 		/*
 		 * If the desired CPU (where last recvmsg was done) is
Signature · 1 line
2.37.3
author
#10 lore raw

Re: [PATCH net-next v2 3/3] net: rps: locklessly access rflow->cpu

Quoted 25 lines · Jason Xing

From: Jason Xing <[email protected]>

This is the last member in struct rps_dev_flow which should be protected locklessly. So finish it.

---
 net/core/dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/core/dev.c +1 -1
index 40a535158e45..aeb45025e2bc 100644
@@ -4533,7 +4533,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
                rps_input_queue_tail_save(&rflow->last_qtail, head);
        }
 
-       rflow->cpu = next_cpu;
+       WRITE_ONCE(rflow->cpu, next_cpu);
        return rflow;
 }
 

net/core/dev.c static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4600

Quoted 5 lines
net/core/dev.c +1 -1
@@ -4597,7 +4597,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
                 * we can look at the local (per receive queue) flow table
                 */
                rflow = &flow_table->flows[hash & flow_table->mask];
-               tcpu = rflow->cpu;
+               tcpu = READ_ONCE(rflow->cpu);

Hello Eric,

I think I don't need this one either, right? Only protecting the writer side in set_rps_cpu() and the reader side in rps_may_expire_flow() is enough.

Thanks,
Jason

Quoted 6 lines
net/core/dev.c +0 -1
                /*
                 * If the desired CPU (where last recvmsg was done) is
--

2.37.3