locklessly protect left members in struct rps_dev_flow
Subsystems (2)
v2 cover locklessly protect left members in struct rps_dev_flow
From: Jason Xing <[email protected]>
Since Eric did a more complicated locklessly change to last_qtail member[1] in struct rps_dev_flow, the left members are easier to change as the same.
[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=3b4cf29bdab
v2
1. fix passing wrong type qtail.
Jason Xing (3):
net: rps: protect last_qtail with rps_input_queue_tail_save() helper
net: rps: protect filter locklessly
net: rps: locklessly access rflow->cpu
net/core/dev.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
Signature · 1 line
2.37.3
PATCH v2 1/3 net: rps: protect last_qtail with rps_input_queue_tail_save() helper
From: Jason Xing <[email protected]>
Only one left place should be proctected locklessly. This patch made it.
--- net/core/dev.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +3 -3
@@ -4501,7 +4501,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,struct netdev_rx_queue *rxqueue;struct rps_dev_flow_table *flow_table;struct rps_dev_flow *old_rflow; -u32 flow_id; +u32 flow_id, head;u16 rxq_index;int rc;
@@ -4529,8 +4529,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, old_rflow->filter =RPS_NO_FILTER ; out:#endif - rflow->last_qtail = -READ_ONCE (per_cpu (softnet_data, next_cpu).input_queue_head); + head =READ_ONCE (per_cpu (softnet_data, next_cpu).input_queue_head); +rps_input_queue_tail_save (&rflow->last_qtail, head); } rflow->cpu = next_cpu;
Signature · 1 line
2.37.3
PATCH v2 2/3 net: rps: protect filter locklessly
In reply to Jason XingFrom: Jason Xing <[email protected]>
As we can see, rflow->filter can be written/read concurrently, so lockless access is needed.
---
I'm not very sure if the READ_ONCE in set_rps_cpu() is useful. I scaned/checked the codes and found no lock can prevent multiple threads from calling set_rps_cpu() and handling the same flow simultaneously. The same question still exists in patch [3/3].
--- net/core/dev.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/dev.c +3 -3
@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, goto out; old_rflow = rflow; rflow = &flow_table->flows[flow_id]; - rflow->filter = rc; -if (old_rflow->filter == rflow->filter) +WRITE_ONCE (rflow->filter, rc); +if (old_rflow->filter ==READ_ONCE (rflow->filter)) old_rflow->filter =RPS_NO_FILTER ; out:#endif
@@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,if (flow_table && flow_id <= flow_table->mask) { rflow = &flow_table->flows[flow_id]; cpu =READ_ONCE (rflow->cpu); -if (rflow->filter == filter_id && cpu < nr_cpu_ids && +if (READ_ONCE (rflow->filter) == filter_id && cpu < nr_cpu_ids && ((int )(READ_ONCE (per_cpu (softnet_data, cpu).input_queue_head) -READ_ONCE (rflow->last_qtail)) < (int )(10 * flow_table->mask)))
Signature · 1 line
2.37.3
Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly
Quoted 21 lines · Jason Xing
From: Jason Xing <[email protected]>
As we can see,
rflow->filtercan be written/read concurrently, so lockless access is needed.Signed-off-by Jason Xing <[email protected]>---
I'm not very sure if theREAD_ONCEinset_rps_cpu()is useful. I scaned/checked the codes and found no lock can prevent multiple threads from callingset_rps_cpu()and handling the same flow simultaneously. The same question still exists in patch [3/3].--- net/core/dev.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)net/core/dev.c +0
-0
net/core/dev.c set_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4528
Quoted 7 lines
net/core/dev.c +2
-2@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, goto out; old_rflow = rflow; rflow = &flow_table->flows[flow_id]; - rflow->filter = rc; -if (old_rflow->filter == rflow->filter) +WRITE_ONCE (rflow->filter, rc); +if (old_rflow->filter ==READ_ONCE (rflow->filter))
You missed the obvious opportunity to use
if (old_rflow->filter == rc)
Here your code is going to force the compiler to read the memory right after a prior write, adding a stall on some arches.
Quoted 15 lines
net/core/dev.c +1
-2old_rflow->filter =RPS_NO_FILTER ; out:#endif @@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,if (flow_table && flow_id <= flow_table->mask) { rflow = &flow_table->flows[flow_id]; cpu =READ_ONCE (rflow->cpu); -if (rflow->filter == filter_id && cpu < nr_cpu_ids && +if (READ_ONCE (rflow->filter) == filter_id && cpu < nr_cpu_ids && ((int )(READ_ONCE (per_cpu (softnet_data, cpu).input_queue_head) -READ_ONCE (rflow->last_qtail)) < (int )(10 * flow_table->mask))) --2.37.3
Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly
Quoted 33 lines · Eric Dumazet
Quoted 21 lines · Jason Xing
From: Jason Xing <[email protected]>
As we can see,
rflow->filtercan be written/read concurrently, so lockless access is needed.Signed-off-by Jason Xing <[email protected]>---
I'm not very sure if theREAD_ONCEinset_rps_cpu()is useful. I scaned/checked the codes and found no lock can prevent multiple threads from callingset_rps_cpu()and handling the same flow simultaneously. The same question still exists in patch [3/3].--- net/core/dev.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)net/core/dev.c +0
-0net/core/dev.c set_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4528
Quoted 7 lines
net/core/dev.c +2
-2@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, goto out; old_rflow = rflow; rflow = &flow_table->flows[flow_id]; - rflow->filter = rc; -if (old_rflow->filter == rflow->filter) +WRITE_ONCE (rflow->filter, rc); +if (old_rflow->filter ==READ_ONCE (rflow->filter))You missed the obvious opportunity to use
Quoted 4 lines
if (old_rflow->filter == rc)Here your code is going to force the compiler to read the memory right after a prior write, adding a stall on some arches.
Thanks. I see. I will remove READ_ONCE() and then reuse 'rc'.
I would like to ask one relational question: could multiple threads access the same rflow in set_rps_cpu() concurrently? Because I was thinking a lot about whether I should use the READ_ONCE() here to prevent another thread accessing/modifying this value concurrently. The answer is probably yes?
Thanks,
Jason
Quoted 16 lines
Quoted 16 lines
net/core/dev.c +1
-2old_rflow->filter =RPS_NO_FILTER ; out:#endif @@ -4666,7 +4666,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,if (flow_table && flow_id <= flow_table->mask) { rflow = &flow_table->flows[flow_id]; cpu =READ_ONCE (rflow->cpu); -if (rflow->filter == filter_id && cpu < nr_cpu_ids && +if (READ_ONCE (rflow->filter) == filter_id && cpu < nr_cpu_ids && ((int )(READ_ONCE (per_cpu (softnet_data, cpu).input_queue_head) -READ_ONCE (rflow->last_qtail)) < (int )(10 * flow_table->mask))) --2.37.3
Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly
Quoted 40 lines · Jason Xing
Quoted 33 lines · Eric Dumazet
Quoted 21 lines · Jason Xing
From: Jason Xing <[email protected]>
As we can see,
rflow->filtercan be written/read concurrently, so lockless access is needed.Signed-off-by Jason Xing <[email protected]>---
I'm not very sure if theREAD_ONCEinset_rps_cpu()is useful. I scaned/checked the codes and found no lock can prevent multiple threads from callingset_rps_cpu()and handling the same flow simultaneously. The same question still exists in patch [3/3].--- net/core/dev.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)net/core/dev.c +0
-0net/core/dev.c set_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4528
Quoted 7 lines
net/core/dev.c +2
-2@@ -4524,8 +4524,8 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb, goto out; old_rflow = rflow; rflow = &flow_table->flows[flow_id]; - rflow->filter = rc; -if (old_rflow->filter == rflow->filter) +WRITE_ONCE (rflow->filter, rc); +if (old_rflow->filter ==READ_ONCE (rflow->filter))You missed the obvious opportunity to use
Quoted 4 lines
if (old_rflow->filter == rc)Here your code is going to force the compiler to read the memory right after a prior write, adding a stall on some arches.
Quoted 6 lines
Thanks. I see. I will remove
READ_ONCE()and then reuse 'rc'.I would like to ask one relational question: could multiple threads access the same rflow in
set_rps_cpu()concurrently? Because I was thinking a lot about whether I should use theREAD_ONCE()here to prevent another thread accessing/modifying this value concurrently.
READ_ONCE() would not prevent this.
Quoted 1 line
The answer is probably yes?
I think the answer is no. rflow is located in rxqueue->rps_flow_table, it is thus private to current thread.
Only one cpu can service an RX queue at a time.
I think you can scrap the patch series.
I will instead remove the not needed annotations :
include/net/rps.h +0 -0
100644
include/net/rps.h +0 -0
@@ -134,7 +134,7 @@ static inline u32 rps_input_queue_tail_incr(struct
softnet_data *sd)
static inline void rps_input_queue_tail_save(u32 *dest, u32 tail) {
#ifdef CONFIG_RPS
- WRITE_ONCE(*dest, tail);
+ *dest = tail;
#endif
}
net/core/dev.c +0 -0
100644
net/core/dev.c +0 -0
@@ -4613,7 +4613,7 @@ static int get_rps_cpu(struct net_device *dev,
struct sk_buff *skb,
if (unlikely(tcpu != next_cpu) &&
(tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
((int)(READ_ONCE(per_cpu(softnet_data,
tcpu).input_queue_head) -
- READ_ONCE(rflow->last_qtail))) >= 0)) {
+ rflow->last_qtail)) >= 0)) {
tcpu = next_cpu;
rflow = set_rps_cpu(dev, skb, rflow, next_cpu);
}
@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev,
u16 rxq_index,
cpu = READ_ONCE(rflow->cpu);
if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
((int)(READ_ONCE(per_cpu(softnet_data,
cpu).input_queue_head) -
- READ_ONCE(rflow->last_qtail)) <
+ rflow->last_qtail) <
(int)(10 * flow_table->mask)))
expire = false;
}Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly
Review
Quoted 12 lines · Eric Dumazet
(quoted diff) +0
-0@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && ((int)(READ_ONCE(per_cpu(softnet_data,cpu).input_queue_head) -
(quoted diff) +1
-1- READ_ONCE(rflow->last_qtail)) < + rflow->last_qtail) < (int)(10 * flow_table->mask))) expire = false; }
Oh well, rps_may_expire_flow() might be called from other contexts, so only the READ_ONCE()
from get_rps_cpu() is not really necessary.
Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly
Quoted 18 lines · Eric Dumazet
Review
Quoted 12 lines · Eric Dumazet
(quoted diff) +0
-0@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && ((int)(READ_ONCE(per_cpu(softnet_data,cpu).input_queue_head) -
(quoted diff) +1
-1- READ_ONCE(rflow->last_qtail)) < + rflow->last_qtail) < (int)(10 * flow_table->mask))) expire = false; }Oh well,
rps_may_expire_flow()might be called from other contexts, so only theREAD_ONCE()
fromget_rps_cpu()is not really necessary.
Thanks for telling me the access logic about qtail in the previous email.
Yes, I'm writing exactly what you're saying now :) I can keep protecting rflow->cpu and rflow->filter locklessly.
I can remove the unneeded annotations around qtail as you suggested with those two patches if I can, or you can submit it first. It's up to you :)
Thanks,
Jason
Re: [PATCH net-next v2 2/3] net: rps: protect filter locklessly
Quoted 21 lines · Jason Xing
Quoted 18 lines · Eric Dumazet
Review
Quoted 12 lines · Eric Dumazet
(quoted diff) +0
-0@@ -4668,7 +4668,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, cpu = READ_ONCE(rflow->cpu); if (rflow->filter == filter_id && cpu < nr_cpu_ids && ((int)(READ_ONCE(per_cpu(softnet_data,cpu).input_queue_head) -
(quoted diff) +1
-1- READ_ONCE(rflow->last_qtail)) < + rflow->last_qtail) < (int)(10 * flow_table->mask))) expire = false; }Oh well,
rps_may_expire_flow()might be called from other contexts, so only theREAD_ONCE()
fromget_rps_cpu()is not really necessary.
Quoted 5 lines
Thanks for telling me the access logic about qtail in the previous email.
Yes, I'm writing exactly what you're saying now :) I can keep protecting
rflow->cpuandrflow->filterlocklessly.
I can keep these three patches just like now only without that READ_ONCE(), I have to update my statement.
[...]
Quoted 3 lines
I can remove the unneeded annotations around qtail as you suggested with those two patches if I can, or you can submit it first. It's up to you :)
The 'qtail' also needs protection. What I was saying is not true.
Quoted 3 lines
Thanks,
Jason
PATCH v2 3/3 net: rps: locklessly access rflow->cpu
In reply to Jason XingFrom: Jason Xing <[email protected]>
This is the last member in struct rps_dev_flow which should be protected locklessly. So finish it.
--- net/core/dev.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/core/dev.c +2 -2
@@ -4533,7 +4533,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,rps_input_queue_tail_save (&rflow->last_qtail, head); } - rflow->cpu = next_cpu; +WRITE_ONCE (rflow->cpu, next_cpu);return rflow; }
@@ -4597,7 +4597,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,* we can look at the local (per receive queue) flow table */ rflow = &flow_table->flows[hash & flow_table->mask]; - tcpu = rflow->cpu; + tcpu =READ_ONCE (rflow->cpu);/* * If the desired CPU (where last recvmsg was done) is
Signature · 1 line
2.37.3
Re: [PATCH net-next v2 3/3] net: rps: locklessly access rflow->cpu
Quoted 25 lines · Jason Xing
From: Jason Xing <[email protected]>
This is the last member in struct
rps_dev_flowwhich should be protected locklessly. So finish it.Signed-off-by Jason Xing <[email protected]>--- net/core/dev.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)net/core/dev.c +1
-1@@ -4533,7 +4533,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,rps_input_queue_tail_save (&rflow->last_qtail, head); } - rflow->cpu = next_cpu; +WRITE_ONCE (rflow->cpu, next_cpu);return rflow; }
net/core/dev.c static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, · line 4600
Quoted 5 lines
net/core/dev.c +1
-1@@ -4597,7 +4597,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,* we can look at the local (per receive queue) flow table */ rflow = &flow_table->flows[hash & flow_table->mask]; - tcpu = rflow->cpu; + tcpu =READ_ONCE (rflow->cpu);
Hello Eric,
I think I don't need this one either, right? Only protecting the writer side in set_rps_cpu() and the reader side in rps_may_expire_flow() is enough.
Thanks,
Jason
Quoted 6 lines
net/core/dev.c +0
-1/* * If the desired CPU (where last recvmsg was done) is -- 2.37.3