Skip to contentSkip to conversation

net: remove superfluous call to synchronize_net()

PATCH In review Eric Dumazet 3 participants 9 messages Apr 15, 2009 – Apr 17, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: remove superfluous call to synchronize_net()

inet_register_protosw() is adding inet_protosw to inetsw[] with appropriate locking section and rcu variant. No need to call synchronize_net() to wait for a RCU grace period. Changes are immediatly visible to other cpus anyway.

This saves about 13 ms on boot time on a HZ=1000 8 cpus machine ;) (4 calls to inet_register_protosw(), and about 3200 us per call)

But more seriously, we should audit all synchronize_{rcu|net}() calls to make sure we dont waste time and hide some bugs because of artificial delays.

net/ipv4/af_inet.c +0 -2
index 7f03373..1706896 100644
@@ -1003,8 +1003,6 @@ void inet_register_protosw(struct inet_protosw *p)
 out:
 	spin_unlock_bh(&inetsw_lock);
 
-	synchronize_net();
-
 	return;
 
 out_permanent:
#1 lore raw
Quoted 3 lines · Eric Dumazet

inet_register_protosw() is adding inet_protosw to inetsw[] with appropriate locking section and rcu variant. No need to call synchronize_net() to wait for a RCU grace period. Changes are immediatly visible to other cpus anyway.

I agree with the conclusion (that this change is safe), but not with the reasoning process. ;-)

The reason that this change is safe is that any inter-process communication mechanism used to tell other CPUs that this protocol has been registered must contain relevant memory barriers, otherwise, that mechanism won't be reliable.

If an unreliable mechanism was to be used, the other CPU might not yet see the protocol. For example, if the caller did a simple non-atomic store to a variable that the other CPU accessed with a simple non-atomic load, then that other CPU could potentially see the inetsw[] without the new protocol, given that inet_create() is lockless. Unlikely, but possible.

But if a proper inter-process communication mechanism is used to inform the other CPU, then the first CPU's memory operations will be seen.

So I suggest a comment to this effect.

Quoted 6 lines

This saves about 13 ms on boot time on a HZ=1000 8 cpus machine ;) (4 calls to inet_register_protosw(), and about 3200 us per call)

But more seriously, we should audit all synchronize_{rcu|net}() calls to make sure we dont waste time and hide some bugs because of artificial delays.

Good point!

Thanx, Paul

Quoted 15 lines
net/ipv4/af_inet.c +0 -2
index 7f03373..1706896 100644
@@ -1003,8 +1003,6 @@ void inet_register_protosw(struct inet_protosw *p)
 out:
 	spin_unlock_bh(&inetsw_lock);
 
-	synchronize_net();
-
 	return;
 
 out_permanent:
#2 lore raw
Quoted 5 lines · Paul E. McKenney
Quoted 3 lines · Eric Dumazet

inet_register_protosw() is adding inet_protosw to inetsw[] with appropriate locking section and rcu variant. No need to call synchronize_net() to wait for a RCU grace period. Changes are immediatly visible to other cpus anyway.

Quoted 7 lines

I agree with the conclusion (that this change is safe), but not with the reasoning process. ;-)

The reason that this change is safe is that any inter-process communication mechanism used to tell other CPUs that this protocol has been registered must contain relevant memory barriers, otherwise, that mechanism won't be reliable.

But my patch is not fixing some unreliable algo. It is already reliable, but pessimistic since containing a superflous call to not-related function.

Quoted 6 lines

If an unreliable mechanism was to be used, the other CPU might not yet see the protocol. For example, if the caller did a simple non-atomic store to a variable that the other CPU accessed with a simple non-atomic load, then that other CPU could potentially see the inetsw[] without the new protocol, given that inet_create() is lockless. Unlikely, but possible.

Well, this reasoning process is a litle it wrong too ;)
store or loads of the pointer are always atomic.
You probably meant to say that the store had to be done when memory state is stable and committed by the processor doing the _register() thing.

Quoted 5 lines

But if a proper inter-process communication mechanism is used to inform the other CPU, then the first CPU's memory operations will be seen.

So I suggest a comment to this effect.

Yes, I should really take special attention to ChangeLogs :)

Thanks a lot Patrick

[PATCH] net: remove superfluous call to synchronize_net()

inet_register_protosw() function is responsible for adding a new inet protocol into a global table (inetsw[]) that is used with RCU rules.

As soon as the store of the pointer is done, other cpus might see this new protocol in inetsw[], so we have to make sure new protocol is ready for use. All pending memory updates should thus be committed to memory before setting the pointer.
This is correctly done using rcu_assign_pointer()

synchronize_net() is typically used at unregister time, after unsetting the pointer, to make sure no other cpu is still using the object we want to dismantle. Using it at register time is only adding an artificial delay that could hide a real bug, and this bug could popup if/when synchronize_rcu() can proceed faster than now.

This saves about 13 ms on boot time on a HZ=1000 8 cpus machine ;) (4 calls to inet_register_protosw(), and about 3200 us per call)

net/ipv4/af_inet.c +0 -2
index 7f03373..1706896 100644
@@ -1003,8 +1003,6 @@ void inet_register_protosw(struct inet_protosw *p)
 out:
 	spin_unlock_bh(&inetsw_lock);
 
-	synchronize_net();
-
 	return;
 
 out_permanent:
#3 lore raw
Quoted 23 lines · Eric Dumazet
Quoted 5 lines · Paul E. McKenney
Quoted 3 lines · Eric Dumazet

inet_register_protosw() is adding inet_protosw to inetsw[] with appropriate locking section and rcu variant. No need to call synchronize_net() to wait for a RCU grace period. Changes are immediatly visible to other cpus anyway.

Quoted 7 lines

I agree with the conclusion (that this change is safe), but not with the reasoning process. ;-)

The reason that this change is safe is that any inter-process communication mechanism used to tell other CPUs that this protocol has been registered must contain relevant memory barriers, otherwise, that mechanism won't be reliable.

But my patch is not fixing some unreliable algo. It is already reliable, but pessimistic since containing a superflous call to not-related function.

Quoted 5 lines

If an unreliable mechanism was to be used, the other CPU might not yet see the protocol. For example, if the caller did a simple non-atomic store to a variable that the other CPU accessed with a simple non-atomic load, then that other CPU could potentially see the inetsw[] without the new protocol, given that inet_create() is lockless. Unlikely, but possible.

Quoted 4 lines

Well, this reasoning process is a litle it wrong too ;)
store or loads of the pointer are always atomic.
You probably meant to say that the store had to be done when memory state is stable and committed by the processor doing the _register() thing.

They are indeed atomic, but not necessarily ordered. So if you did something like:

	if (flag)
		operation_needing_protocol();

Then it is possible for things to get re-ordered so that the operation_needing_protocol() doesn't see the newly registered protocol.

Quoted 6 lines
Quoted 4 lines

But if a proper inter-process communication mechanism is used to inform the other CPU, then the first CPU's memory operations will be seen.

So I suggest a comment to this effect.

Yes, I should really take special attention to ChangeLogs :)

;-)

Quoted 13 lines

Thanks a lot Patrick

[PATCH] net: remove superfluous call to synchronize_net()

inet_register_protosw() function is responsible for adding a new inet protocol into a global table (inetsw[]) that is used with RCU rules.

As soon as the store of the pointer is done, other cpus might see this new protocol in inetsw[], so we have to make sure new protocol is ready for use. All pending memory updates should thus be committed to memory before setting the pointer.
This is correctly done using rcu_assign_pointer()

Quoted 6 lines

synchronize_net() is typically used at unregister time, after unsetting the pointer, to make sure no other cpu is still using the object we want to dismantle. Using it at register time is only adding an artificial delay that could hide a real bug, and this bug could popup if/when synchronize_rcu() can proceed faster than now.

Actually, if you make a change, then do a synchronize_rcu(), then use -any- interprocess communications mechanism, safe or not, that causes an RCU read-side critical section to execute, then that RCU read-side critical section is guaranteed to see the change.

But if you restrict yourself to safe communication mechanisms that maintain ordering (locking, atomic operations that return values, POSIX primitives, ...), then you don't need the synchronize_rcu().

Yes, I am being pedantic, but then again, I am the guy who would have to straighten out any later confusion. ;-)

Thanx, Paul

Quoted 19 lines

This saves about 13 ms on boot time on a HZ=1000 8 cpus machine ;) (4 calls to inet_register_protosw(), and about 3200 us per call)

net/ipv4/af_inet.c +0 -2
index 7f03373..1706896 100644
@@ -1003,8 +1003,6 @@ void inet_register_protosw(struct inet_protosw *p)
 out:
 	spin_unlock_bh(&inetsw_lock);
 
-	synchronize_net();
-
 	return;
 
 out_permanent:
#4 lore raw
Quoted 68 lines · Paul E. McKenney
Quoted 20 lines · Eric Dumazet
Quoted 11 lines · Paul E. McKenney
Quoted 3 lines · Eric Dumazet

inet_register_protosw() is adding inet_protosw to inetsw[] with appropriate locking section and rcu variant. No need to call synchronize_net() to wait for a RCU grace period. Changes are immediatly visible to other cpus anyway.

I agree with the conclusion (that this change is safe), but not with the reasoning process. ;-)

The reason that this change is safe is that any inter-process communication mechanism used to tell other CPUs that this protocol has been registered must contain relevant memory barriers, otherwise, that mechanism won't be reliable.

But my patch is not fixing some unreliable algo. It is already reliable, but pessimistic since containing a superflous call to not-related function.

Quoted 5 lines

If an unreliable mechanism was to be used, the other CPU might not yet see the protocol. For example, if the caller did a simple non-atomic store to a variable that the other CPU accessed with a simple non-atomic load, then that other CPU could potentially see the inetsw[] without the new protocol, given that inet_create() is lockless. Unlikely, but possible.

Quoted 4 lines

Well, this reasoning process is a litle it wrong too ;)
store or loads of the pointer are always atomic.
You probably meant to say that the store had to be done when memory state is stable and committed by the processor doing the _register() thing.

They are indeed atomic, but not necessarily ordered. So if you did something like:

	if (flag)
		operation_needing_protocol();

Then it is possible for things to get re-ordered so that the operation_needing_protocol() doesn't see the newly registered protocol.

Quoted 5 lines
Quoted 4 lines

But if a proper inter-process communication mechanism is used to inform the other CPU, then the first CPU's memory operations will be seen.

So I suggest a comment to this effect.

Yes, I should really take special attention to ChangeLogs :)

;-)

Quoted 13 lines

Thanks a lot Patrick

[PATCH] net: remove superfluous call to synchronize_net()

inet_register_protosw() function is responsible for adding a new inet protocol into a global table (inetsw[]) that is used with RCU rules.

As soon as the store of the pointer is done, other cpus might see this new protocol in inetsw[], so we have to make sure new protocol is ready for use. All pending memory updates should thus be committed to memory before setting the pointer.
This is correctly done using rcu_assign_pointer()

Quoted 6 lines

synchronize_net() is typically used at unregister time, after unsetting the pointer, to make sure no other cpu is still using the object we want to dismantle. Using it at register time is only adding an artificial delay that could hide a real bug, and this bug could popup if/when synchronize_rcu() can proceed faster than now.

Actually, if you make a change, then do a synchronize_rcu(), then use -any- interprocess communications mechanism, safe or not, that causes an RCU read-side critical section to execute, then that RCU read-side critical section is guaranteed to see the change.

Quoted 7 lines

But if you restrict yourself to safe communication mechanisms that maintain ordering (locking, atomic operations that return values, POSIX primitives, ...), then you don't need the synchronize_rcu().

Yes, I am being pedantic, but then again, I am the guy who would have to straighten out any later confusion. ;-)

OK :)

I suggest applying patch as is, and consider adding a paragraph in Documentation eventually, if you feel a clarification is needed on the subject ?

Thank you

#5 lore raw
Quoted 73 lines · Eric Dumazet
Quoted 72 lines · Paul E. McKenney
Quoted 20 lines · Eric Dumazet
Quoted 11 lines · Paul E. McKenney
Quoted 3 lines · Eric Dumazet

inet_register_protosw() is adding inet_protosw to inetsw[] with appropriate locking section and rcu variant. No need to call synchronize_net() to wait for a RCU grace period. Changes are immediatly visible to other cpus anyway.

I agree with the conclusion (that this change is safe), but not with the reasoning process. ;-)

The reason that this change is safe is that any inter-process communication mechanism used to tell other CPUs that this protocol has been registered must contain relevant memory barriers, otherwise, that mechanism won't be reliable.

But my patch is not fixing some unreliable algo. It is already reliable, but pessimistic since containing a superflous call to not-related function.

Quoted 5 lines

If an unreliable mechanism was to be used, the other CPU might not yet see the protocol. For example, if the caller did a simple non-atomic store to a variable that the other CPU accessed with a simple non-atomic load, then that other CPU could potentially see the inetsw[] without the new protocol, given that inet_create() is lockless. Unlikely, but possible.

Quoted 4 lines

Well, this reasoning process is a litle it wrong too ;)
store or loads of the pointer are always atomic.
You probably meant to say that the store had to be done when memory state is stable and committed by the processor doing the _register() thing.

They are indeed atomic, but not necessarily ordered. So if you did something like:

	if (flag)
		operation_needing_protocol();

Then it is possible for things to get re-ordered so that the operation_needing_protocol() doesn't see the newly registered protocol.

Quoted 5 lines
Quoted 4 lines

But if a proper inter-process communication mechanism is used to inform the other CPU, then the first CPU's memory operations will be seen.

So I suggest a comment to this effect.

Yes, I should really take special attention to ChangeLogs :)

;-)

Quoted 13 lines

Thanks a lot Patrick

[PATCH] net: remove superfluous call to synchronize_net()

inet_register_protosw() function is responsible for adding a new inet protocol into a global table (inetsw[]) that is used with RCU rules.

As soon as the store of the pointer is done, other cpus might see this new protocol in inetsw[], so we have to make sure new protocol is ready for use. All pending memory updates should thus be committed to memory before setting the pointer.
This is correctly done using rcu_assign_pointer()

Quoted 6 lines

synchronize_net() is typically used at unregister time, after unsetting the pointer, to make sure no other cpu is still using the object we want to dismantle. Using it at register time is only adding an artificial delay that could hide a real bug, and this bug could popup if/when synchronize_rcu() can proceed faster than now.

Actually, if you make a change, then do a synchronize_rcu(), then use -any- interprocess communications mechanism, safe or not, that causes an RCU read-side critical section to execute, then that RCU read-side critical section is guaranteed to see the change.

But if you restrict yourself to safe communication mechanisms that maintain ordering (locking, atomic operations that return values, POSIX primitives, ...), then you don't need the synchronize_rcu().

Quoted 8 lines
Quoted 3 lines

Yes, I am being pedantic, but then again, I am the guy who would have to straighten out any later confusion. ;-)

OK :)

I suggest applying patch as is, and consider adding a paragraph in Documentation eventually, if you feel a clarification is needed on the subject ?

Please add a comment where the synchronize_rcu() used to be explaining why it is not needed. The poor slob who copies your code isn't going to read theh Documentation/RCU, he is just going to expect it to magically work.

With the synchronize_rcu(), it does just magically work. Without the synchronize_rcu(), you have to be careful. Therefore, please add the comment saying that care is required.

Thanx, Paul

#6 lore raw
Quoted 9 lines · Paul E. McKenney

Please add a comment where the synchronize_rcu() used to be explaining why it is not needed. The poor slob who copies your code isn't going to read theh Documentation/RCU, he is just going to expect it to magically work.

With the synchronize_rcu(), it does just magically work. Without the synchronize_rcu(), you have to be careful. Therefore, please add the comment saying that care is required.

Sorry Paul, I dont understand why I should put a comment to say :

/*
* Dont need to use synchronize_net() or call_rcu() or msleep(100) or
* whatever function here because bla bla ...
*/

We could add this comment in about 99% of all functions in linux kernel ;)

I checked inet6_register_protosw(struct inet_protosw *p)
and it doesnt have this synchronize_rcu() neither the comment you advise...

Following construct is obvious and should not be commented in code itself.

spin_lock_bh(&somelock);

list_for_each(..., ...) {
	if (some_condition) {
		list_add_rcu(..., ...)
or		rcu_assign_pointer(...) 
		break;
	}
}

spin_unlock_bh(&somelock);

If it is not obvious, then it should be documented once in Documentation/RCU, since we find hundred of similar code in kernel.

On the contrary, places where we *use* synchronize_{rcu|net}() should get a comment to explain why this is really necessary since this function can be a real problem.

Thanks

maintainer
#7 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 16 Apr 2009 07:40:23 +0200

Quoted 18 lines

[PATCH] net: remove superfluous call to synchronize_net()

inet_register_protosw() function is responsible for adding a new inet protocol into a global table (inetsw[]) that is used with RCU rules.

As soon as the store of the pointer is done, other cpus might see this new protocol in inetsw[], so we have to make sure new protocol is ready for use. All pending memory updates should thus be committed to memory before setting the pointer.
This is correctly done using rcu_assign_pointer()

synchronize_net() is typically used at unregister time, after unsetting the pointer, to make sure no other cpu is still using the object we want to dismantle. Using it at register time is only adding an artificial delay that could hide a real bug, and this bug could popup if/when synchronize_rcu() can proceed faster than now.

Quoted 4 lines

This saves about 13 ms on boot time on a HZ=1000 8 cpus machine ;) (4 calls to inet_register_protosw(), and about 3200 us per call)

I think this change is fine, so I'm adding it to net-next-2.6

If you guys want to continue discussing the merits of putting comments in every spot where we lack a RCU sync call, that's your call. :-)

#8 lore raw
Quoted 24 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Thu, 16 Apr 2009 07:40:23 +0200

Quoted 21 lines

[PATCH] net: remove superfluous call to synchronize_net()

inet_register_protosw() function is responsible for adding a new inet protocol into a global table (inetsw[]) that is used with RCU rules.

As soon as the store of the pointer is done, other cpus might see this new protocol in inetsw[], so we have to make sure new protocol is ready for use. All pending memory updates should thus be committed to memory before setting the pointer.
This is correctly done using rcu_assign_pointer()

synchronize_net() is typically used at unregister time, after unsetting the pointer, to make sure no other cpu is still using the object we want to dismantle. Using it at register time is only adding an artificial delay that could hide a real bug, and this bug could popup if/when synchronize_rcu() can proceed faster than now.

This saves about 13 ms on boot time on a HZ=1000 8 cpus machine ;) (4 calls to inet_register_protosw(), and about 3200 us per call)

Quoted 7 lines
Quoted 1 line

I think this change is fine, so I'm adding it to net-next-2.6

If you guys want to continue discussing the merits of putting comments in every spot where we lack a RCU sync call, that's your call. :-)

Your Honor, I plead guilty to charges as read. ;-)

Thanx, Paul