Skip to contentSkip to conversation

NMI reentrant RCU list for -rt kernels

DISCUSSION Mathieu Desnoyers 3 participants 4 messages Jul 22, 2006 – Jul 24, 2006

lore
Share & tools
lore
#0 lore raw

Hi Paul,

Following your presentation on RCU lists for -rt kernel, discussing with Bill Huey led me to the following idea that could solve the problem of NMI reentrancy of RCU read side in the -rt kernels.

If we consider that the RCU list modification that makes the read side lock preemptible is only needed for very long code paths, we could leave the original RCU implementation along with the preemptible one, so that very short and frequent code paths could benefit of using the very cheap preempt count protection without having a too big impact on the scheduler latency.

For instance, my LTTng tracer disables the preemption for about 95 ns, which I doubt would be a problem for real-time behavior. I could easily fix maximum a maximum list size so it can be run in a constant time.

So, basically, the idea is to have two RCU API that could take names like :
atomic_rcu_* and rcu_*

Does this idea make sense ?

Mathieu

OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 
#1 lore raw
Quoted 12 lines · Mathieu Desnoyers

Hi Paul,

Following your presentation on RCU lists for -rt kernel, discussing with Bill Huey led me to the following idea that could solve the problem of NMI reentrancy of RCU read side in the -rt kernels.

If we consider that the RCU list modification that makes the read side lock preemptible is only needed for very long code paths, we could leave the original RCU implementation along with the preemptible one, so that very short and frequent code paths could benefit of using the very cheap preempt count protection without having a too big impact on the scheduler latency.

Quoted 8 lines

For instance, my LTTng tracer disables the preemption for about 95 ns, which I doubt would be a problem for real-time behavior. I could easily fix maximum a maximum list size so it can be run in a constant time.

So, basically, the idea is to have two RCU API that could take names like :
atomic_rcu_* and rcu_*

Does this idea make sense ?

No,

1) Can you readily identify the very short code pathes? What about future code added to the kernel?
2) Having two parellel systems is a bad idea.
3) I believe RCU can be made much cheaper than the current implementation which look horrible.

I remember once discussing RCU on the list. I came up with the idea rcu_read_lock()/unlock() to be implemented as a per-task counter just as preempt_disable()/disable(). The run-queue then has a sum of all the counters of tasks on that cpu (minus the counter for the current task). I even made some sample code...
The only reason this wasn't considered working was the migration from CPU to CPU. I frankly can't see why this couldn't be fixed.

So the answer to you is: No. Fix the preemptible RCU instead. You have an idea above.

Esben

Quoted 12 lines

Mathieu

OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 22 lines · Esben Nielsen
Quoted 4 lines · Mathieu Desnoyers

So, basically, the idea is to have two RCU API that could take names like :
atomic_rcu_* and rcu_*

Does this idea make sense ?

No,

1) Can you readily identify the very short code pathes? What about future code added to the kernel?
2) Having two parellel systems is a bad idea.
3) I believe RCU can be made much cheaper than the
current implementation which look horrible.

I remember once discussing RCU on the list. I came up with the idea rcu_read_lock()/unlock() to be implemented as a per-task counter just as preempt_disable()/disable(). The run-queue then has a sum of all the counters of tasks on that cpu (minus the counter for the current task). I even made some sample code...
The only reason this wasn't considered working was the migration from CPU to CPU. I frankly can't see why this couldn't be fixed.

Quoted 2 lines

So the answer to you is: No. Fix the preemptible RCU instead. You have an idea above.

Hello,

For Mathieu's uses, it's critical to have a short a path as possible in his instrumentation code since the results can be effected by it as well as general impact on the kernel.

The reason why the old RCU read-side logic is ok is that in the -rt kernel RCU is use to protect things like dcache_lock and other large kernel subsystems. A non-preemptible RCU would otherwise make all locks in the file system with a RCU critical section above it in the lock graph non-preemptible or else it violates the locking rules resulting in dead locking. Since Mathieu's NMI code doesn't take other kernel locks outside of his own code, it won't create a situation where it forces parts of the -rt system back below an RCU read-side section to be non-preemptible.

A preempt_disable/enable should be good enough to restore the previous RCU behavior just for Mathieu's NMI code with maybe a different function for RCU synchronization. Making RCU safe for NMI isn't really necessary, but I'm sure it won't stop you (Paul) from trying. :)

bill

#3 lore raw
Quoted 2 lines · Bill Huey
Quoted 1 line · Esben Nielsen

On Sat, 22 Jul 2006, Mathieu Desnoyers wrote:

[discussion about making preemptible RCU both safe and fast for use in NMI]

Quoted 7 lines
Quoted 7 lines

I remember once discussing RCU on the list. I came up with the idea rcu_read_lock()/unlock() to be implemented as a per-task counter just as preempt_disable()/disable(). The run-queue then has a sum of all the counters of tasks on that cpu (minus the counter for the current task). I even made some sample code...
The only reason this wasn't considered working was the migration from CPU to CPU. I frankly can't see why this couldn't be fixed.

Forgot to mention that after talking to Paul that one of his ideas for NMI safe preemptible RCU was very close to what preemption threshold (threadX RTOS) and that if he was going to go that route that he should think about if he should generalize it for the scheduler.

Quoted 18 lines
Quoted 2 lines

So the answer to you is: No. Fix the preemptible RCU instead. You have an idea above.

Hello,

For Mathieu's uses, it's critical to have a short a path as possible in his instrumentation code since the results can be effected by it as well as general impact on the kernel.

The reason why the old RCU read-side logic is ok is that in the -rt kernel RCU is use to protect things like dcache_lock and other large kernel subsystems. A non-preemptible RCU would otherwise make all locks in the file system with a RCU critical section above it in the lock graph non-preemptible or else it violates the locking rules resulting in dead locking. Since Mathieu's NMI code doesn't take other kernel locks outside of his own code, it won't create a situation where it forces parts of the -rt system back below an RCU read-side section to be non-preemptible.

Quoted 4 lines

A preempt_disable/enable should be good enough to restore the previous RCU behavior just for Mathieu's NMI code with maybe a different function for RCU synchronization. Making RCU safe for NMI isn't really necessary, but I'm sure it won't stop you (Paul) from trying. :)

bill