Skip to contentSkip to conversation

Which reminds me...

DISCUSSION Paul E. McKenney 1 participant 1 message Jan 3, 2018 10 messages not in the archive

lore
Share & tools
lore
#10 lore raw
In reply to a message not in the archive
Quoted 3 lines · Lihao Liang

Hi Paul,

Happy New Year!

And to you and yours as well!

Quoted 1 line

Do you mind my asking some RCU questions, please?

Not only do I not mind, I will also actually attempt to answer them. ;-)

Quoted 10 lines

1. I have noticed that __call_rcu_core() checks whether interrupts are disabled and if so it returns immediately:
http://elixir.free-electrons.com/linux/v4.15-rc6/source/kernel/rcu/tree.c#L3039

And it is only called in __call_rcu():
http://elixir.free-electrons.com/linux/v4.15-rc6/source/kernel/rcu/tree.c#L3153

My understanding is that if __call_rcu() is called with interrupts disabled, __call_rcu_core() will return without invoking RCU core. If so, why?

You are correct.

The reason is that the scheduler is permitted to invoke call_rcu() while holding various interrupt-disabled scheduler locks. RCU uses interrupts being disabled as a proxy for this situation. If RCU were to fail to include this check, then such calls from the scheduler would result in self-deadlock if RCU needed to wake up the grace-period kthread.

Quoted 6 lines

2. It appears you didn't use locks in __call_rcu() to protect the per-CPU callback list in rcu_data. If we consider the following
scenario: call_rcu() is invoked by CPU 0 to add a callback to its callback list, at the same time CPU 1 is calling rcu_barrier() to register an rcu_barrier_callback in CPU 0's callback list. Would it be a race?

No, it is not.

The reason is that _rcu_barrier() invokes smp_call_function_single() to cause the other CPU to enqueue the callback on its own callback list.

Can you tell me why _rcu_barrier() doesn't check for enqueuing the callback on the current CPU?

Quoted 3 lines

3. I have performed some rcuperf experiments on rcu_ops and found the results varied a lot between different runs. Would setting dur=30*60 in kvm.sh be suitable? Shall I make it longer?

Hmmm... In the past, I have gotten repeatable results with short runs on the order of a few minutes.

But why not instead use the --duration argument to the kvm.sh script? More convenient and easier to script.

Here are some reasons that your performance might vary:

1. You have varying competing load on the same hardware, perhaps
in some other guest OS.

2. Your system has a NUMA topology and different runs are spread
differently across the NUMA nodes.

3. You are seeing random effects of thermal throttling (which messes
up scalability measurements on my laptop big time). The usual approach is to set the CPU clock frequency to a sufficiently low value to avoid thermal throttling.

4. You have different memory or cache layouts on different
runs, so are sometimes seeing false sharing and sometimes not.

5. If you are running on some other RCU implementation, it might
have hysteresis effects. This can happen in a number of ways, but one way is for the fastpath to sometimes always be taken and other times almost never be taken. This is usually a tuning problem.

For that matter, it has been some time since I did careful rcuperf studies of Linux-kernel RCU, so perhaps I managed to introduce some hysteresis. If so, please bisect!

Thanx, Paul

Quoted 191 lines

Many thanks,
Lihao.

Quoted 185 lines · Paul E. McKenney
Quoted 3 lines · Lihao Liang
Quoted 1 line · Paul E. McKenney

On Fri, Dec 15, 2017 at 02:25:46AM +0000, Lihao Liang wrote:

...

Quoted 10 lines
Quoted 2 lines

In practice, making the rcu_head structure larger will not make RCU users at all happy.

You're right...Adding an unsigned long long field to rcu_head structure would result in the following compilation error:

call to '__compiletime_assert_255' declared with attribute error:
BUILD_BUG_ON failed: offsetof(struct ds_entry, __refcnt) & 63

I have to make my own rcu_cblist structure with additional fields...

Or group the rcu_head structures by the grace-period number that they are waiting for. ;-)

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Btw, I have been reading your RCU hot-plug code and noticed the function rcu_cleanup_dead_cpu() which allows RCU to ignore a given CPU:

http://elixir.free-electrons.com/linux/v3.10/source/kernel/rcutree.c#L1960

Is it only for hot-plug operations? Can we use it to make RCU ignore an overloaded or faulty core in a given CPU?

Quoted 8 lines
Quoted 5 lines

Not directly.

If you need to ignore a faulty core/CPU, you should instead use a CPU-hotplug operation to remove that core/CPU. This will invoke rcu_cleanup_dead_cpu() along the way.

Is it what is documented at
https://www.kernel.org/doc/html/v4.11/core-api/cpu_hotplug.html?

Yes. It is possible to use this at a higher level, as is done in the online/offline code in kernel/torture.c

Quoted 5 lines
Quoted 5 lines

Making RCU ignore an overloaded CPU is an excellent way to crash your system, so please don't do that. And if a CPU became faulty while executing in an RCU read-side critical section, things become very difficult.

Quoted 7 lines
Quoted 4 lines

Note also that RCU automatically ignores idle CPUs and CPUs executing in usermode, not just offline CPUs. However, the ignoring of usermode CPUs in CONFIG_NO_HZ_FULL CPUs requires each usermode CPU to execute some RCU code.

Just out of curiosity, does the organisation of CPUs in Tree RCU take into account the cpu topology of a NUMA machine?

No.

There have been quite a few requests for this, but no one has been willing to actually run tests showing that it helps anything at the system level. (My belief is that it does not help because all the common-case RCU code avoids touching the combining tree, by design. Plus this would require RCU to keep two sets of book because some architectures number their CPUs in very odd ways. Plus I bet that they -did- run the tests, saw no improvement, and didn't bother to tell me.)

If you are using an architecture that numbers its CPUs consecutively, so that each node of the NUMA machine contains the same number of consecutively numbered CPUs, you can adjust the RCU_FANOUT and RCU_FANOUT_LEAF Kconfig options to align the rcu_node boundaries to the NUMA boundaries.

Quoted 7 lines

Btw, there is a typo in kernel/cpu.c: controll processor -> control processor (http://elixir.free-electrons.com/linux/v4.15-rc4/source/kernel/cpu.c#L1293)

I can't find the maintainer of this part of code from
https://www.kernel.org/doc/linux/MAINTAINERS. Shall I just send a patch to [email protected] against the master branch of
git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial.git

Run scripts/get_maintainer.pl with the patch as standard input, and it will output a list of maintainers and mailing lists.

Thanx, Paul

Quoted 100 lines

Many thanks,
Lihao.

Quoted 96 lines
Quoted 95 lines
Quoted 94 lines
Quoted 93 lines
Quoted 92 lines
Quoted 91 lines
Quoted 90 lines
Quoted 5 lines
Quoted 5 lines

Are there any other Linux kernel test modules that only use rcu_read_lock()/unlock() and synchronize_rcu() so that I can also play with?

The only ones intended to test RCU are rcutorture and rcuperf, although pretty much any other Linux-kernel test suite will test RCU to at least some extent.

Quoted 57 lines
Quoted 49 lines
Quoted 42 lines

Are sync and exp_sync tested very differently in these test cases?

Not in current kernels, but the handling of exp_sync has changed
over time. But recall my earlier email -- there are other rcutorture
parameters that are set by default that will result in warnings, cbflood
and the like.

But even with the same parameters, why sync complains but exp_sync
doesn't, providing both use the same function?

Exactly how did it complain? Something like this?

rcu_torture_writer: gp_normal without primitives

Or something different?

As I want to test .sync, so I only set the torture parameter gp_sync
to true and provide an implementation to .sync. The NULL pointer
exception is caused by the following line of code as gp_normal is
false and .exp_sync is NULL.

http://elixir.free-electrons.com/linux/v4.12.9/source/kernel/rcu/rcutorture.c#L1120

Shall the above if statement be if (gp_sync) instead of if (gp_normal)?

Wait... If you are setting .sync but not .exp_sync, you would also need
to set the rcutorture module parameters to prevent rcutorture from ever
testing expedited grace periods. That should mean setting gp_normal but
not gp_exp, which would mean that control would always go to line 1121.
Or you could just make .exp_sync reference the same function as .sync.

You could argue that this is a usability bug in rcutorture, and you would
be right. If you would like, you could send me a patch that issued a
warning if the gp_normal and gp_exp parameters didn't make sense given
the rcu_torture_ops entry selected. This would add to the big "if"
statement in rcu_torture_writer().

Please note that such patches require a surprising amount of testing,
given the rather large number of possible combinations of rcu_torture_ops
fields and rcutorture module-parameter settings.

If there are only three types wait primitives:

gp_cond -> cond_sync
gp_exp ->exp_sync
gp_sync/gp_normal -> sync

Quoted 7 lines

However, gp_normal is also related to deferred_free as indicated at
http://elixir.free-electrons.com/linux/v4.12.9/source/kernel/rcu/rcutorture.c#L1000

This might be confusing to a new user of rcutorture (probably only to me though...). It might be clearer to use gp_sync to test sync and gp_normal (or a different name such as gp_deferred_free) to test deferred_free only.

Quoted 6 lines

So the new dependencies become:

gp_cond -> cond_sync
gp_exp ->exp_sync
gp_sync -> sync
gp_normal (or gp_deferred_free) -> deferred_free

I might be willing to consider a patch to this effect, but it took me a few rounds to get the current approach working naturally, so please take the time to study it. I don't -think- there is any effect on the rcutorture scripting, but it would be good to double-check.

Thanx, Paul

Quoted 8 lines

Best,
Lihao.

Quoted 4 lines

Am I missing something here?

Many thanks,
Lihao.