Skip to content

rcu: call_rcu() power improvements

Joel Fernandes (Google) lkml rcu 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 Latest13 patches 5 repliesIn review
  2. v813 patches 11 repliesSuperseded
  3. v711 patches 33 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v64 patches57 repliesSuperseded

Patches in v9 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/13rcu: Fix missing nocb gp wake on rcu_barrier()kernel/rcu/tree.c kernel/rcu/tree.h kernel/rcu/tree_nocb.h+17 -02 replies
2/13rcu: Make call_rcu() lazy to save powerinclude/linux/rcupdate.h kernel/rcu/Kconfig kernel/rcu/rcu.h +5 more+244 -820 replies
3/13rcu: Refactor code a bit in rcu_nocb_do_flush_bypass()kernel/rcu/tree_nocb.h+9 -80 replies
4/13rcu: shrinker for lazy rcukernel/rcu/tree_nocb.h+52 -00 replies
5/13rcuscale: Add laziness and kfree testskernel/rcu/rcuscale.c+66 -20 replies
6/13percpu-refcount: Use call_rcu_flush() for atomic switchlib/percpu-refcount.c+2 -10 replies
7/13rcu/sync: Use call_rcu_flush() instead of call_rcukernel/rcu/sync.c+1 -10 replies
8/13rcu/rcuscale: Use call_rcu_flush() for async reader testkernel/rcu/rcuscale.c+1 -10 replies
9/13rcu/rcutorture: Use call_rcu_flush() where neededkernel/rcu/rcutorture.c+8 -80 replies
10/13scsi/scsi_error: Use call_rcu_flush() instead of call_rcu()drivers/scsi/scsi_error.c+1 -10 replies
11/13workqueue: Make queue_rcu_work() use call_rcu_flush()kernel/workqueue.c+1 -10 replies
12/13rxrpc: Use call_rcu_flush() instead of call_rcu()net/rxrpc/conn_object.c+1 -10 replies
13/13rcu/debug: Add wake-up debugging for lazy callbackskernel/rcu/Kconfig kernel/rcu/lazy-debug.h kernel/rcu/tree.c+170 -00 replies

Reviews trailers per patch

1 of 13 patches has a Reviewed-by or Acked-by.

PatchFrederic WeisbeckerJoel Fernandes (Google)
1/13
2/13A-b
3/13
4/13
5/13
6/13
7/13
8/13
9/13
10/13T-b
11/13
12/13
13/13

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v9 thread

Cover letter [PATCH v9 00/13] rcu: call_rcu() power improvements

v9 version of RCU lazy patches based on rcu/next branch.
Only change since v8 is this discussion:
https://lore.kernel.org/rcu/[email protected]/T/#m8eff15110477f3430b3b02561b66f7b0d34a73b0

To facilitate easier merge, I dropped tracing and other patches and just implemented the new changes. I will post the tracing patches later along with rcutop as I need to add new tracepoints that Frederic suggested.

Main recent changes:
1. rcu_barrier() wake up only for lazy bypass list.
2. Make all call_rcu() default-lazy and add call_rcu_flush() API.
3. Take care of some callers using call_rcu_flush() API.
4. Several refactorings suggested by Paul/Frederic.
5. New call_rcu() to call_rcu_flush() conversions by Joel/Vlad/Paul.

I am seeing good performance and power with these patches on real ChromeOS x86 asymmetric hardware.

Earlier cover letter with lots of details is here:
https://lore.kernel.org/all/[email protected]/

List of recent changes:

[ Frederic Weisbec: Program the lazy timer only if WAKE_NOT, since other
deferral levels wake much earlier so for those it is not needed. ]

[ Frederic Weisbec: Use flush flags to keep bypass API code clean. ]

[ Frederic Weisbec: Make rcu_barrier() wake up only if main list empty. ]

[ Frederic Weisbec: Remove extra 'else if' branch in rcu_nocb_try_bypass(). ]

[ Joel: Fix issue where I was not resetting lazy_len after moving it to rdp ]

[ Paul/Thomas/Joel: Make call_rcu() default lazy so users don't mess up. ]

[ Paul/Frederic : Cosmetic changes, split out wakeup of nocb thread. ]

[ Vlad/Joel : More call_rcu -> flush conversions ]

[ debug code for detecting "wake" in kernel's call_rcu() callbacks. ]

The following 2 scripts can be used to check if any callbacks in the kernel are doing a wake up (it is best effort and may miss some things, but we found issues using it)

1. Script to search for call_rcu() references and dump the callback list to a file:
#!/bin/bash

rm func-list
touch func-list

for f in $(find . \( -name "*.c" -o -name "*.h" \) | grep -v rcu); do

	funcs=$(perl -0777 -ne 'while(m/call_rcu\([&]?.+,\s?(.+)\).*;/g){print "$1\n";}' $f)

	if [ "x$funcs" != "x" ]; then
		for func in $funcs; do
			echo "$f $func" >> func-list
			echo "$f $func"
		done
	fi

done

cat func-list | sort | uniq | tee func-list-sorted

2. Script to search "wake" after callback references:

#!/bin/bash

while read fl; do

	file=$(echo $fl | cut -d " " -f1)
	func=$(echo $fl | cut -d " " -f2)

	grep -A 30 $func $file | grep wake > /dev/null

	if [ $? -eq 0 ]; then
		echo "keyword wake found after function reference $func in $file"
		echo "Output:"
		grep -A 30 $func $file 
		echo "==========================================================="
	fi

done < func-list-sorted

Frederic Weisbecker (1):
rcu: Fix missing nocb gp wake on rcu_barrier()

Joel Fernandes (Google) (9):
rcu: Make call_rcu() lazy to save power
rcu: Refactor code a bit in rcu_nocb_do_flush_bypass()
rcuscale: Add laziness and kfree tests
percpu-refcount: Use call_rcu_flush() for atomic switch rcu/sync: Use call_rcu_flush() instead of call_rcu rcu/rcuscale: Use call_rcu_flush() for async reader test rcu/rcutorture: Use call_rcu_flush() where needed
rxrpc: Use call_rcu_flush() instead of call_rcu() rcu/debug: Add wake-up debugging for lazy callbacks

Uladzislau Rezki (2):
scsi/scsi_error: Use call_rcu_flush() instead of call_rcu()
workqueue: Make queue_rcu_work() use call_rcu_flush()

Vineeth Pillai (1):
rcu: shrinker for lazy rcu

drivers/scsi/scsi_error.c |   2 +-
include/linux/rcupdate.h  |   7 ++
kernel/rcu/Kconfig        |  15 +++
kernel/rcu/lazy-debug.h   | 154 +++++++++++++++++++++++++++
kernel/rcu/rcu.h          |   8 ++
kernel/rcu/rcuscale.c     |  70 +++++++++++-
kernel/rcu/rcutorture.c   |  16 +--
kernel/rcu/sync.c         |   2 +-
kernel/rcu/tiny.c         |   2 +-
kernel/rcu/tree.c         | 149 ++++++++++++++++++--------
kernel/rcu/tree.h         |  12 ++-
kernel/rcu/tree_exp.h     |   2 +-
kernel/rcu/tree_nocb.h    | 217 ++++++++++++++++++++++++++++++++------
kernel/workqueue.c        |   2 +-
lib/percpu-refcount.c     |   3 +-
net/rxrpc/conn_object.c   |   2 +-
16 files changed, 565 insertions(+), 98 deletions(-)

create mode 100644 kernel/rcu/lazy-debug.h

--
2.38.0.413.g74048e4d9e-goog