RCU tree changes for v3.20
Subsystems (36)
Linus,
Please pull the latest core-rcu-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core-rcu-for-linus
# HEAD: f49028292c13b958fdf4f36c8cc8119d0dde187b Merge branch 'for-mingo' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
The main RCU changes in this cycle are:
- Documentation updates.
- Miscellaneous fixes.
- Preemptible-RCU fixes, including fixing an old bug in the interaction of RCU priority boosting and CPU hotplug.
- SRCU updates.
- RCU CPU stall-warning updates.
- RCU torture-test updates.
Thanks,
Ingo
------------------>
Alexander Gordeev (1):
rcu: Remove redundant rcu_is_cpu_rrupt_from_idle() from tiny RCU
Calvin Owens (1):
ksoftirqd: Enable IRQs and call cond_resched() before poking RCU
David Hildenbrand (1):
hotplugcpu: Avoid deadlocks by waking active_writer
Davidlohr Bueso (1):
documentation: Fix smp typo in memory-barriers.txt
Lai Jiangshan (3):
tiny_rcu: Directly force QS when call_rcu_[bh|sched]() on idle_task
rcu: Remove "select IRQ_WORK" from config TREE_RCU
rcu: Revert "Allow post-unlock reference for rt_mutex" to avoid priority-inversion
Miroslav Benes (1):
rcu: Fix RCU CPU stall detection in tiny implementation
Paul E. McKenney (42):
rcu: Make rcu_nmi_enter() handle nesting
rcu: Fix invoke_rcu_callbacks() comment
compiler: Allow 1- and 2-byte smp_load_acquire() and smp_store_release()
rcu: Fix rcu_barrier() race that could result in too-short wait
rcu: Protect rcu_boost() lockless accesses with ACCESS_ONCE()
rcu: Rename "empty" to "empty_norm" in preparation for boost rework
rcu: Abstract rcu_cleanup_dead_rnp() from rcu_cleanup_dead_cpu()
rcu: Make rcu_read_unlock_special() propagate ->qsmaskinit bit clearing
rcu: Don't migrate blocked tasks even if all corresponding CPUs offline
rcu: Shorten irq-disable region in rcu_cleanup_dead_cpu()
rcu: Make use of rcu_preempt_has_tasks()
rcu: Don't spawn rcub kthreads on root rcu_node structure
rcu: Don't initiate RCU priority boosting on root rcu_node
rcu: Don't bother affinitying rcub kthreads away from offline CPUs
rcu: Note quiescent state when CPU goes offline
rcu: Don't scan root rcu_node structure for stalled tasks
rcu: Remove redundant callback-list initialization
rcu: Expand SRCU ->completed to 64 bits
rcu: Combine DEFINE_SRCU() and DEFINE_STATIC_SRCU()
rcu: Set default to RCU_CPU_STALL_INFO=y
rcutorture: Add checks for stall ending before dump start
rcu: Make RCU_CPU_STALL_INFO include number of fqs attempts
rcu: Improve diagnostics for spurious RCU CPU stall warnings
rcu: Handle gpnum/completed wrap while dyntick idle
documentation: Record limitations of bitfields and small variables
rcutorture: Issue warnings on close calls due to Reader Batch blows
rcu: Make _batches_completed() functions return unsigned long
rcutorture: Make build-output parsing correctly flag RCU's warnings
rcutorture: Use unsigned for Reader Batch computations
rcu: Provide rcu_batches_completed_sched() for TINY_RCU
rcutorture: Drop rcu_torture_completed() and friends
rcu: Remove redundant rcu_batches_completed() declaration
rcutorture: Check from beginning to end of grace period
rcutorture: Handle different mpstat versions
torture: Add "-enable-kvm -soundhw pcspk" to qemu command line
torture: Flag console.log file to prevent holdovers from earlier runs
rcutorture: Add more diagnostics in rcu_barrier() test failure case
ksoftirqd: Use new cond_resched_rcu_qs() function
rcu: Optionally run grace-period kthreads at real-time priority
rcu: Make cond_resched_rcu_qs() apply to normal RCU flavors
rcu: Add GP-kthread-starvation checks to CPU stall warnings
rcu: Initialize tiny RCU stall-warning timeouts at boot
Pranith Kumar (2):
rcupdate: Replace smp_read_barrier_depends() with lockless_dereference()
rcu: Make SRCU optional by using CONFIG_SRCU
Xie XiuQi (1):
documentation: Update sysfs path for rcu_cpu_stall_timeout
Ying Xue (1):
rculist: Fix sparse warning
Documentation/RCU/stallwarn.txt | 16 +- Documentation/RCU/trace.txt | 32 +- Documentation/memory-barriers.txt | 46 ++- arch/arm/kvm/Kconfig | 1 + arch/arm64/kvm/Kconfig | 1 + arch/mips/kvm/Kconfig | 1 + arch/powerpc/kvm/Kconfig | 1 + arch/s390/kvm/Kconfig | 1 + arch/tile/kvm/Kconfig | 1 + arch/x86/Kconfig | 1 + arch/x86/kvm/Kconfig | 1 + drivers/clk/Kconfig | 1 + drivers/cpufreq/Kconfig | 1 + drivers/devfreq/Kconfig | 1 + drivers/md/Kconfig | 1 + drivers/net/Kconfig | 1 + fs/btrfs/Kconfig | 1 + fs/notify/Kconfig | 1 + fs/quota/Kconfig | 1 + include/linux/compiler.h | 2 +- include/linux/rculist.h | 16 +- include/linux/rcupdate.h | 13 +- include/linux/rcutiny.h | 45 ++- include/linux/rcutree.h | 11 +- include/linux/srcu.h | 14 +- init/Kconfig | 18 +- kernel/cpu.c | 56 ++-- kernel/notifier.c | 3 + kernel/power/Kconfig | 1 + kernel/rcu/Makefile | 3 +- kernel/rcu/rcu.h | 6 + kernel/rcu/rcutorture.c | 66 ++-- kernel/rcu/srcu.c | 2 +- kernel/rcu/tiny.c | 113 +------ kernel/rcu/tiny_plugin.h | 9 +- kernel/rcu/tree.c | 355 +++++++++++++++------ kernel/rcu/tree.h | 62 +--- kernel/rcu/tree_plugin.h | 271 +++++----------- kernel/rcu/tree_trace.c | 8 +- kernel/softirq.c | 3 +- lib/Kconfig.debug | 3 +- mm/Kconfig | 1 + security/tomoyo/Kconfig | 1 + tools/testing/selftests/rcutorture/bin/cpus2use.sh | 2 +- .../selftests/rcutorture/bin/kvm-recheck-rcu.sh | 18 ++ .../selftests/rcutorture/bin/kvm-test-1-run.sh | 9 +- .../selftests/rcutorture/bin/parse-build.sh | 20 +- .../selftests/rcutorture/bin/parse-console.sh | 2 +- 48 files changed, 657 insertions(+), 586 deletions(-)
Documentation/RCU/stallwarn.txt +15 -1
@@ -15,7 +15,7 @@ CONFIG_RCU_CPU_STALL_TIMEOUT 21 seconds. This configuration parameter may be changed at runtime via the - /sys/module/rcutree/parameters/rcu_cpu_stall_timeout, however + /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout, however this parameter is checked only at the beginning of a cycle. So if you are 10 seconds into a 40-second stall, setting this sysfs parameter to (say) five will shorten the timeout for the
@@ -152,6 +152,15 @@ no non-lazy callbacks ("." is printed otherwise, as shown above) and "D" indicates that dyntick-idle processing is enabled ("." is printed otherwise, for example, if disabled via the "nohz=" kernel boot parameter). +If the relevant grace-period kthread has been unable to run prior to +the stall warning, the following additional line is printed: + + rcu_preempt kthread starved for 2023 jiffies! + +Starving the grace-period kthreads of CPU time can of course result in +RCU CPU stall warnings even when all CPUs and tasks have passed through +the required quiescent states. + Multiple Warnings From One Stall
@@ -187,6 +196,11 @@ o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the behavior, you might need to replace some of the cond_resched() calls with calls to cond_resched_rcu_qs(). +o Anything that prevents RCU's grace-period kthreads from running. + This can result in the "All QSes seen" console-log message. + This message will include information on when the kthread last + ran and how often it should be expected to run. + o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might happen to preempt a low-priority task in the middle of an RCU read-side critical section. This is especially damaging if
Documentation/RCU/trace.txt +16 -16
@@ -56,14 +56,14 @@ Within each flavor directory (rcu/rcu_bh, rcu/rcu_sched, and possibly The output of "cat rcu/rcu_preempt/rcudata" looks as follows: - 0!c=30455 g=30456 pq=1 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716 - 1!c=30719 g=30720 pq=1 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982 - 2!c=30150 g=30151 pq=1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458 - 3 c=31249 g=31250 pq=1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622 - 4!c=29502 g=29503 pq=1 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521 - 5 c=31201 g=31202 pq=1 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698 - 6!c=30253 g=30254 pq=1 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353 - 7 c=31178 g=31178 pq=1 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969 + 0!c=30455 g=30456 pq=1/0 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716 + 1!c=30719 g=30720 pq=1/0 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982 + 2!c=30150 g=30151 pq=1/1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458 + 3 c=31249 g=31250 pq=1/1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622 + 4!c=29502 g=29503 pq=1/0 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521 + 5 c=31201 g=31202 pq=1/0 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698 + 6!c=30253 g=30254 pq=1/0 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353 + 7 c=31178 g=31178 pq=1/0 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969 This file has one line per CPU, or eight for this 8-CPU system. The fields are as follows:
@@ -188,14 +188,14 @@ o "ca" is the number of RCU callbacks that have been adopted by this Kernels compiled with CONFIG_RCU_BOOST=y display the following from /debug/rcu/rcu_preempt/rcudata: - 0!c=12865 g=12866 pq=1 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871 - 1 c=14407 g=14408 pq=1 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485 - 2 c=14407 g=14408 pq=1 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490 - 3 c=14407 g=14408 pq=1 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290 - 4 c=14405 g=14406 pq=1 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114 - 5!c=14168 g=14169 pq=1 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722 - 6 c=14404 g=14405 pq=1 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811 - 7 c=14407 g=14408 pq=1 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042 + 0!c=12865 g=12866 pq=1/0 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871 + 1 c=14407 g=14408 pq=1/0 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485 + 2 c=14407 g=14408 pq=1/0 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490 + 3 c=14407 g=14408 pq=1/0 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290 + 4 c=14405 g=14406 pq=1/0 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114 + 5!c=14168 g=14169 pq=1/0 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722 + 6 c=14404 g=14405 pq=1/0 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811 + 7 c=14407 g=14408 pq=1/0 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042 This is similar to the output discussed above, but contains the following additional fields:
Documentation/memory-barriers.txt +45 -1
58 lines not on this page · raw message
arch/arm/kvm/Kconfig +1 -0
@@ -23,6 +23,7 @@ config KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_ARM_HOST + select SRCU depends on ARM_VIRT_EXT && ARM_LPAE ---help--- Support hosting virtualized guest machines. You will also
arch/arm64/kvm/Kconfig +1 -0
@@ -26,6 +26,7 @@ config KVM select KVM_ARM_HOST select KVM_ARM_VGIC select KVM_ARM_TIMER + select SRCU ---help--- Support hosting virtualized guest machines.
arch/mips/kvm/Kconfig +1 -0
@@ -20,6 +20,7 @@ config KVM select PREEMPT_NOTIFIERS select ANON_INODES select KVM_MMIO + select SRCU ---help--- Support for hosting Guest kernels. Currently supported on MIPS32 processors.
arch/powerpc/kvm/Kconfig +1 -0
@@ -21,6 +21,7 @@ config KVM select PREEMPT_NOTIFIERS select ANON_INODES select HAVE_KVM_EVENTFD + select SRCU config KVM_BOOK3S_HANDLER bool
arch/s390/kvm/Kconfig +1 -0
@@ -28,6 +28,7 @@ config KVM select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQFD select HAVE_KVM_IRQ_ROUTING + select SRCU ---help--- Support hosting paravirtualized guest machines using the SIE virtualization capability on the mainframe. This should work
arch/tile/kvm/Kconfig +1 -0
@@ -21,6 +21,7 @@ config KVM depends on HAVE_KVM && MODULES select PREEMPT_NOTIFIERS select ANON_INODES + select SRCU ---help--- Support hosting paravirtualized guest machines.
arch/x86/Kconfig +1 -0
@@ -138,6 +138,7 @@ config X86 select HAVE_ACPI_APEI_NMI if ACPI select ACPI_LEGACY_TABLES_LOOKUP if ACPI select X86_FEATURE_NAMES if PROC_FS + select SRCU config INSTRUCTION_DECODER def_bool y
arch/x86/kvm/Kconfig +1 -0
@@ -40,6 +40,7 @@ config KVM select HAVE_KVM_MSI select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_VFIO + select SRCU ---help--- Support hosting fully virtualized guest machines using hardware virtualization extensions. You will need a fairly recent
drivers/clk/Kconfig +1 -0
@@ -13,6 +13,7 @@ config COMMON_CLK bool select HAVE_CLK_PREPARE select CLKDEV_LOOKUP + select SRCU ---help--- The common clock framework is a single definition of struct clk, useful across many platforms, as well as an
drivers/cpufreq/Kconfig +1 -0
@@ -2,6 +2,7 @@ menu "CPU Frequency scaling" config CPU_FREQ bool "CPU Frequency scaling" + select SRCU help CPU Frequency scaling allows you to change the clock speed of CPUs on the fly. This is a nice method to save power, because
drivers/devfreq/Kconfig +1 -0
@@ -1,5 +1,6 @@ menuconfig PM_DEVFREQ bool "Generic Dynamic Voltage and Frequency Scaling (DVFS) support" + select SRCU help A device may have a list of frequencies and voltages available. devfreq, a generic DVFS framework can be registered for a device
drivers/md/Kconfig +1 -0
@@ -5,6 +5,7 @@ menuconfig MD bool "Multiple devices driver support (RAID and LVM)" depends on BLOCK + select SRCU help Support multiple physical spindles through a single logical device. Required for RAID and logical volume management.
drivers/net/Kconfig +1 -0
@@ -197,6 +197,7 @@ config NETCONSOLE_DYNAMIC config NETPOLL def_bool NETCONSOLE + select SRCU config NET_POLL_CONTROLLER def_bool NETPOLL
fs/btrfs/Kconfig +1 -0
@@ -8,6 +8,7 @@ config BTRFS_FS select LZO_DECOMPRESS select RAID6_PQ select XOR_BLOCKS + select SRCU help Btrfs is a general purpose copy-on-write filesystem with extents,
fs/notify/Kconfig +1 -0
@@ -1,5 +1,6 @@ config FSNOTIFY def_bool n + select SRCU source "fs/notify/dnotify/Kconfig" source "fs/notify/inotify/Kconfig"
fs/quota/Kconfig +1 -0
@@ -5,6 +5,7 @@ config QUOTA bool "Quota support" select QUOTACTL + select SRCU help If you say Y here, you will be able to set per user limits for disk usage (also called disk quotas). Currently, it works for the
include/linux/compiler.h +1 -1
@@ -385,7 +385,7 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s/* Is this type a native word size -- useful for atomic operations */ #ifndef __native_word -# define __native_word (t) (sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long)) +# define __native_word (t) (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long))#endif /* Compile time object size, -1 for unknown */
include/linux/rculist.h +8 -8
@@ -524,11 +524,11 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* @member: the name of the hlist_node within the struct. */ #define hlist_for_each_entry_continue_rcu (pos, member) \ - for (pos = hlist_entry_safe(rcu_dereference((pos)->member.next),\ - typeof(*(pos)), member); \ + for (pos = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu( \ + &(pos)->member)), typeof(*(pos)), member); \ pos; \ - pos = hlist_entry_safe(rcu_dereference((pos)->member.next),\ - typeof(*(pos)), member)) + pos = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu( \ + &(pos)->member)), typeof(*(pos)), member))/** * hlist_for_each_entry_continue_rcu_bh - iterate over a hlist continuing after current point
@@ -536,11 +536,11 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* @member: the name of the hlist_node within the struct. */ #define hlist_for_each_entry_continue_rcu_bh (pos, member) \ - for (pos = hlist_entry_safe(rcu_dereference_bh((pos)->member.next),\ - typeof(*(pos)), member); \ + for (pos = hlist_entry_safe(rcu_dereference_bh(hlist_next_rcu( \ + &(pos)->member)), typeof(*(pos)), member); \ pos; \ - pos = hlist_entry_safe(rcu_dereference_bh((pos)->member.next),\ - typeof(*(pos)), member)) + pos = hlist_entry_safe(rcu_dereference_bh(hlist_next_rcu( \ + &(pos)->member)), typeof(*(pos)), member))/** * hlist_for_each_entry_from_rcu - iterate over a hlist continuing from current point
include/linux/rcupdate.h +7 -6
@@ -331,12 +331,13 @@ static inline void rcu_init_nohz(void)extern struct srcu_struct tasks_rcu_exit_srcu;#define rcu_note_voluntary_context_switch (t) \ do { \ + rcu_all_qs(); \ if (ACCESS_ONCE((t)->rcu_tasks_holdout)) \ ACCESS_ONCE((t)->rcu_tasks_holdout) = false; \ } while (0)#else /* #ifdef CONFIG_TASKS_RCU */ #define TASKS_RCU (x) do { } while (0) -#define rcu_note_voluntary_context_switch (t) do { } while (0) +#define rcu_note_voluntary_context_switch (t) rcu_all_qs()#endif /* #else #ifdef CONFIG_TASKS_RCU */ /**
@@ -582,11 +583,11 @@ static inline void rcu_preempt_sleep_check(void) })#define __rcu_dereference_check (p, c, space) \ ({ \ - typeof(*p) *_________p1 = (typeof(*p) *__force)ACCESS_ONCE(p); \ +/* Dependency order vs. p above. */ \ + typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \rcu_lockdep_assert (c,"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ - smp_read_barrier_depends();/* Dependency order vs. p above. */ \ - ((typeof(*p) __force __kernel *)(_________p1)); \ + ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \ ({ \
@@ -603,10 +604,10 @@ static inline void rcu_preempt_sleep_check(void) })#define __rcu_dereference_index_check (p, c) \ ({ \ - typeof(p) _________p1 = ACCESS_ONCE(p); \ +/* Dependency order vs. p above. */ \ +typeof (p) _________p1 =lockless_dereference (p); \rcu_lockdep_assert (c, \"suspicious rcu_dereference_index_check() usage" ); \ - smp_read_barrier_depends();/* Dependency order vs. p above. */ \ (_________p1); \ })
include/linux/rcutiny.h +40 -5
64 lines not on this page · raw message
include/linux/rcutree.h +8 -3
@@ -81,9 +81,12 @@ void cond_synchronize_rcu(unsigned long oldstate);extern unsigned long rcutorture_testseq;extern unsigned long rcutorture_vernum; -long rcu_batches_completed (void ); -long rcu_batches_completed_bh (void ); -long rcu_batches_completed_sched (void ); +unsigned long rcu_batches_started (void ); +unsigned long rcu_batches_started_bh (void ); +unsigned long rcu_batches_started_sched (void ); +unsigned long rcu_batches_completed (void ); +unsigned long rcu_batches_completed_bh (void ); +unsigned long rcu_batches_completed_sched (void );void show_rcu_gp_kthreads (void );void rcu_force_quiescent_state (void );
@@ -97,4 +100,6 @@ extern int rcu_scheduler_active __read_mostly;bool rcu_is_watching (void ); +void rcu_all_qs (void ); +#endif /* __LINUX_RCUTREE_H */
include/linux/srcu.h +6 -8
@@ -45,7 +45,7 @@ struct rcu_batch {#define RCU_BATCH_INIT (name) { NULL, &(name.head) }struct srcu_struct { -unsigned completed; +unsigned long completed;struct srcu_struct_array __percpu *per_cpu_ref;spinlock_t queue_lock;/* protect ->batch_queue, ->running */ bool running;
@@ -102,13 +102,11 @@ void process_srcu(struct work_struct *work);* define and init a srcu struct at build time. * dont't call init_srcu_struct() nor cleanup_srcu_struct() on it. */ -#define DEFINE_SRCU (name) \ +#define __DEFINE_SRCU (name, is_static) \ static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\ - struct srcu_struct name = __SRCU_STRUCT_INIT(name); - -#define DEFINE_STATIC_SRCU (name) \ - static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\ - static struct srcu_struct name = __SRCU_STRUCT_INIT(name); + is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name) +#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) +#define DEFINE_STATIC_SRCU (name) __DEFINE_SRCU(name, static)/** * call_srcu() - Queue a callback for invocation after an SRCU grace period
@@ -135,7 +133,7 @@ int __srcu_read_lock(struct srcu_struct *sp) __acquires(sp); void __srcu_read_unlock(struct srcu_struct *sp, int idx) __releases(sp);void synchronize_srcu (struct srcu_struct *sp);void synchronize_srcu_expedited (struct srcu_struct *sp); -long srcu_batches_completed (struct srcu_struct *sp); +unsigned long srcu_batches_completed (struct srcu_struct *sp);void srcu_barrier (struct srcu_struct *sp);#ifdef CONFIG_DEBUG_LOCK_ALLOC
init/Kconfig +13 -5
@@ -470,7 +470,6 @@ choice config TREE_RCU bool "Tree-based hierarchical RCU" depends on !PREEMPT && SMP - select IRQ_WORK help This option selects the RCU implementation that is designed for very large SMP system with hundreds or
@@ -480,7 +479,6 @@ config TREE_RCU config PREEMPT_RCU bool "Preemptible tree-based hierarchical RCU" depends on PREEMPT - select IRQ_WORK help This option selects the RCU implementation that is designed for very large SMP systems with hundreds or
@@ -501,9 +499,17 @@ config TINY_RCU endchoice +config SRCU + bool + help + This option selects the sleepable version of RCU. This version + permits arbitrary sleeping or blocking within RCU read-side critical + sections. + config TASKS_RCU bool "Task_based RCU implementation using voluntary context switch" default n + select SRCU help This option enables a task-based RCU implementation that uses only voluntary context switch (not preemption!), idle, and
@@ -668,9 +674,10 @@ config RCU_BOOST config RCU_KTHREAD_PRIO int "Real-time priority to use for RCU worker threads" - range 1 99 - depends on RCU_BOOST - default 1 + range 1 99 if RCU_BOOST + range 0 99 if !RCU_BOOST + default 1 if RCU_BOOST + default 0 if !RCU_BOOST help This option specifies the SCHED_FIFO priority value that will be assigned to the rcuc/n and rcub/n threads and is also the value
@@ -1595,6 +1602,7 @@ config PERF_EVENTS depends on HAVE_PERF_EVENTS select ANON_INODES select IRQ_WORK + select SRCU help Enable kernel support for various performance events provided by software and hardware.
kernel/cpu.c +23 -33
114 lines not on this page · raw message
kernel/notifier.c +3 -0
@@ -402,6 +402,7 @@ int raw_notifier_call_chain(struct raw_notifier_head *nh, }EXPORT_SYMBOL_GPL (raw_notifier_call_chain); +#ifdef CONFIG_SRCU /* * SRCU notifier chain routines. Registration and unregistration * use a mutex, and call_chain is synchronized by SRCU (no locks).
@@ -528,6 +529,8 @@ void srcu_init_notifier_head(struct srcu_notifier_head *nh) }EXPORT_SYMBOL_GPL (srcu_init_notifier_head); +#endif /* CONFIG_SRCU */ +static ATOMIC_NOTIFIER_HEAD (die_chain); int notrace notify_die(enum die_val val, const char *str,
kernel/power/Kconfig +1 -0
@@ -251,6 +251,7 @@ config APM_EMULATION config PM_OPP bool + select SRCU ---help--- SOCs have a standard set of tuples consisting of frequency and voltage pairs that the device will support per voltage domain. This
kernel/rcu/Makefile +2 -1
@@ -1,4 +1,5 @@ -obj-y += update.o srcu.o +obj-y += update.o +obj-$(CONFIG_SRCU) += srcu.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o obj-$(CONFIG_TREE_RCU) += tree.o obj-$(CONFIG_PREEMPT_RCU) += tree.o
kernel/rcu/rcu.h +6 -0
@@ -137,4 +137,10 @@ int rcu_jiffies_till_stall_check(void);void rcu_early_boot_tests (void ); +/* +* This function really isn't for public consumption, but RCU is special in +* that context switches can allow the state machine to make progress. +*/ +extern void resched_cpu (int cpu); +#endif /* __LINUX_RCU_H */
kernel/rcu/rcutorture.c +38 -28
184 lines not on this page · raw message
kernel/rcu/srcu.c +1 -1
@@ -546,7 +546,7 @@ EXPORT_SYMBOL_GPL(srcu_barrier);* Report the number of batches, correlated with, but not necessarily * precisely the same as, the number of grace periods that have elapsed. */ -long srcu_batches_completed (struct srcu_struct *sp) +unsigned long srcu_batches_completed (struct srcu_struct *sp) {return sp->completed; }
kernel/rcu/tiny.c +10 -103
176 lines not on this page · raw message
kernel/rcu/tiny_plugin.h +4 -5
@@ -145,17 +145,16 @@ static void check_cpu_stall(struct rcu_ctrlblk *rcp) rcp->ticks_this_gp++; j = jiffies; js =ACCESS_ONCE (rcp->jiffies_stall); -if (*rcp->curtail &&ULONG_CMP_GE (j, js)) { +if (rcp->rcucblist &&ULONG_CMP_GE (j, js)) {pr_err ("INFO: %s stall on CPU (%lu ticks this GP) idle=%llx (t=%lu jiffies q=%ld)\n" , - rcp->name, rcp->ticks_this_gp, rcu_dynticks_nesting, + rcp->name, rcp->ticks_this_gp,DYNTICK_TASK_EXIT_IDLE , jiffies - rcp->gp_start, rcp->qlen);dump_stack (); - } -if (*rcp->curtail &&ULONG_CMP_GE (j, js))ACCESS_ONCE (rcp->jiffies_stall) = jiffies +3 *rcu_jiffies_till_stall_check () +3 ; -else if (ULONG_CMP_GE (j, js)) + }else if (ULONG_CMP_GE (j, js)) {ACCESS_ONCE (rcp->jiffies_stall) = jiffies +rcu_jiffies_till_stall_check (); + } }static void reset_cpu_stall_ticks (struct rcu_ctrlblk *rcp)
kernel/rcu/tree.c +258 -97
638 lines not on this page · raw message
kernel/rcu/tree.h +15 -47
138 lines not on this page · raw message
kernel/rcu/tree_plugin.h +72 -199
470 lines not on this page · raw message
kernel/rcu/tree_trace.c +6 -2
@@ -46,6 +46,8 @@#define RCU_TREE_NONCORE #include "tree.h" +DECLARE_PER_CPU_SHARED_ALIGNED(unsigned long, rcu_qs_ctr); +static int r_open (struct inode *inode,struct file *file,const struct seq_operations *op) {
@@ -115,11 +117,13 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp)if (!rdp->beenonline)return ; -seq_printf (m,"%3d%cc=%ld g=%ld pq=%d qp=%d" , +seq_printf (m,"%3d%cc=%ld g=%ld pq=%d/%d qp=%d" , rdp->cpu,cpu_is_offline (rdp->cpu) ?'!' :' ' ,ulong2long (rdp->completed),ulong2long (rdp->gpnum), - rdp->passed_quiesce, rdp->qs_pending); + rdp->passed_quiesce, + rdp->rcu_qs_ctr_snap ==per_cpu (rcu_qs_ctr, rdp->cpu), + rdp->qs_pending); seq_printf(m, " dt=%d/%llx/%d df=%lu", atomic_read(&rdp->dynticks->dynticks), rdp->dynticks->dynticks_nesting,
kernel/softirq.c +1 -2
@@ -656,9 +656,8 @@ static void run_ksoftirqd(unsigned int cpu)* in the task stack here. */ __do_softirq (); -rcu_note_context_switch ();local_irq_enable (); -cond_resched (); +cond_resched_rcu_qs ();return ; }local_irq_enable ();
lib/Kconfig.debug +2 -1
@@ -1215,6 +1215,7 @@ config RCU_TORTURE_TEST tristate "torture tests for RCU" depends on DEBUG_KERNEL select TORTURE_TEST + select SRCU default n help This option provides a kernel module that runs torture tests
@@ -1257,7 +1258,7 @@ config RCU_CPU_STALL_TIMEOUT config RCU_CPU_STALL_INFO bool "Print additional diagnostics on RCU CPU stall" depends on (TREE_RCU || PREEMPT_RCU) && DEBUG_KERNEL - default n + default y help For each stalled CPU that is aware of the current RCU grace period, print out additional per-CPU diagnostic information
mm/Kconfig +1 -0
@@ -325,6 +325,7 @@ config VIRT_TO_BUS config MMU_NOTIFIER bool + select SRCU config KSM bool "Enable KSM for page merging"
security/tomoyo/Kconfig +1 -0
@@ -5,6 +5,7 @@ config SECURITY_TOMOYO select SECURITYFS select SECURITY_PATH select SECURITY_NETWORK + select SRCU default n help This selects TOMOYO Linux, pathname-based access control.
tools/testing/selftests/rcutorture/bin/cpus2use.sh +1 -1
@@ -24,7 +24,7 @@ ncpus=`grep '^processor' /proc/cpuinfo | wc -l` idlecpus=`mpstat | tail -1 | \ - awk -v ncpus=$ncpus '{ print ncpus * ($7 + $12) / 100 }'` + awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'` awk -v ncpus=$ncpus -v idlecpus=$idlecpus < /dev/null ' BEGIN { cpus2use = idlecpus;
tools/testing/selftests/rcutorture/bin/kvm-recheck-rcu.sh +18 -0
@@ -30,6 +30,7 @@ else echo Unreadable results directory: $i exit 1 fi +. tools/testing/selftests/rcutorture/bin/functions.sh configfile=`echo $i | sed -e 's/^.*\///'` ngps=`grep ver: $i/console.log 2> /dev/null | tail -1 | sed -e 's/^.* ver: //' -e 's/ .*$//'`
@@ -48,4 +49,21 @@ else title="$title ($ngpsps per second)" fi echo $title + nclosecalls=`grep --binary-files=text 'torture: Reader Batch' $i/console.log | tail -1 | awk '{for (i=NF-8;i<=NF;i++) sum+=$i; } END {print sum}'` + if test -z "$nclosecalls" + then + exit 0 + fi + if test "$nclosecalls" -eq 0 + then + exit 0 + fi + # Compute number of close calls per tenth of an hour + nclosecalls10=`awk -v nclosecalls=$nclosecalls -v dur=$dur 'BEGIN { print int(nclosecalls * 36000 / dur) }' < /dev/null` + if test $nclosecalls10 -gt 5 -a $nclosecalls -gt 1 + then + print_bug $nclosecalls "Reader Batch close calls in" $(($dur/60)) minute run: $i + else + print_warning $nclosecalls "Reader Batch close calls in" $(($dur/60)) minute run: $i + fi fi
tools/testing/selftests/rcutorture/bin/kvm-test-1-run.sh +5 -4
@@ -8,9 +8,9 @@ # # Usage: kvm-test-1-run.sh config builddir resdir minutes qemu-args boot_args # -# qemu-args defaults to "-nographic", along with arguments specifying the -# number of CPUs and other options generated from -# the underlying CPU architecture. +# qemu-args defaults to "-enable-kvm -soundhw pcspk -nographic", along with +# arguments specifying the number of CPUs and other +# options generated from the underlying CPU architecture. # boot_args defaults to value returned by the per_version_boot_params # shell function. #
@@ -138,7 +138,7 @@ then fi # Generate -smp qemu argument. -qemu_args="-nographic $qemu_args" +qemu_args="-enable-kvm -soundhw pcspk -nographic $qemu_args" cpu_count=`configNR_CPUS.sh $config_template` cpu_count=`configfrag_boot_cpus "$boot_args" "$config_template" "$cpu_count"` vcpus=`identify_qemu_vcpus`
@@ -168,6 +168,7 @@ then touch $resdir/buildonly exit 0 fi +echo "NOTE: $QEMU either did not run or was interactive" > $builddir/console.log echo $QEMU $qemu_args -m 512 -kernel $resdir/bzImage -append \"$qemu_append $boot_args\" > $resdir/qemu-cmd ( $QEMU $qemu_args -m 512 -kernel $resdir/bzImage -append "$qemu_append $boot_args"; echo $? > $resdir/qemu-retval ) & qemu_pid=$!
tools/testing/selftests/rcutorture/bin/parse-build.sh +13 -7
@@ -26,12 +26,15 @@ # # Authors: Paul E. McKenney <[email protected]> -T=$1 +F=$1 title=$2 +T=/tmp/parse-build.sh.$$ +trap 'rm -rf $T' 0 +mkdir $T . functions.sh -if grep -q CC < $T +if grep -q CC < $F then : else
@@ -39,18 +42,21 @@ else exit 1 fi -if grep -q "error:" < $T +if grep -q "error:" < $F then print_bug $title build errors: - grep "error:" < $T + grep "error:" < $F exit 2 fi -exit 0 -if egrep -q "rcu[^/]*\.c.*warning:|rcu.*\.h.*warning:" < $T +grep warning: < $F > $T/warnings +grep "include/linux/*rcu*\.h:" $T/warnings > $T/hwarnings +grep "kernel/rcu/[^/]*:" $T/warnings > $T/cwarnings +cat $T/hwarnings $T/cwarnings > $T/rcuwarnings +if test -s $T/rcuwarnings then print_warning $title build errors: - egrep "rcu[^/]*\.c.*warning:|rcu.*\.h.*warning:" < $T + cat $T/rcuwarnings exit 2 fi exit 0
tools/testing/selftests/rcutorture/bin/parse-console.sh +1 -1
@@ -36,7 +36,7 @@ if grep -Pq '\x00' < $file then print_warning Console output contains nul bytes, old qemu still running? fi -egrep 'Badness|WARNING:|Warn|BUG|===========|Call Trace:|Oops:' < $file | grep -v 'ODEBUG: ' | grep -v 'Warning: unable to open an initial console' > $T +egrep 'Badness|WARNING:|Warn|BUG|===========|Call Trace:|Oops:|Stall ended before state dump start' < $file | grep -v 'ODEBUG: ' | grep -v 'Warning: unable to open an initial console' > $T if test -s $T then print_warning Assertion failure in $file $title