[PATCH tip/core/rcu 0/15]
Subsystems (14)
Paul E. McKenney
maintainercover [PATCH tip/core/rcu 0/15]
Hello!
This is a repost of Thomas Gleixner's "Make preempt count unconditional" series [1], but with the addition of a kvfree_rcu() bug-fix patch making use of this PREEMPT_COUNT addition. This series is intended for the upcoming v5.10 merge window.
This addition fixes the -rt bug reported by Sebastian Andrzej Siewior [2]. Please note that with the advent of the new lockdep Kconfig option CONFIG_PROVE_RAW_LOCK_NESTING, this is now also a mainline bug. In happy contrast to the surprisingly large number of earlier versions of this fix, this version uses only pre-existing kernel interfaces, and furthermore uses them in conventional ways.
1-13. Thomas's series [1].
14. Use the newly accurate preemptible() macro to cause kvfree_rcu()
to do its allocations only when it is safe to do so, even in CONFIG_PREEMPT_NONE=y kernels, courtesy of Uladzislau Rezki. Again, this fixes the bug reported by Sebastian [2].
15. Checkpatch fix removing NULL check guarding kfree(), courtesy
of kernel test robot and Julia Lawall.
In addition, this series reduces the size of the kernel by 100 lines of code, about two thirds of which is from Thomas's original series and the remaining one third from the bug fix.
Changes from v1 [1]:
o Fix trivial !SMP build failure.
o Apply checkpatch spelling suggestions.
Thanx, Paul
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
------------------------------------------------------------------------
arch/arm/include/asm/assembler.h | 11 - arch/arm/kernel/iwmmxt.S | 2 arch/arm/mach-ep93xx/crunch-bits.S | 2 arch/xtensa/kernel/entry.S | 2 drivers/gpu/drm/i915/Kconfig.debug | 1 drivers/gpu/drm/i915/i915_utils.h | 3 include/linux/bit_spinlock.h | 4 include/linux/lockdep.h | 6 include/linux/pagemap.h | 4 include/linux/preempt.h | 37 ----- include/linux/uaccess.h | 6 kernel/Kconfig.preempt | 6 kernel/rcu/tree.c | 73 ++-------- kernel/sched/core.c | 6 lib/Kconfig.debug | 5 tools/testing/selftests/rcutorture/configs/rcu/SRCU-t | 1 tools/testing/selftests/rcutorture/configs/rcu/SRCU-u | 1 tools/testing/selftests/rcutorture/configs/rcu/TINY01 | 1 tools/testing/selftests/rcutorture/doc/TINY_RCU.txt | 5 tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt | 1 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h | 1 21 files changed, 39 insertions(+), 139 deletions(-)
[email protected]
maintainerPATCH 1/15 lib/debug: Remove pointless ARCH_NO_PREEMPT dependencies
From: Thomas Gleixner <[email protected]>
ARCH_NO_PREEMPT disables the selection of CONFIG_PREEMPT_VOLUNTARY and CONFIG_PREEMPT, but architectures which set this config option still support preempt count for hard and softirq accounting.
There is absolutely no reason to prevent lockdep from using the preempt counter nor is there a reason to prevent the enablement of CONFIG_DEBUG_ATOMIC_SLEEP on such architectures.
Remove the dependencies.
--- lib/Kconfig.debug | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/Kconfig.debug +1 -2
@@ -1161,7 +1161,7 @@ config PROVE_LOCKING select DEBUG_RWSEMS select DEBUG_WW_MUTEX_SLOWPATH select DEBUG_LOCK_ALLOC - select PREEMPT_COUNT if !ARCH_NO_PREEMPT + select PREEMPT_COUNT select TRACE_IRQFLAGS default n help
@@ -1323,7 +1323,6 @@ config DEBUG_ATOMIC_SLEEP bool "Sleep inside atomic section checking" select PREEMPT_COUNT depends on DEBUG_KERNEL - depends on !ARCH_NO_PREEMPT help If you say Y here, various routines which may sleep will become very noisy if they are called inside atomic sections: when a spinlock is
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 2/15 preempt: Make preempt count unconditional
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
The handling of preempt_count() is inconsistent across kernel configurations. On kernels which have PREEMPT_COUNT=n preempt_disable/enable() and the lock/unlock functions are not affecting the preempt count, only local_bh_disable/enable() and _bh variants of locking, soft interrupt delivery, hard interrupt and NMI context affect it.
It's therefore impossible to have a consistent set of checks which provide information about the context in which a function is called. In many cases it makes sense to have separate functions for separate contexts, but there are valid reasons to avoid that and handle different calling contexts conditionally.
The lack of such indicators which work on all kernel configuratios is a constant source of trouble because developers either do not understand the implications or try to work around this inconsistency in weird ways. Neither seem these issues be catched by reviewers and testing.
Recently merged code does:
gfp = preemptible() ? GFP_KERNEL : GFP_ATOMIC;
Looks obviously correct, except for the fact that preemptible() is unconditionally false for CONFIF_PREEMPT_COUNT=n, i.e. all allocations in that code use GFP_ATOMIC on such kernels.
Attempts to make preempt count unconditional and consistent have been rejected in the past with handwaving performance arguments.
Freshly conducted benchmarks did not reveal any measurable impact from enabling preempt count unconditionally. On kernels with CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY the preempt count is only incremented and decremented but the result of the decrement is not tested. Contrary to that enabling CONFIG_PREEMPT which tests the result has a small but measurable impact due to the conditional branch/call.
It's about time to make essential functionality of the kernel consistent across the various preemption models.
Enable CONFIG_PREEMPT_COUNT unconditionally. Follow up changes will remove the #ifdeffery and remove the config option at the end.
--- kernel/Kconfig.preempt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/Kconfig.preempt +1 -2
@@ -75,8 +75,7 @@ config PREEMPT_RT endchoice config PREEMPT_COUNT - bool + def_bool y config PREEMPTION bool - select PREEMPT_COUNT
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 3/15 preempt: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- include/linux/preempt.h | 37 ++++--------------------------------- 1 file changed, 4 insertions(+), 33 deletions(-)
include/linux/preempt.h +4 -33
@@ -56,8 +56,7 @@#define PREEMPT_DISABLED (PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)/* -* Disable preemption until the scheduler is running -- use an unconditional -* value so that it also works on !PREEMPT_COUNT kernels. +* Disable preemption until the scheduler is running. * * Reset by start_kernel()->sched_init()->init_idle()->init_idle_preempt_count(). */
@@ -69,7 +68,6 @@* * preempt_count() == 2*PREEMPT_DISABLE_OFFSET * -* Note: PREEMPT_DISABLE_OFFSET is 0 for !PREEMPT_COUNT kernels. * Note: See finish_task_switch(). */ #define FORK_PREEMPT_COUNT (2*PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)
@@ -106,11 +104,7 @@/* * The preempt_count offset after preempt_disable(); */ -#if defined(CONFIG_PREEMPT_COUNT ) -# define PREEMPT_DISABLE_OFFSET PREEMPT_OFFSET -#else -# define PREEMPT_DISABLE_OFFSET 0 -#endif +#define PREEMPT_DISABLE_OFFSET PREEMPT_OFFSET/* * The preempt_count offset after spin_lock()
@@ -122,8 +116,8 @@* * spin_lock_bh() * -* Which need to disable both preemption (CONFIG_PREEMPT_COUNT) and -* softirqs, such that unlock sequences of: +* Which need to disable both preemption and softirqs, such that unlock +* sequences of: * * spin_unlock(); * local_bh_enable();
@@ -164,8 +158,6 @@ extern void preempt_count_sub(int val);#define preempt_count_inc () preempt_count_add(1)#define preempt_count_dec () preempt_count_sub(1) -#ifdef CONFIG_PREEMPT_COUNT -#define preempt_disable () \ do { \ preempt_count_inc(); \
@@ -231,27 +223,6 @@ do { \__preempt_count_dec (); \ }while (0 ) -#else /* !CONFIG_PREEMPT_COUNT */ - -/* -* Even if we don't have any preemption, we need preempt disable/enable -* to be barriers, so that we don't have things like get_user/put_user -* that can cause faults and scheduling migrate into our preempt-protected -* region. -*/ -#define preempt_disable () barrier() -#define sched_preempt_enable_no_resched () barrier() -#define preempt_enable_no_resched () barrier() -#define preempt_enable () barrier() -#define preempt_check_resched () do { } while (0) - -#define preempt_disable_notrace () barrier() -#define preempt_enable_no_resched_notrace () barrier() -#define preempt_enable_notrace () barrier() -#define preemptible () 0 - -#endif /* CONFIG_PREEMPT_COUNT */ - #ifdef MODULE/* * Modules have no business playing preemption tricks.
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 4/15 lockdep: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- include/linux/lockdep.h | 6 ++---- lib/Kconfig.debug | 1 - 2 files changed, 2 insertions(+), 5 deletions(-)
include/linux/lockdep.h +2 -4
@@ -560,16 +560,14 @@ do { \#define lockdep_assert_preemption_enabled () \ do { \ - WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT) && \ - debug_locks && \ + WARN_ON_ONCE(debug_locks && \ (preempt_count() != 0 || \ !raw_cpu_read(hardirqs_enabled))); \ } while (0)#define lockdep_assert_preemption_disabled () \ do { \ - WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT) && \ - debug_locks && \ + WARN_ON_ONCE(debug_locks && \ (preempt_count() == 0 && \ raw_cpu_read(hardirqs_enabled))); \ } while (0)
lib/Kconfig.debug +0 -1
@@ -1161,7 +1161,6 @@ config PROVE_LOCKING select DEBUG_RWSEMS select DEBUG_WW_MUTEX_SLOWPATH select DEBUG_LOCK_ALLOC - select PREEMPT_COUNT select TRACE_IRQFLAGS default n help
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 5/15 mm/pagemap: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- include/linux/pagemap.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
include/linux/pagemap.h +1 -3
@@ -168,9 +168,7 @@ void release_pages(struct page **pages, int nr);static inline int __page_cache_add_speculative (struct page *page,int count) {#ifdef CONFIG_TINY_RCU -# ifdef CONFIG_PREEMPT_COUNT -VM_BUG_ON (!in_atomic () && !irqs_disabled ()); -# endif +VM_BUG_ON (preemptible ());/* * Preempt must be disabled here - we rely on rcu_read_lock doing * this for us.
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 6/15 locking/bitspinlock: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- include/linux/bit_spinlock.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
include/linux/bit_spinlock.h +1 -3
@@ -90,10 +90,8 @@ static inline int bit_spin_is_locked(int bitnum, unsigned long *addr) {#if defined(CONFIG_SMP ) || defined(CONFIG_DEBUG_SPINLOCK )return test_bit (bitnum, addr); -#elif definedCONFIG_PREEMPT_COUNT -return preempt_count ();#else -return 1 ; +return preempt_count ();#endif }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 7/15 uaccess: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- include/linux/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/uaccess.h +3 -3
@@ -230,9 +230,9 @@ static inline bool pagefault_disabled(void)* * This function should only be used by the fault handlers. Other users should * stick to pagefault_disabled(). -* Please NEVER use preempt_disable() to disable the fault handler. With -* !CONFIG_PREEMPT_COUNT, this is like a NOP. So the handler won't be disabled. -* in_atomic() will report different values based on !CONFIG_PREEMPT_COUNT. +* +* Please NEVER use preempt_disable() or local_irq_disable() to disable the +* fault handler. */ #define faulthandler_disabled () (pagefault_disabled() || in_atomic())
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 8/15 sched: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- kernel/sched/core.c | 6 +----- lib/Kconfig.debug | 1 - 2 files changed, 1 insertion(+), 6 deletions(-)
kernel/sched/core.c +1 -5
@@ -3706,8 +3706,7 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev)* finish_task_switch() for details. * * finish_task_switch() will drop rq->lock() and lower preempt_count -* and the preempt_enable() will end up enabling preemption (on -* PREEMPT_COUNT kernels). +* and the preempt_enable() will end up enabling preemption. */ rq =finish_task_switch (prev);
@@ -7308,9 +7307,6 @@ void __cant_sleep(const char *file, int line, int preempt_offset)if (irqs_disabled ())return ; -if (!IS_ENABLED (CONFIG_PREEMPT_COUNT )) -return ; -if (preempt_count () > preempt_offset)return ;
lib/Kconfig.debug +0 -1
@@ -1320,7 +1320,6 @@ config DEBUG_LOCKDEP config DEBUG_ATOMIC_SLEEP bool "Sleep inside atomic section checking" - select PREEMPT_COUNT depends on DEBUG_KERNEL help If you say Y here, various routines which may sleep will become very
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 9/15 ARM: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- arch/arm/include/asm/assembler.h | 11 ----------- arch/arm/kernel/iwmmxt.S | 2 -- arch/arm/mach-ep93xx/crunch-bits.S | 2 -- 3 files changed, 15 deletions(-)
arch/arm/include/asm/assembler.h +0 -11
@@ -212,7 +212,6 @@/* * Increment/decrement the preempt count. */ -#ifdef CONFIG_PREEMPT_COUNT .macro inc_preempt_count, ti, tmp ldr \tmp, [\ti, #TI_PREEMPT] @ get preempt count add \tmp, \tmp, #1 @ increment it
@@ -229,16 +228,6 @@ get_thread_info \ti dec_preempt_count \ti, \tmp .endm -#else - .macro inc_preempt_count, ti, tmp - .endm - - .macro dec_preempt_count, ti, tmp - .endm - - .macro dec_preempt_count_ti, ti, tmp - .endm -#endif #define USERL(l, x...) \ 9999: x; \
arch/arm/kernel/iwmmxt.S +0 -2
@@ -94,9 +94,7 @@ ENTRY(iwmmxt_task_enable) mov r2, r2 @ cpwait bl concan_save -#ifdef CONFIG_PREEMPT_COUNT get_thread_info r10 -#endif 4: dec_preempt_count r10, r3 ret r9 @ normal exit from exception
arch/arm/mach-ep93xx/crunch-bits.S +0 -2
@@ -191,9 +191,7 @@ crunch_load: cfldr64 mvdx15, [r0, #CRUNCH_MVDX15] 1: -#ifdef CONFIG_PREEMPT_COUNT get_thread_info r10 -#endif 2: dec_preempt_count r10, r3 ret lr
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 10/15 xtensa: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- arch/xtensa/kernel/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/xtensa/kernel/entry.S +1 -1
@@ -819,7 +819,7 @@ ENTRY(debug_exception) * preemption if we have HW breakpoints to preserve DEBUGCAUSE.DBNUM * meaning. */ -#if defined(CONFIG_PREEMPT_COUNT) && defined(CONFIG_HAVE_HW_BREAKPOINT) +#ifdef CONFIG_HAVE_HW_BREAKPOINT GET_THREAD_INFO(a2, a1) l32i a3, a2, TI_PRE_COUNT addi a3, a3, 1
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 11/15 drm/i915: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- drivers/gpu/drm/i915/Kconfig.debug | 1 - drivers/gpu/drm/i915/i915_utils.h | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/gpu/drm/i915/Kconfig.debug +0 -1
@@ -20,7 +20,6 @@ config DRM_I915_DEBUG bool "Enable additional driver debugging" depends on DRM_I915 select DEBUG_FS - select PREEMPT_COUNT select I2C_CHARDEV select STACKDEPOT select DRM_DP_AUX_CHARDEV
drivers/gpu/drm/i915/i915_utils.h +1 -2
@@ -337,8 +337,7 @@ wait_remaining_ms_from_jiffies(unsigned long timestamp_jiffies, int to_wait_ms) (Wmax))#define wait_for (COND ,MS ) _wait_for((COND), (MS) * 1000, 10, 1000) -/* If CONFIG_PREEMPT_COUNT is disabled, in_atomic() always reports false. */ -#if defined(CONFIG_DRM_I915_DEBUG ) && defined(CONFIG_PREEMPT_COUNT ) +#ifdef CONFIG_DRM_I915_DEBUG # define _WAIT_FOR_ATOMIC_CHECK (ATOMIC ) WARN_ON_ONCE((ATOMIC) && !in_atomic())#else # define _WAIT_FOR_ATOMIC_CHECK (ATOMIC ) do { } while (0)
Signature · 6 lines
2.9.5 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH tip/core/rcu 11/15] drm/i915: Cleanup PREEMPT_COUNT leftovers
Quoting [email protected] (2020-09-29 02:30:58)
Quoted 4 lines
From: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNTis now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
Change looks fine:
Are you looking for us to merge or merge through another tree?
If us, did the base patch always enabling PREEMPT_COUNT go into 5.9 or is it heading to 5.10? We can queue this earliest for 5.11 as drm-next closed for 5.10 at week of -rc5.
Regards, Joonas
Quoted 43 lines
Signed-off-by Thomas Gleixner <[email protected]>Cc Jani Nikula <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Rodrigo Vivi <[email protected]>Cc David Airlie <[email protected]>Cc Daniel Vetter <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>--- drivers/gpu/drm/i915/Kconfig.debug | 1 - drivers/gpu/drm/i915/i915_utils.h | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-)drivers/gpu/drm/i915/Kconfig.debug +0
-1@@ -20,7 +20,6 @@ config DRM_I915_DEBUG bool "Enable additional driver debugging" depends on DRM_I915 select DEBUG_FS - select PREEMPT_COUNT select I2C_CHARDEV select STACKDEPOT select DRM_DP_AUX_CHARDEVdrivers/gpu/drm/i915/i915_utils.h +1
-2@@ -337,8 +337,7 @@ wait_remaining_ms_from_jiffies(unsigned long timestamp_jiffies, int to_wait_ms) (Wmax))#define wait_for (COND ,MS ) _wait_for((COND), (MS) * 1000, 10, 1000) -/* If CONFIG_PREEMPT_COUNT is disabled, in_atomic() always reports false. */ -#if defined(CONFIG_DRM_I915_DEBUG ) && defined(CONFIG_PREEMPT_COUNT ) +#ifdef CONFIG_DRM_I915_DEBUG # define _WAIT_FOR_ATOMIC_CHECK (ATOMIC ) WARN_ON_ONCE((ATOMIC) && !in_atomic())#else # define _WAIT_FOR_ATOMIC_CHECK (ATOMIC ) do { } while (0)Signature · 1 line
2.9.5
Re: [PATCH tip/core/rcu 11/15] drm/i915: Cleanup PREEMPT_COUNT leftovers
Quoted 6 lines · Joonas Lahtinen
Quoting [email protected] (2020-09-29 02:30:58)
Quoted 2 lines
CONFIG_PREEMPT_COUNTis now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.Change looks fine:
Quoted 7 lines
Reviewed-by Joonas Lahtinen <[email protected]>Are you looking for us to merge or merge through another tree?
If us, did the base patch always enabling
PREEMPT_COUNTgo into 5.9 or is it heading to 5.10? We can queue this earliest for 5.11 as drm-next closed for 5.10 at week of -rc5.
If at all it goes through rcu/tip because it depends on the earlier patches.
Thanks,
tglx
Re: [PATCH tip/core/rcu 11/15] drm/i915: Cleanup PREEMPT_COUNT leftovers
Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Joonas Lahtinen
Quoting [email protected] (2020-09-29 02:30:58)
Quoted 2 lines
CONFIG_PREEMPT_COUNTis now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.Change looks fine:
Reviewed-by Joonas Lahtinen <[email protected]>
Applied, thank you!
Quoted 7 lines
Quoted 5 lines
Are you looking for us to merge or merge through another tree?
If us, did the base patch always enabling
PREEMPT_COUNTgo into 5.9 or is it heading to 5.10? We can queue this earliest for 5.11 as drm-next closed for 5.10 at week of -rc5.If at all it goes through rcu/tip because it depends on the earlier patches.
I was figuring on sending a pull request later today, Pacific Time.
Thanx, Paul
[email protected]
maintainerPATCH 12/15 rcutorture: Cleanup PREEMPT_COUNT leftovers
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.
--- tools/testing/selftests/rcutorture/configs/rcu/SRCU-t | 1 - tools/testing/selftests/rcutorture/configs/rcu/SRCU-u | 1 - tools/testing/selftests/rcutorture/configs/rcu/TINY01 | 1 - tools/testing/selftests/rcutorture/doc/TINY_RCU.txt | 5 ++--- tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt | 1 - tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h | 1 - 6 files changed, 2 insertions(+), 8 deletions(-)
tools/testing/selftests/rcutorture/configs/rcu/SRCU-t +0 -1
@@ -7,4 +7,3 @@ CONFIG_RCU_TRACE=n CONFIG_DEBUG_LOCK_ALLOC=n CONFIG_DEBUG_OBJECTS_RCU_HEAD=n CONFIG_DEBUG_ATOMIC_SLEEP=y -#CHECK#CONFIG_PREEMPT_COUNT=y
tools/testing/selftests/rcutorture/configs/rcu/SRCU-u +0 -1
@@ -7,4 +7,3 @@ CONFIG_RCU_TRACE=n CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y CONFIG_DEBUG_OBJECTS_RCU_HEAD=n -CONFIG_PREEMPT_COUNT=n
tools/testing/selftests/rcutorture/configs/rcu/TINY01 +0 -1
@@ -10,4 +10,3 @@ CONFIG_RCU_TRACE=n #CHECK#CONFIG_RCU_STALL_COMMON=n CONFIG_DEBUG_LOCK_ALLOC=n CONFIG_DEBUG_OBJECTS_RCU_HEAD=n -CONFIG_PREEMPT_COUNT=n
tools/testing/selftests/rcutorture/doc/TINY_RCU.txt +2 -3
@@ -3,11 +3,10 @@ This document gives a brief rationale for the TINY_RCU test cases. Kconfig Parameters: -CONFIG_DEBUG_LOCK_ALLOC -- Do all three and none of the three. -CONFIG_PREEMPT_COUNT +CONFIG_DEBUG_LOCK_ALLOC -- Do both and none of the two. CONFIG_RCU_TRACE -The theory here is that randconfig testing will hit the other six possible +The theory here is that randconfig testing will hit the other two possible combinations of these parameters.
tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt +0 -1
@@ -43,7 +43,6 @@ CONFIG_64BIT Used only to check CONFIG_RCU_FANOUT value, inspection suffices. -CONFIG_PREEMPT_COUNT CONFIG_PREEMPT_RCU Redundant with CONFIG_PREEMPT, ignore.
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h +0 -1
@@ -8,7 +8,6 @@#undef CONFIG_HOTPLUG_CPU#undef CONFIG_MODULES#undef CONFIG_NO_HZ_FULL_SYSIDLE -#undef CONFIG_PREEMPT_COUNT#undef CONFIG_PREEMPT_RCU#undef CONFIG_PROVE_RCU#undef CONFIG_RCU_NOCB_CPU
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 13/15 preempt: Remove PREEMPT_COUNT from Kconfig
In reply to Paul E. McKenneyFrom: Thomas Gleixner <[email protected]>
All conditionals and irritations are gone.
--- kernel/Kconfig.preempt | 3 --- 1 file changed, 3 deletions(-)
kernel/Kconfig.preempt +0 -3
@@ -74,8 +74,5 @@ config PREEMPT_RT endchoice -config PREEMPT_COUNT - def_bool y - config PREEMPTION bool
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 14/15 rcu/tree: Allocate a page when caller is preemptible
In reply to Paul E. McKenneyFrom: "Uladzislau Rezki (Sony)" <[email protected]>
The current memory-allocation interface poses the following challenges:
a) In kernels built with CONFIG_PROVE_RAW_LOCK_NESTING, lockdep
complains ("BUG: Invalid wait context"). This complaint is due to the memory allocator acquiring non-raw spinlocks while a raw spinlocks is held. This problem can also arise if kvfree_rcu() is invoked while holding a raw spinlock.
b) In -rt kernels built with CONFIG_PREEMPT_RT, the situation
described in (a) above results in an attempt to acquire a sleeplock while holding a spinlock, which is of course forbidden. This can lead to "BUG: scheduling while atomic".
c) Please note that call_rcu() is invoked from raw atomic context,
so that kfree_rcu() and kvfree_rcu() are therefore also expected to be callable from atomic raw context as well.
However given that CONFIG_PREEMPT_COUNT is unconditionally enabled by the earlier commits in this series, the preemptible() macro now properly detects preempt-disable code regions even in kernels built with CONFIG_PREEMPT_NONE.
This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu(). If !preemptible(), then allocation is not possible, and kvfree_rcu() falls back to using the less cache-friendly rcu_head approach. Even when preemptible(), the caller might be involved in reclaim, so the GFP_ flags used by double-argument kvfree_rcu() must avoid invoking reclaim processing.
Note that single-argument kvfree_rcu() must be invoked in sleepable contexts, and that its fallback is the relatively high latency synchronize_rcu(). Single-argument kvfree_rcu() therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.
--- kernel/rcu/tree.c | 70 ++++++++++++++++--------------------------------------- 1 file changed, 20 insertions(+), 50 deletions(-)
kernel/rcu/tree.c +20 -50
@@ -3166,7 +3166,7 @@ static void kfree_rcu_work(struct work_struct *work)krc_this_cpu_unlock (krcp, flags);if (bkvhead[i]) -free_page ((unsigned long ) bkvhead[i]); +kfree (bkvhead[i]);cond_resched_tasks_rcu_qs (); }
@@ -3291,43 +3291,28 @@ static void kfree_rcu_monitor(struct work_struct *work) }static inline bool -kvfree_call_rcu_add_ptr_to_bulk (struct kfree_rcu_cpu *krcp,void *ptr) +add_ptr_to_bulk_krc_lock (struct kfree_rcu_cpu **krcp, +unsigned long *flags,void *ptr,bool can_sleep) {struct kvfree_rcu_bulk_data *bnode; +bool can_alloc_page =preemptible (); +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN;int idx; -if (unlikely (!krcp->initialized)) + *krcp =krc_this_cpu_lock (flags); +if (unlikely (!(*krcp)->initialized))return false; -lockdep_assert_held (&krcp->lock); idx = !!is_vmalloc_addr (ptr);/* Check if a new block is required. */ -if (!krcp->bkvhead[idx] || - krcp->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { - bnode =get_cached_bnode (krcp); -if (!bnode) { -/* -* To keep this path working on raw non-preemptible -* sections, prevent the optional entry into the -* allocator as it uses sleeping locks. In fact, even -* if the caller of kfree_rcu() is preemptible, this -* path still is not, as krcp->lock is a raw spinlock. -* With additional page pre-allocation in the works, -* hitting this return is going to be much less likely. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT )) -return false; - -/* -* NOTE: For one argument of kvfree_rcu() we can -* drop the lock and get the page in sleepable -* context. That would allow to maintain an array -* for the CONFIG_PREEMPT_RT as well if no cached -* pages are available. -*/ - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); +if (!(*krcp)->bkvhead[idx] || + (*krcp)->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { + bnode =get_cached_bnode (*krcp); +if (!bnode && can_alloc_page) { +krc_this_cpu_unlock (*krcp, *flags); + bnode =kmalloc (PAGE_SIZE , gfp); + *krcp =krc_this_cpu_lock (flags); }/* Switch to emergency path. */
@@ -3336,15 +3321,15 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)/* Initialize the new block. */ bnode->nr_records =0 ; - bnode->next = krcp->bkvhead[idx]; + bnode->next = (*krcp)->bkvhead[idx];/* Attach it to the head. */ - krcp->bkvhead[idx] = bnode; + (*krcp)->bkvhead[idx] = bnode; }/* Finally insert. */ - krcp->bkvhead[idx]->records - [krcp->bkvhead[idx]->nr_records++] = ptr; + (*krcp)->bkvhead[idx]->records + [(*krcp)->bkvhead[idx]->nr_records++] = ptr;return true; }
@@ -3382,24 +3367,20 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) ptr = (unsigned long *) func; } - krcp =krc_this_cpu_lock (&flags); -// Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue (ptr)) {// Probable double kfree_rcu(), just leak. WARN_ONCE (1 ,"%s(): Double-freed call. rcu_head %p\n" , __func__, head); -// Mark as success and leave. - success = true; - goto unlock_return; +return ; }/* * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - success =kvfree_call_rcu_add_ptr_to_bulk (krcp, ptr); + success =add_ptr_to_bulk_krc_lock (&krcp, &flags, ptr, !head);if (!success) {if (head ==NULL )// Inline if kvfree_rcu(one_arg) call.
@@ -4394,23 +4375,12 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); -struct kvfree_rcu_bulk_data *bnode;for (i =0 ; i <KFREE_N_BATCHES ; i++) {INIT_RCU_WORK (&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; } -for (i =0 ; i < rcu_min_cached_objs; i++) { - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); - -if (bnode) -put_cached_bnode (krcp, bnode); -else -pr_err ("Failed to preallocate for %d CPU!\n" , cpu); - } -INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); krcp->initialized = true; }
Signature · 1 line
2.9.5
Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]
Quoted 2 lines
This commit therefore uses
preemptible()to determine whether allocation is possible at all for double-argumentkvfree_rcu().
This deserves a comment. Because GFP_ATOMIC is possible for many !preemptible() contexts. It is the raw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.
Quoted 5 lines
If !preemptible(),
then allocation is not possible, andkvfree_rcu()falls back to using the less cache-friendlyrcu_headapproach. Even whenpreemptible(), the caller might be involved in reclaim, so theGFP_flags used by double-argumentkvfree_rcu()must avoid invoking reclaim processing.
Could you be more specific? Is this about being called directly in the reclaim context and you want to prevent a recursion? If that is the case, do you really need to special case this in any way? Any memory reclaim will set PF_MEMALLOC so allocations called from that context will not perform reclaim. So if you are called from the reclaim directly then you might want to do GFP_KERNEL | __GFP_NOMEMALLOC | __GFP_NOWARN. That should handle both from-the-recalim and outside of reclaim contexts just fine (assuming you don't allocated from !preemptible()) context.
Quoted 5 lines
Note that single-argument
kvfree_rcu()must be invoked in sleepable contexts, and that its fallback is the relatively high latencysynchronize_rcu(). Single-argumentkvfree_rcu()therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.
[...]
kernel/rcu/tree.c static void kfree_rcu_monitor(struct work_struct *work) · line 3299
Quoted 8 lines
static inline bool
kernel/rcu/tree.c +4
-1-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) +add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, + unsigned long *flags, void *ptr, bool can_sleep) { struct kvfree_rcu_bulk_data *bnode; +bool can_alloc_page =preemptible (); +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN;
This is quite confusing IMHO. At least without a further explanation. can_sleep is not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?
Quoted 32 lines
kernel/rcu/tree.c +2
-23int idx; -if (unlikely (!krcp->initialized)) + *krcp =krc_this_cpu_lock (flags); +if (unlikely (!(*krcp)->initialized))return false; -lockdep_assert_held (&krcp->lock); idx = !!is_vmalloc_addr (ptr);/* Check if a new block is required. */ -if (!krcp->bkvhead[idx] || - krcp->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { - bnode =get_cached_bnode (krcp); -if (!bnode) { -/* -* To keep this path working on raw non-preemptible -* sections, prevent the optional entry into the -* allocator as it uses sleeping locks. In fact, even -* if the caller of kfree_rcu() is preemptible, this -* path still is not, as krcp->lock is a raw spinlock. -* With additional page pre-allocation in the works, -* hitting this return is going to be much less likely. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT )) -return false; - -/* -* NOTE: For one argument of kvfree_rcu() we can -* drop the lock and get the page in sleepable -* context. That would allow to maintain an array -* for the CONFIG_PREEMPT_RT as well if no cached
kernel/rcu/tree.c static void kfree_rcu_monitor(struct work_struct *work) · line 3314
Quoted 10 lines
kernel/rcu/tree.c +6
-4-* pages are available. -*/ - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); +if (!(*krcp)->bkvhead[idx] || + (*krcp)->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { + bnode =get_cached_bnode (*krcp); +if (!bnode && can_alloc_page) { +krc_this_cpu_unlock (*krcp, *flags); + bnode =kmalloc (PAGE_SIZE , gfp);
What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?
Signature · 2 lines
Michal Hocko SUSE Labs
Paul E. McKenney
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
Quoted 2 lines · Michal Hocko
On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]
Apologies for the delay, but today has not been boring.
Quoted 7 lines
Quoted 2 lines
This commit therefore uses
preemptible()to determine whether allocation is possible at all for double-argumentkvfree_rcu().This deserves a comment. Because
GFP_ATOMICis possible for many !preemptible() contexts. It is theraw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.
Good point, and so I have added the following as a header comment to the add_ptr_to_bulk_krc_lock() function:
// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() // state specified by flags. If can_sleep is true, the caller must // be schedulable and not be holding any locks or mutexes that might be // acquired by the memory allocator or anything that it might invoke. // If !can_sleep, then if !preemptible() no allocation will be undertaken, // otherwise the allocation will use GFP_ATOMIC to avoid the remainder of // the aforementioned deadlock possibilities. Returns true iff ptr was // successfully recorded, else the caller must use a fallback.
The updated patch is shown below.
Of course, to Vlastimil's point, lockless access to the allocator would significantly reduce the level of confusion posed by this code. But that is a separate issue at the moment.
Quoted 6 lines
Quoted 5 lines
If !preemptible(),
then allocation is not possible, andkvfree_rcu()falls back to using the less cache-friendlyrcu_headapproach. Even whenpreemptible(), the caller might be involved in reclaim, so theGFP_flags used by double-argumentkvfree_rcu()must avoid invoking reclaim processing.
Quoted 8 lines
Could you be more specific? Is this about being called directly in the reclaim context and you want to prevent a recursion? If that is the case, do you really need to special case this in any way? Any memory reclaim will set
PF_MEMALLOCso allocations called from that context will not perform reclaim. So if you are called from the reclaim directly then you might want to doGFP_KERNEL|__GFP_NOMEMALLOC|__GFP_NOWARN. That should handle both from-the-recalim and outside of reclaim contexts just fine (assuming you don't allocated from !preemptible()) context.
My thought was to point you at Daniel Vetter, but you already got in touch with him, so thank you for that!
Quoted 6 lines
Quoted 5 lines
Note that single-argument
kvfree_rcu()must be invoked in sleepable contexts, and that its fallback is the relatively high latencysynchronize_rcu(). Single-argumentkvfree_rcu()therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.[...]
Quoted 12 lines
Review
Quoted 8 lines
static inline bool
(quoted diff) +4
-1-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) +add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, + unsigned long *flags, void *ptr, bool can_sleep) { struct kvfree_rcu_bulk_data *bnode; + bool can_alloc_page = preemptible(); + gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;This is quite confusing IMHO. At least without a further explanation.
can_sleepis not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?
No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.
The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().
Quoted 37 lines
Quoted 37 lines
(quoted diff) +3
-27int idx; - if (unlikely(!krcp->initialized)) + *krcp = krc_this_cpu_lock(flags); + if (unlikely(!(*krcp)->initialized)) return false; - lockdep_assert_held(&krcp->lock); idx = !!is_vmalloc_addr(ptr); /* Check if a new block is required. */ - if (!krcp->bkvhead[idx] || - krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) { - bnode = get_cached_bnode(krcp); - if (!bnode) { - /* - * To keep this path working on raw non-preemptible - * sections, prevent the optional entry into the - * allocator as it uses sleeping locks. In fact, even - * if the caller of kfree_rcu() is preemptible, this - * path still is not, as krcp->lock is a raw spinlock. - * With additional page pre-allocation in the works, - * hitting this return is going to be much less likely. - */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - return false; - - /* - * NOTE: For one argument of kvfree_rcu() we can - * drop the lock and get the page in sleepable - * context. That would allow to maintain an array - * for the CONFIG_PREEMPT_RT as well if no cached - * pages are available. - */ - bnode = (struct kvfree_rcu_bulk_data *) - __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + if (!(*krcp)->bkvhead[idx] ||
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ (*krcp)->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) { + bnode = get_cached_bnode(*krcp); + if (!bnode && can_alloc_page) { + krc_this_cpu_unlock(*krcp, *flags); + bnode = kmalloc(PAGE_SIZE, gfp);What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?
Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)
Again, please see below for the updated patch. The only change from the earlier verison is the addition of the header comment mentioned above.
Thanx, Paul
------------------------------------------------------------------------
commit 3a8e10ea2a2ecf59a33d7d80ad6c4372c0c27f78
Author: Uladzislau Rezki (Sony) <[email protected]> Date: Tue Sep 22 21:06:22 2020 +0200
rcu/tree: Allocate a page when caller is preemptible
The current memory-allocation interface poses the following challenges:
a) In kernels built with CONFIG_PROVE_RAW_LOCK_NESTING, lockdep
complains ("BUG: Invalid wait context"). This complaint is due to the memory allocator acquiring non-raw spinlocks while a raw spinlocks is held. This problem can also arise if kvfree_rcu() is invoked while holding a raw spinlock.
b) In -rt kernels built with CONFIG_PREEMPT_RT, the situation
described in (a) above results in an attempt to acquire a sleeplock while holding a spinlock, which is of course forbidden. This can lead to "BUG: scheduling while atomic".
c) Please note that call_rcu() is invoked from raw atomic context,
so that kfree_rcu() and kvfree_rcu() are therefore also expected to be callable from atomic raw context as well.
However given that CONFIG_PREEMPT_COUNT is unconditionally enabled by the earlier commits in this series, the preemptible() macro now properly detects preempt-disable code regions even in kernels built with CONFIG_PREEMPT_NONE.
This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu(). If !preemptible(), then allocation is not possible, and kvfree_rcu() falls back to using the less cache-friendly rcu_head approach. Even when preemptible(), the caller might be involved in reclaim, so the GFP_ flags used by double-argument kvfree_rcu() must avoid invoking reclaim processing.
Note that single-argument kvfree_rcu() must be invoked in sleepable contexts, and that its fallback is the relatively high latency synchronize_rcu(). Single-argument kvfree_rcu() therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.
kernel/rcu/tree.c +28 -50
@@ -3166,7 +3166,7 @@ static void kfree_rcu_work(struct work_struct *work)krc_this_cpu_unlock (krcp, flags);if (bkvhead[i]) -free_page ((unsigned long ) bkvhead[i]); +kfree (bkvhead[i]);cond_resched_tasks_rcu_qs (); }
@@ -3290,44 +3290,37 @@ static void kfree_rcu_monitor(struct work_struct *work)raw_spin_unlock_irqrestore (&krcp->lock, flags); } +// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() +// state specified by flags. If can_sleep is true, the caller must +// be schedulable and not be holding any locks or mutexes that might be +// acquired by the memory allocator or anything that it might invoke. +// If !can_sleep, then if !preemptible() no allocation will be undertaken, +// otherwise the allocation will use GFP_ATOMIC to avoid the remainder of +// the aforementioned deadlock possibilities. Returns true iff ptr was +// successfully recorded, else the caller must use a fallback. static inline bool -kvfree_call_rcu_add_ptr_to_bulk (struct kfree_rcu_cpu *krcp,void *ptr) +add_ptr_to_bulk_krc_lock (struct kfree_rcu_cpu **krcp, +unsigned long *flags,void *ptr,bool can_sleep) {struct kvfree_rcu_bulk_data *bnode; +bool can_alloc_page =preemptible (); +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN;int idx; -if (unlikely (!krcp->initialized)) + *krcp =krc_this_cpu_lock (flags); +if (unlikely (!(*krcp)->initialized))return false; -lockdep_assert_held (&krcp->lock); idx = !!is_vmalloc_addr (ptr);/* Check if a new block is required. */ -if (!krcp->bkvhead[idx] || - krcp->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { - bnode =get_cached_bnode (krcp); -if (!bnode) { -/* -* To keep this path working on raw non-preemptible -* sections, prevent the optional entry into the -* allocator as it uses sleeping locks. In fact, even -* if the caller of kfree_rcu() is preemptible, this -* path still is not, as krcp->lock is a raw spinlock. -* With additional page pre-allocation in the works, -* hitting this return is going to be much less likely. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT )) -return false; - -/* -* NOTE: For one argument of kvfree_rcu() we can -* drop the lock and get the page in sleepable -* context. That would allow to maintain an array -* for the CONFIG_PREEMPT_RT as well if no cached -* pages are available. -*/ - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); +if (!(*krcp)->bkvhead[idx] || + (*krcp)->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { + bnode =get_cached_bnode (*krcp); +if (!bnode && can_alloc_page) { +krc_this_cpu_unlock (*krcp, *flags); + bnode =kmalloc (PAGE_SIZE , gfp); + *krcp =krc_this_cpu_lock (flags); }/* Switch to emergency path. */
@@ -3336,15 +3329,15 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)/* Initialize the new block. */ bnode->nr_records =0 ; - bnode->next = krcp->bkvhead[idx]; + bnode->next = (*krcp)->bkvhead[idx];/* Attach it to the head. */ - krcp->bkvhead[idx] = bnode; + (*krcp)->bkvhead[idx] = bnode; }/* Finally insert. */ - krcp->bkvhead[idx]->records - [krcp->bkvhead[idx]->nr_records++] = ptr; + (*krcp)->bkvhead[idx]->records + [(*krcp)->bkvhead[idx]->nr_records++] = ptr;return true; }
@@ -3382,24 +3375,20 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) ptr = (unsigned long *) func; } - krcp =krc_this_cpu_lock (&flags); -// Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue (ptr)) {// Probable double kfree_rcu(), just leak. WARN_ONCE (1 ,"%s(): Double-freed call. rcu_head %p\n" , __func__, head); -// Mark as success and leave. - success = true; - goto unlock_return; +return ; }/* * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - success =kvfree_call_rcu_add_ptr_to_bulk (krcp, ptr); + success =add_ptr_to_bulk_krc_lock (&krcp, &flags, ptr, !head);if (!success) {if (head ==NULL )// Inline if kvfree_rcu(one_arg) call.
@@ -4394,23 +4383,12 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); -struct kvfree_rcu_bulk_data *bnode;for (i =0 ; i <KFREE_N_BATCHES ; i++) {INIT_RCU_WORK (&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; } -for (i =0 ; i < rcu_min_cached_objs; i++) { - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); - -if (bnode) -put_cached_bnode (krcp, bnode); -else -pr_err ("Failed to preallocate for %d CPU!\n" , cpu); - } -INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); krcp->initialized = true; }
Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
Quoted 17 lines · Paul E. McKenney
Quoted 2 lines · Michal Hocko
On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]Apologies for the delay, but today has not been boring.
Quoted 7 lines
Quoted 2 lines
This commit therefore uses
preemptible()to determine whether allocation is possible at all for double-argumentkvfree_rcu().This deserves a comment. Because
GFP_ATOMICis possible for many !preemptible() contexts. It is theraw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.Good point, and so I have added the following as a header comment to the
add_ptr_to_bulk_krc_lock()function:
Quoted 8 lines
// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() // state specified by flags. If
can_sleepis true, the caller must // be schedulable and not be holding any locks or mutexes that might be // acquired by the memory allocator or anything that it might invoke. // If !can_sleep, then if !preemptible() no allocation will be undertaken, // otherwise the allocation will useGFP_ATOMICto avoid the remainder of // the aforementioned deadlock possibilities. Returns true iff ptr was // successfully recorded, else the caller must use a fallback.
OK, not trivial to follow but at least verbose enough to understand the intention after some mulling. Definitely an improvement, thanks!
[...]
Quoted 19 lines
Quoted 11 lines
kernel/rcu/tree.c static void kfree_rcu_monitor(struct work_struct *work) · line 3307
Quoted 7 lines
kernel/rcu/tree.c +4
-1-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) +add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, + unsigned long *flags, void *ptr, bool can_sleep) { struct kvfree_rcu_bulk_data *bnode; +bool can_alloc_page =preemptible (); +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN;This is quite confusing IMHO. At least without a further explanation.
can_sleepis not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.
Quoted 3 lines
The only way that can_sleep==true is if this function was invoked due to a call to single-argument
kvfree_rcu(), which must be schedulable because its fallback is to invokesynchronize_rcu().
OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need __GFP_NOMEMALLOC as well.
[...]
Quoted 6 lines
Quoted 2 lines
What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?Well, you guys gave me considerable heat about abusing internal allocator interfaces, and
kmalloc()andkfree()seem to be about as non-internal as you can get and still be invoking the allocator. ;-)
alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?
Quoted 8 lines
Quoted 3 lines
Well, you guys gave me considerable heat about abusing internal allocator interfaces, and
kmalloc()andkfree()seem to be about as non-internal as you can get and still be invoking the allocator. ;-)
alloc_pagesresp.__get_free_pagesis a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.
I do not have a strong opinion here but i would tend to using of the page allocator directly. Because we operate on a page level where the page allocator is supposed to be used. From the other hand i saw a slightly better performance in case of SLAB only. I think that is because of
a) slab caches; b) more objects which can be cached. But i have one more concern about using of kmalloc(). That is about fragmentations it can cause with PAGE_SIZE requests.
As for SLUB it was identical whereas the SLOB i din not check.
--
Vlad Rezki
Paul E. McKenney
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
In reply to Michal HockoQuoted 27 lines · Michal Hocko
Quoted 17 lines · Paul E. McKenney
Quoted 2 lines · Michal Hocko
On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]Apologies for the delay, but today has not been boring.
Quoted 7 lines
Quoted 2 lines
This commit therefore uses
preemptible()to determine whether allocation is possible at all for double-argumentkvfree_rcu().This deserves a comment. Because
GFP_ATOMICis possible for many !preemptible() contexts. It is theraw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.Good point, and so I have added the following as a header comment to the
add_ptr_to_bulk_krc_lock()function:Quoted 8 lines
// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() // state specified by flags. If
can_sleepis true, the caller must // be schedulable and not be holding any locks or mutexes that might be // acquired by the memory allocator or anything that it might invoke. // If !can_sleep, then if !preemptible() no allocation will be undertaken, // otherwise the allocation will useGFP_ATOMICto avoid the remainder of // the aforementioned deadlock possibilities. Returns true iff ptr was // successfully recorded, else the caller must use a fallback.
Quoted 2 lines
OK, not trivial to follow but at least verbose enough to understand the intention after some mulling. Definitely an improvement, thanks!
Glad it helped! With some luck, perhaps it will improve with time...
Quoted 20 lines
[...]
Quoted 19 lines
Quoted 11 lines
Review
Quoted 7 lines
(quoted diff) +4
-1-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) +add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, + unsigned long *flags, void *ptr, bool can_sleep) { struct kvfree_rcu_bulk_data *bnode; + bool can_alloc_page = preemptible(); + gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;This is quite confusing IMHO. At least without a further explanation.
can_sleepis not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.
Quoted 7 lines
Quoted 3 lines
The only way that can_sleep==true is if this function was invoked due to a call to single-argument
kvfree_rcu(), which must be schedulable because its fallback is to invokesynchronize_rcu().OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need
__GFP_NOMEMALLOCas well.
Right now the restriction is that single-argument (AKA can_sleep==true) kvfree_rcu() cannot be invoked from memory reclaim context.
But would adding __GFP_NOMEMALLOC to the can_sleep==true GFP_ flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).
Quoted 5 lines
[...]
Quoted 3 lines
Quoted 2 lines
What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?
Quoted 7 lines
Quoted 3 lines
Well, you guys gave me considerable heat about abusing internal allocator interfaces, and
kmalloc()andkfree()seem to be about as non-internal as you can get and still be invoking the allocator. ;-)
alloc_pagesresp.__get_free_pagesis a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.
OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.
You mentioned alloc_pages(). I reverted to __get_free_pages(), but alloc_pages() of course looks nicer. What are the tradeoffs between __get_free_pages() and alloc_pages()?
Thanx, Paul
------------------------------------------------------------------------
commit 490b638d7c241ac06cee168ccf8688bb8b872478
Author: Paul E. McKenney <[email protected]> Date: Wed Sep 30 16:16:39 2020 -0700
kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.
The advantages of using kmalloc() and kfree() are a possible small speedup on CONFIG_SLAB=y systems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using __get_free_page() and free_page() are a possible reduction in fragmentation and direct access to the buddy allocator.
To help settle the question as to which to use, this commit switches from kmalloc() and kfree() to __get_free_page() and free_page().
kernel/rcu/tree.c +3 -2
@@ -3225,7 +3225,8 @@ static void kfree_rcu_work(struct work_struct *work) bkvhead[i] =NULL ;krc_this_cpu_unlock (krcp, flags); -kfree (bkvhead[i]); +if (bkvhead[i]) +free_page ((unsigned long )bkvhead[i]);cond_resched_tasks_rcu_qs (); }
@@ -3378,7 +3379,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, bnode =get_cached_bnode (*krcp);if (!bnode && can_alloc_page) {krc_this_cpu_unlock (*krcp, *flags); - bnode =kmalloc (PAGE_SIZE , gfp); + bnode = (struct kvfree_rcu_bulk_data *)__get_free_page (gfp); *krcp =krc_this_cpu_lock (flags); }
Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
Quoted 2 lines · Paul E. McKenney
Quoted 1 line · Michal Hocko
On Tue 29-09-20 18:53:27, Paul E. McKenney wrote:
[...]
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.
Quoted 7 lines
Quoted 3 lines
The only way that can_sleep==true is if this function was invoked due to a call to single-argument
kvfree_rcu(), which must be schedulable because its fallback is to invokesynchronize_rcu().OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need
__GFP_NOMEMALLOCas well.
Quoted 7 lines
Right now the restriction is that single-argument (AKA can_sleep==true)
kvfree_rcu()cannot be invoked from memory reclaim context.But would adding
__GFP_NOMEMALLOCto the can_sleep==trueGFP_flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).
It would be quite daring to claim it will be ok but it will certainly be less problematic. Adding the flag will not hurt in any case. As this is a shared called that might be called from many contexts I think it will be safer to have it there. The justification is that it will prevent consumption of memory reserves from MEMALLOC contexts.
Quoted 10 lines
Quoted 10 lines
[...]
Quoted 6 lines
Quoted 2 lines
What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?Well, you guys gave me considerable heat about abusing internal allocator interfaces, and
kmalloc()andkfree()seem to be about as non-internal as you can get and still be invoking the allocator. ;-)
Quoted 8 lines
Quoted 3 lines
alloc_pagesresp.__get_free_pagesis a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.
It doesn't matter all that much. Both allocators will work. It is just a matter of using optimal tool for the specific purose.
Quoted 3 lines
You mentioned
alloc_pages(). I reverted to__get_free_pages(), butalloc_pages()of course looks nicer. What are the tradeoffs between__get_free_pages()andalloc_pages()?
alloc_pages will return struct page but you need a kernel pointer. That is what __get_free_pages will give you (or you can call page_address directly).
Quoted 16 lines
Thanx, Paul
------------------------------------------------------------------------
commit
490b638d7c241ac06cee168ccf8688bb8b872478Author: Paul E. McKenney <[email protected]> Date: Wed Sep 30 16:16:39 2020 -0700
kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.The advantages of using
kmalloc()andkfree()are a possible small speedup onCONFIG_SLAB=ysystems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using__get_free_page()andfree_page()are a possible reduction in fragmentation and direct access to the buddy allocator.
Quoted 6 lines
To help settle the question as to which to use, this commit switches from
kmalloc()andkfree()to__get_free_page()andfree_page().Suggested-by Michal Hocko <[email protected]>Suggested-by "Uladzislau Rezki (Sony)" <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Yes, looks good to me. I am not entirely sure about the fragmentation argument. It really depends on the SL.B allocator internals. The same applies for the potential speed up. I would be even surprised if the SLAB was faster in average considering it has to use the page allocator as well. So to me the primary motivation would be "use the right tool for the purpose".
Quoted 23 lines
kernel/rcu/tree.c +3
-2@@ -3225,7 +3225,8 @@ static void kfree_rcu_work(struct work_struct *work) bkvhead[i] =NULL ;krc_this_cpu_unlock (krcp, flags); -kfree (bkvhead[i]); +if (bkvhead[i]) +free_page ((unsigned long )bkvhead[i]);cond_resched_tasks_rcu_qs (); }@@ -3378,7 +3379,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, bnode =get_cached_bnode (*krcp);if (!bnode && can_alloc_page) {krc_this_cpu_unlock (*krcp, *flags); - bnode =kmalloc (PAGE_SIZE , gfp); + bnode = (struct kvfree_rcu_bulk_data *)__get_free_page (gfp); *krcp =krc_this_cpu_lock (flags); }
Signature · 2 lines
Michal Hocko SUSE Labs
Paul E. McKenney
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
Quoted 27 lines · Michal Hocko
Quoted 2 lines · Paul E. McKenney
Quoted 1 line · Michal Hocko
On Tue 29-09-20 18:53:27, Paul E. McKenney wrote:
[...]
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.
Quoted 7 lines
Quoted 3 lines
The only way that can_sleep==true is if this function was invoked due to a call to single-argument
kvfree_rcu(), which must be schedulable because its fallback is to invokesynchronize_rcu().OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need
__GFP_NOMEMALLOCas well.Quoted 7 lines
Right now the restriction is that single-argument (AKA can_sleep==true)
kvfree_rcu()cannot be invoked from memory reclaim context.But would adding
__GFP_NOMEMALLOCto the can_sleep==trueGFP_flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).
Quoted 5 lines
It would be quite daring to claim it will be ok but it will certainly be less problematic. Adding the flag will not hurt in any case. As this is a shared called that might be called from many contexts I think it will be safer to have it there. The justification is that it will prevent consumption of memory reserves from MEMALLOC contexts.
Ah, so a different goal (and yes, I finally went over and read the relevant documentation). Agreed, the can_sleep path does not really need to be dipping into the emergency reserves. And it looks like the not-from-reclaim restriction is still at least partially in effect, but one step at a time.
The patch is shown below, which I have queued for a later release.
Quoted 21 lines
Quoted 9 lines
Quoted 9 lines
[...]
Quoted 6 lines
Quoted 2 lines
What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?Well, you guys gave me considerable heat about abusing internal allocator interfaces, and
kmalloc()andkfree()seem to be about as non-internal as you can get and still be invoking the allocator. ;-)Quoted 8 lines
Quoted 3 lines
alloc_pagesresp.__get_free_pagesis a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.
It doesn't matter all that much. Both allocators will work. It is just a matter of using optimal tool for the specific purose.
Quoted 7 lines
Quoted 3 lines
You mentioned
alloc_pages(). I reverted to__get_free_pages(), butalloc_pages()of course looks nicer. What are the tradeoffs between__get_free_pages()andalloc_pages()?
alloc_pageswill return struct page but you need a kernel pointer. That is what__get_free_pageswill give you (or you can callpage_addressdirectly).
Thank you, looks like __get_free_pages() is the tool for this job.
Please see below for the aforementioned patch.
Thanx, Paul
------------------------------------------------------------------------
commit ea5c19d21233b5e8d3d06c0d4ecd6be9f2829dc3
Author: Paul E. McKenney <[email protected]> Date: Thu Oct 1 09:24:40 2020 -0700
kvfree_rcu: Use __GFP_NOMEMALLOC for single-argument kvfree_rcu()
This commit applies the __GFP_NOMEMALLOC gfp flag to memory allocations carried out by the single-argument variant of kvfree_rcu(), thus avoiding this can-sleep code path from dipping into the emergency reserves.
kernel/rcu/tree.c +2 -1
@@ -3364,7 +3364,8 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, {struct kvfree_rcu_bulk_data *bnode;bool can_alloc_page =preemptible (); -gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN; +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOMEMALLOC + :GFP_ATOMIC ) | __GFP_NOWARN;int idx; *krcp =krc_this_cpu_lock (flags);
Michal Hocko
Acked-byRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
On Thu 01-10-20 09:27:09, Paul E. McKenney wrote:
[...]
Quoted 4 lines
commit
ea5c19d21233b5e8d3d06c0d4ecd6be9f2829dc3Author: Paul E. McKenney <[email protected]> Date: Thu Oct 1 09:24:40 2020 -0700
Quoted 8 lines
kvfree_rcu: Use__GFP_NOMEMALLOCfor single-argumentkvfree_rcu()This commit applies the
__GFP_NOMEMALLOCgfp flag to memory allocations carried out by the single-argument variant ofkvfree_rcu(), thus avoiding this can-sleep code path from dipping into the emergency reserves.Suggested-by Michal Hocko <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
LGTM. At least for this one I feel competent to give you
Quoted 15 lines
kernel/rcu/tree.c +2
-1@@ -3364,7 +3364,8 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, {struct kvfree_rcu_bulk_data *bnode;bool can_alloc_page =preemptible (); -gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN; +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOMEMALLOC + :GFP_ATOMIC ) | __GFP_NOWARN;int idx; *krcp =krc_this_cpu_lock (flags);
Signature · 2 lines
Michal Hocko SUSE Labs
Paul E. McKenney
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
Quoted 12 lines · Michal Hocko
On Thu 01-10-20 09:27:09, Paul E. McKenney wrote:
[...]Quoted 10 lines
commit
ea5c19d21233b5e8d3d06c0d4ecd6be9f2829dc3Author: Paul E. McKenney <[email protected]> Date: Thu Oct 1 09:24:40 2020 -0700
kvfree_rcu: Use__GFP_NOMEMALLOCfor single-argumentkvfree_rcu()This commit applies the
__GFP_NOMEMALLOCgfp flag to memory allocations carried out by the single-argument variant ofkvfree_rcu(), thus avoiding this can-sleep code path from dipping into the emergency reserves.
Quoted 5 lines
Quoted 2 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>LGTM. At least for this one I feel competent to give you
Acked-by Michal Hocko <[email protected]>
Thank you very much! I will apply this on the next rebase later today, Pacific Time.
Thanx, Paul
Quoted 18 lines
Quoted 14 lines
kernel/rcu/tree.c +2
-1@@ -3364,7 +3364,8 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, {struct kvfree_rcu_bulk_data *bnode;bool can_alloc_page =preemptible (); -gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL :GFP_ATOMIC ) | __GFP_NOWARN; +gfp_t gfp = (can_sleep ?GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOMEMALLOC + :GFP_ATOMIC ) | __GFP_NOWARN;int idx; *krcp =krc_this_cpu_lock (flags);Signature · 2 lines
Michal Hocko SUSE Labs
Paul E. McKenney
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
In reply to Michal HockoQuoted 1 line · Michal Hocko
On Wed 30-09-20 16:21:54, Paul E. McKenney wrote:
[ . . . ]
Hit "send" too soon, apologies...
Quoted 21 lines
Quoted 14 lines
------------------------------------------------------------------------
commit
490b638d7c241ac06cee168ccf8688bb8b872478Author: Paul E. McKenney <[email protected]> Date: Wed Sep 30 16:16:39 2020 -0700
kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.The advantages of using
kmalloc()andkfree()are a possible small speedup onCONFIG_SLAB=ysystems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using__get_free_page()andfree_page()are a possible reduction in fragmentation and direct access to the buddy allocator.Quoted 6 lines
To help settle the question as to which to use, this commit switches from
kmalloc()andkfree()to__get_free_page()andfree_page().Suggested-by Michal Hocko <[email protected]>Suggested-by "Uladzislau Rezki (Sony)" <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 6 lines
Yes, looks good to me. I am not entirely sure about the fragmentation argument. It really depends on the SL.B allocator internals. The same applies for the potential speed up. I would be even surprised if the SLAB was faster in average considering it has to use the page allocator as well. So to me the primary motivation would be "use the right tool for the purpose".
Very well, I will update the commit message, and thank you!
Thanx, Paul
Quoted 27 lines
Quoted 23 lines
kernel/rcu/tree.c +3
-2@@ -3225,7 +3225,8 @@ static void kfree_rcu_work(struct work_struct *work) bkvhead[i] =NULL ;krc_this_cpu_unlock (krcp, flags); -kfree (bkvhead[i]); +if (bkvhead[i]) +free_page ((unsigned long )bkvhead[i]);cond_resched_tasks_rcu_qs (); }@@ -3378,7 +3379,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, bnode =get_cached_bnode (*krcp);if (!bnode && can_alloc_page) {krc_this_cpu_unlock (*krcp, *flags); - bnode =kmalloc (PAGE_SIZE , gfp); + bnode = (struct kvfree_rcu_bulk_data *)__get_free_page (gfp); *krcp =krc_this_cpu_lock (flags); }Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible
In reply to Michal HockoQuoted 86 lines · Michal Hocko
Quoted 2 lines · Paul E. McKenney
Quoted 1 line · Michal Hocko
On Tue 29-09-20 18:53:27, Paul E. McKenney wrote:
[...]
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.
Quoted 7 lines
Quoted 3 lines
The only way that can_sleep==true is if this function was invoked due to a call to single-argument
kvfree_rcu(), which must be schedulable because its fallback is to invokesynchronize_rcu().OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need
__GFP_NOMEMALLOCas well.Quoted 7 lines
Right now the restriction is that single-argument (AKA can_sleep==true)
kvfree_rcu()cannot be invoked from memory reclaim context.But would adding
__GFP_NOMEMALLOCto the can_sleep==trueGFP_flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).It would be quite daring to claim it will be ok but it will certainly be less problematic. Adding the flag will not hurt in any case. As this is a shared called that might be called from many contexts I think it will be safer to have it there. The justification is that it will prevent consumption of memory reserves from MEMALLOC contexts.
Quoted 10 lines
Quoted 10 lines
[...]
Quoted 6 lines
Quoted 2 lines
What is the point of calling kmalloc for a
PAGE_SIZEobject? Wouldn't using the page allocator directly be better?Well, you guys gave me considerable heat about abusing internal allocator interfaces, and
kmalloc()andkfree()seem to be about as non-internal as you can get and still be invoking the allocator. ;-)Quoted 8 lines
Quoted 3 lines
alloc_pagesresp.__get_free_pagesis a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.
It doesn't matter all that much. Both allocators will work. It is just a matter of using optimal tool for the specific purose.
Quoted 3 lines
You mentioned
alloc_pages(). I reverted to__get_free_pages(), butalloc_pages()of course looks nicer. What are the tradeoffs between__get_free_pages()andalloc_pages()?
alloc_pageswill return struct page but you need a kernel pointer. That is what__get_free_pageswill give you (or you can callpage_addressdirectly).Quoted 16 lines
Thanx, Paul
------------------------------------------------------------------------
commit
490b638d7c241ac06cee168ccf8688bb8b872478Author: Paul E. McKenney <[email protected]> Date: Wed Sep 30 16:16:39 2020 -0700
kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.The advantages of using
kmalloc()andkfree()are a possible small speedup onCONFIG_SLAB=ysystems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using__get_free_page()andfree_page()are a possible reduction in fragmentation and direct access to the buddy allocator.Quoted 6 lines
To help settle the question as to which to use, this commit switches from
kmalloc()andkfree()to__get_free_page()andfree_page().Suggested-by Michal Hocko <[email protected]>Suggested-by "Uladzislau Rezki (Sony)" <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 7 lines
Yes, looks good to me. I am not entirely sure about the fragmentation argument. It really depends on the SL.B allocator internals. The same applies for the potential speed up. I would be even surprised if the SLAB was faster in average considering it has to use the page allocator as well. So to me the primary motivation would be "use the right tool for the purpose".
As for raised a concern about fragmentation, i mostly was thinking about that SLAbs where not designed to do an efficient allocations for sizes which are >= than PAGE_SIZE. But it depends on three different implementations, actually it also a good argument to switch to the page allocator. I mean to get rid of such dependency.
Other side is, SLABs, at least SLAB and SLUB use slab-caches and sizes which they support include up to:
<snip>
kmalloc-8k 420 420 8192 4 kmalloc-4k 1372 1392 4096 8 8 : tunables 0 0
...
<snip>
I would no be surprised that SLAB is faster than using the page allocator in _some_ sense. If it is principle i can double check. I can explain it just in having dynamic caching that can grow based on demand, thus there is no need to go deeper to page allocator if the kmalloc-4k has extra objects. But the worst case of course will be slower :)
--
Vlad Rezki
[email protected]
maintainerPATCH 15/15 kvfree_rcu(): Fix ifnullfree.cocci warnings
In reply to Paul E. McKenneyFrom: kernel test robot <[email protected]>
NULL check before kfree is not needed.
Generated by: scripts/coccinelle/free/ifnullfree.cocci
--- kernel/rcu/tree.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/rcu/tree.c +1 -2
@@ -3165,8 +3165,7 @@ static void kfree_rcu_work(struct work_struct *work) bkvhead[i] =NULL ;krc_this_cpu_unlock (krcp, flags); -if (bkvhead[i]) -kfree (bkvhead[i]); +kfree (bkvhead[i]);cond_resched_tasks_rcu_qs (); }
Signature · 1 line
2.9.5