Skip to contentSkip to conversation

[PATCH tip/core/rcu 0/15]

PATCH In review Paul E. McKenney 5 participants 30 messages Sep 28, 2020 – Oct 2, 2020

Subsystems (14)
lore
Share & tools
lore

Paul E. McKenney

author maintainer
#0 lore raw

cover [PATCH tip/core/rcu 0/15]

Hello!

This is a repost of Thomas Gleixner's "Make preempt count unconditional" series [1], but with the addition of a kvfree_rcu() bug-fix patch making use of this PREEMPT_COUNT addition. This series is intended for the upcoming v5.10 merge window.

This addition fixes the -rt bug reported by Sebastian Andrzej Siewior [2]. Please note that with the advent of the new lockdep Kconfig option CONFIG_PROVE_RAW_LOCK_NESTING, this is now also a mainline bug. In happy contrast to the surprisingly large number of earlier versions of this fix, this version uses only pre-existing kernel interfaces, and furthermore uses them in conventional ways.

1-13. Thomas's series [1].

14. Use the newly accurate preemptible() macro to cause kvfree_rcu()
to do its allocations only when it is safe to do so, even in CONFIG_PREEMPT_NONE=y kernels, courtesy of Uladzislau Rezki. Again, this fixes the bug reported by Sebastian [2].

15. Checkpatch fix removing NULL check guarding kfree(), courtesy
of kernel test robot and Julia Lawall.

In addition, this series reduces the size of the kernel by 100 lines of code, about two thirds of which is from Thomas's original series and the remaining one third from the bug fix.

Changes from v1 [1]:

o Fix trivial !SMP build failure.
o Apply checkpatch spelling suggestions.

Thanx, Paul

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/

------------------------------------------------------------------------

 arch/arm/include/asm/assembler.h                                 |   11 -
 arch/arm/kernel/iwmmxt.S                                         |    2 
 arch/arm/mach-ep93xx/crunch-bits.S                               |    2 
 arch/xtensa/kernel/entry.S                                       |    2 
 drivers/gpu/drm/i915/Kconfig.debug                               |    1 
 drivers/gpu/drm/i915/i915_utils.h                                |    3 
 include/linux/bit_spinlock.h                                     |    4 
 include/linux/lockdep.h                                          |    6 
 include/linux/pagemap.h                                          |    4 
 include/linux/preempt.h                                          |   37 -----
 include/linux/uaccess.h                                          |    6 
 kernel/Kconfig.preempt                                           |    6 
 kernel/rcu/tree.c                                                |   73 ++--------
 kernel/sched/core.c                                              |    6 
 lib/Kconfig.debug                                                |    5 
 tools/testing/selftests/rcutorture/configs/rcu/SRCU-t            |    1 
 tools/testing/selftests/rcutorture/configs/rcu/SRCU-u            |    1 
 tools/testing/selftests/rcutorture/configs/rcu/TINY01            |    1 
 tools/testing/selftests/rcutorture/doc/TINY_RCU.txt              |    5 
 tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt      |    1 
 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h |    1 
 21 files changed, 39 insertions(+), 139 deletions(-)
author maintainer
#1 lore raw

PATCH 1/15 lib/debug: Remove pointless ARCH_NO_PREEMPT dependencies

From: Thomas Gleixner <[email protected]>

ARCH_NO_PREEMPT disables the selection of CONFIG_PREEMPT_VOLUNTARY and CONFIG_PREEMPT, but architectures which set this config option still support preempt count for hard and softirq accounting.

There is absolutely no reason to prevent lockdep from using the preempt counter nor is there a reason to prevent the enablement of CONFIG_DEBUG_ATOMIC_SLEEP on such architectures.

Remove the dependencies.

---
 lib/Kconfig.debug | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
lib/Kconfig.debug +1 -2
index e068c3c..f50fbcf 100644
@@ -1161,7 +1161,7 @@ config PROVE_LOCKING
 	select DEBUG_RWSEMS
 	select DEBUG_WW_MUTEX_SLOWPATH
 	select DEBUG_LOCK_ALLOC
-	select PREEMPT_COUNT if !ARCH_NO_PREEMPT
+	select PREEMPT_COUNT
 	select TRACE_IRQFLAGS
 	default n
 	help
@@ -1323,7 +1323,6 @@ config DEBUG_ATOMIC_SLEEP
 	bool "Sleep inside atomic section checking"
 	select PREEMPT_COUNT
 	depends on DEBUG_KERNEL
-	depends on !ARCH_NO_PREEMPT
 	help
 	  If you say Y here, various routines which may sleep will become very
 	  noisy if they are called inside atomic sections: when a spinlock is
Signature · 1 line
2.9.5
author maintainer
#2 lore raw

PATCH 2/15 preempt: Make preempt count unconditional

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

The handling of preempt_count() is inconsistent across kernel configurations. On kernels which have PREEMPT_COUNT=n preempt_disable/enable() and the lock/unlock functions are not affecting the preempt count, only local_bh_disable/enable() and _bh variants of locking, soft interrupt delivery, hard interrupt and NMI context affect it.

It's therefore impossible to have a consistent set of checks which provide information about the context in which a function is called. In many cases it makes sense to have separate functions for separate contexts, but there are valid reasons to avoid that and handle different calling contexts conditionally.

The lack of such indicators which work on all kernel configuratios is a constant source of trouble because developers either do not understand the implications or try to work around this inconsistency in weird ways. Neither seem these issues be catched by reviewers and testing.

Recently merged code does:

	 gfp = preemptible() ? GFP_KERNEL : GFP_ATOMIC;

Looks obviously correct, except for the fact that preemptible() is unconditionally false for CONFIF_PREEMPT_COUNT=n, i.e. all allocations in that code use GFP_ATOMIC on such kernels.

Attempts to make preempt count unconditional and consistent have been rejected in the past with handwaving performance arguments.

Freshly conducted benchmarks did not reveal any measurable impact from enabling preempt count unconditionally. On kernels with CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY the preempt count is only incremented and decremented but the result of the decrement is not tested. Contrary to that enabling CONFIG_PREEMPT which tests the result has a small but measurable impact due to the conditional branch/call.

It's about time to make essential functionality of the kernel consistent across the various preemption models.

Enable CONFIG_PREEMPT_COUNT unconditionally. Follow up changes will remove the #ifdeffery and remove the config option at the end.

---
 kernel/Kconfig.preempt | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/Kconfig.preempt +1 -2
index bf82259..3f4712f 100644
@@ -75,8 +75,7 @@ config PREEMPT_RT
 endchoice
 
 config PREEMPT_COUNT
-       bool
+       def_bool y
 
 config PREEMPTION
        bool
-       select PREEMPT_COUNT
Signature · 1 line
2.9.5
author maintainer
#3 lore raw

PATCH 3/15 preempt: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Ben Segall <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 include/linux/preempt.h | 37 ++++---------------------------------
 1 file changed, 4 insertions(+), 33 deletions(-)
include/linux/preempt.h +4 -33
index 7d9c1c0..513769b 100644
@@ -56,8 +56,7 @@
 #define PREEMPT_DISABLED	(PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)
 
 /*
- * Disable preemption until the scheduler is running -- use an unconditional
- * value so that it also works on !PREEMPT_COUNT kernels.
+ * Disable preemption until the scheduler is running.
  *
  * Reset by start_kernel()->sched_init()->init_idle()->init_idle_preempt_count().
  */
@@ -69,7 +68,6 @@
  *
  *    preempt_count() == 2*PREEMPT_DISABLE_OFFSET
  *
- * Note: PREEMPT_DISABLE_OFFSET is 0 for !PREEMPT_COUNT kernels.
  * Note: See finish_task_switch().
  */
 #define FORK_PREEMPT_COUNT	(2*PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)
@@ -106,11 +104,7 @@
 /*
  * The preempt_count offset after preempt_disable();
  */
-#if defined(CONFIG_PREEMPT_COUNT)
-# define PREEMPT_DISABLE_OFFSET	PREEMPT_OFFSET
-#else
-# define PREEMPT_DISABLE_OFFSET	0
-#endif
+#define PREEMPT_DISABLE_OFFSET	PREEMPT_OFFSET
 
 /*
  * The preempt_count offset after spin_lock()
@@ -122,8 +116,8 @@
  *
  *  spin_lock_bh()
  *
- * Which need to disable both preemption (CONFIG_PREEMPT_COUNT) and
- * softirqs, such that unlock sequences of:
+ * Which need to disable both preemption and softirqs, such that unlock
+ * sequences of:
  *
  *  spin_unlock();
  *  local_bh_enable();
@@ -164,8 +158,6 @@ extern void preempt_count_sub(int val);
 #define preempt_count_inc() preempt_count_add(1)
 #define preempt_count_dec() preempt_count_sub(1)
 
-#ifdef CONFIG_PREEMPT_COUNT
-
 #define preempt_disable() \
 do { \
 	preempt_count_inc(); \
@@ -231,27 +223,6 @@ do { \
 	__preempt_count_dec(); \
 } while (0)
 
-#else /* !CONFIG_PREEMPT_COUNT */
-
-/*
- * Even if we don't have any preemption, we need preempt disable/enable
- * to be barriers, so that we don't have things like get_user/put_user
- * that can cause faults and scheduling migrate into our preempt-protected
- * region.
- */
-#define preempt_disable()			barrier()
-#define sched_preempt_enable_no_resched()	barrier()
-#define preempt_enable_no_resched()		barrier()
-#define preempt_enable()			barrier()
-#define preempt_check_resched()			do { } while (0)
-
-#define preempt_disable_notrace()		barrier()
-#define preempt_enable_no_resched_notrace()	barrier()
-#define preempt_enable_notrace()		barrier()
-#define preemptible()				0
-
-#endif /* CONFIG_PREEMPT_COUNT */
-
 #ifdef MODULE
 /*
  * Modules have no business playing preemption tricks.
Signature · 1 line
2.9.5
author maintainer
#4 lore raw

PATCH 4/15 lockdep: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

---
 include/linux/lockdep.h | 6 ++----
 lib/Kconfig.debug       | 1 -
 2 files changed, 2 insertions(+), 5 deletions(-)
include/linux/lockdep.h +2 -4
index 6a584b3..879de69 100644
@@ -560,16 +560,14 @@ do {									\
 
 #define lockdep_assert_preemption_enabled()				\
 do {									\
-	WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT)	&&		\
-		     debug_locks			&&		\
+	WARN_ON_ONCE(debug_locks			&&		\
 		     (preempt_count() != 0		||		\
 		      !raw_cpu_read(hardirqs_enabled)));		\
 } while (0)
 
 #define lockdep_assert_preemption_disabled()				\
 do {									\
-	WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT)	&&		\
-		     debug_locks			&&		\
+	WARN_ON_ONCE(debug_locks			&&		\
 		     (preempt_count() == 0		&&		\
 		      raw_cpu_read(hardirqs_enabled)));			\
 } while (0)
lib/Kconfig.debug +0 -1
index f50fbcf..d4d0574 100644
@@ -1161,7 +1161,6 @@ config PROVE_LOCKING
 	select DEBUG_RWSEMS
 	select DEBUG_WW_MUTEX_SLOWPATH
 	select DEBUG_LOCK_ALLOC
-	select PREEMPT_COUNT
 	select TRACE_IRQFLAGS
 	default n
 	help
Signature · 1 line
2.9.5
author maintainer
#5 lore raw

PATCH 5/15 mm/pagemap: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Cc Andrew Morton <[email protected]>
[ paulmck: Fix !SMP build error per kernel test robot feedback. ]
---
 include/linux/pagemap.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
include/linux/pagemap.h +1 -3
index 7de11dc..b3d9d92 100644
@@ -168,9 +168,7 @@ void release_pages(struct page **pages, int nr);
 static inline int __page_cache_add_speculative(struct page *page, int count)
 {
 #ifdef CONFIG_TINY_RCU
-# ifdef CONFIG_PREEMPT_COUNT
-	VM_BUG_ON(!in_atomic() && !irqs_disabled());
-# endif
+	VM_BUG_ON(preemptible());
 	/*
 	 * Preempt must be disabled here - we rely on rcu_read_lock doing
 	 * this for us.
Signature · 1 line
2.9.5
author maintainer
#6 lore raw

PATCH 6/15 locking/bitspinlock: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

---
 include/linux/bit_spinlock.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
include/linux/bit_spinlock.h +1 -3
index bbc4730..1e03d54 100644
@@ -90,10 +90,8 @@ static inline int bit_spin_is_locked(int bitnum, unsigned long *addr)
 {
 #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK)
 	return test_bit(bitnum, addr);
-#elif defined CONFIG_PREEMPT_COUNT
-	return preempt_count();
 #else
-	return 1;
+	return preempt_count();
 #endif
 }
 
Signature · 1 line
2.9.5
author maintainer
#7 lore raw

PATCH 7/15 uaccess: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

---
 include/linux/uaccess.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/uaccess.h +3 -3
index 94b2854..b79a5a9 100644
@@ -230,9 +230,9 @@ static inline bool pagefault_disabled(void)
  *
  * This function should only be used by the fault handlers. Other users should
  * stick to pagefault_disabled().
- * Please NEVER use preempt_disable() to disable the fault handler. With
- * !CONFIG_PREEMPT_COUNT, this is like a NOP. So the handler won't be disabled.
- * in_atomic() will report different values based on !CONFIG_PREEMPT_COUNT.
+ *
+ * Please NEVER use preempt_disable() or local_irq_disable() to disable the
+ * fault handler.
  */
 #define faulthandler_disabled() (pagefault_disabled() || in_atomic())
 
Signature · 1 line
2.9.5
author maintainer
#8 lore raw

PATCH 8/15 sched: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Ben Segall <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 kernel/sched/core.c | 6 +-----
 lib/Kconfig.debug   | 1 -
 2 files changed, 1 insertion(+), 6 deletions(-)
kernel/sched/core.c +1 -5
index 2d95dc3..1c304a1 100644
@@ -3706,8 +3706,7 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev)
 	 * finish_task_switch() for details.
 	 *
 	 * finish_task_switch() will drop rq->lock() and lower preempt_count
-	 * and the preempt_enable() will end up enabling preemption (on
-	 * PREEMPT_COUNT kernels).
+	 * and the preempt_enable() will end up enabling preemption.
 	 */
 
 	rq = finish_task_switch(prev);
@@ -7308,9 +7307,6 @@ void __cant_sleep(const char *file, int line, int preempt_offset)
 	if (irqs_disabled())
 		return;
 
-	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
-		return;
-
 	if (preempt_count() > preempt_offset)
 		return;
 
lib/Kconfig.debug +0 -1
index d4d0574..52af6ad 100644
@@ -1320,7 +1320,6 @@ config DEBUG_LOCKDEP
 
 config DEBUG_ATOMIC_SLEEP
 	bool "Sleep inside atomic section checking"
-	select PREEMPT_COUNT
 	depends on DEBUG_KERNEL
 	help
 	  If you say Y here, various routines which may sleep will become very
Signature · 1 line
2.9.5
author maintainer
#9 lore raw

PATCH 9/15 ARM: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

---
 arch/arm/include/asm/assembler.h   | 11 -----------
 arch/arm/kernel/iwmmxt.S           |  2 --
 arch/arm/mach-ep93xx/crunch-bits.S |  2 --
 3 files changed, 15 deletions(-)
arch/arm/include/asm/assembler.h +0 -11
index feac2c8..fce52eed 100644
@@ -212,7 +212,6 @@
 /*
  * Increment/decrement the preempt count.
  */
-#ifdef CONFIG_PREEMPT_COUNT
 	.macro	inc_preempt_count, ti, tmp
 	ldr	\tmp, [\ti, #TI_PREEMPT]	@ get preempt count
 	add	\tmp, \tmp, #1			@ increment it
@@ -229,16 +228,6 @@
 	get_thread_info \ti
 	dec_preempt_count \ti, \tmp
 	.endm
-#else
-	.macro	inc_preempt_count, ti, tmp
-	.endm
-
-	.macro	dec_preempt_count, ti, tmp
-	.endm
-
-	.macro	dec_preempt_count_ti, ti, tmp
-	.endm
-#endif
 
 #define USERL(l, x...)				\
 9999:	x;					\
arch/arm/kernel/iwmmxt.S +0 -2
index 0dcae78..1f845be 100644
@@ -94,9 +94,7 @@ ENTRY(iwmmxt_task_enable)
 	mov	r2, r2				@ cpwait
 	bl	concan_save
 
-#ifdef CONFIG_PREEMPT_COUNT
 	get_thread_info r10
-#endif
 4:	dec_preempt_count r10, r3
 	ret	r9				@ normal exit from exception
 
arch/arm/mach-ep93xx/crunch-bits.S +0 -2
index fb2dbf7..0aabcf4 100644
@@ -191,9 +191,7 @@ crunch_load:
 	cfldr64		mvdx15, [r0, #CRUNCH_MVDX15]
 
 1:
-#ifdef CONFIG_PREEMPT_COUNT
 	get_thread_info r10
-#endif
 2:	dec_preempt_count r10, r3
 	ret	lr
 
Signature · 1 line
2.9.5
author maintainer
#10 lore raw

PATCH 10/15 xtensa: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

---
 arch/xtensa/kernel/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/xtensa/kernel/entry.S +1 -1
index 703cf62..5a27dd3 100644
@@ -819,7 +819,7 @@ ENTRY(debug_exception)
 	 * preemption if we have HW breakpoints to preserve DEBUGCAUSE.DBNUM
 	 * meaning.
 	 */
-#if defined(CONFIG_PREEMPT_COUNT) && defined(CONFIG_HAVE_HW_BREAKPOINT)
+#ifdef CONFIG_HAVE_HW_BREAKPOINT
 	GET_THREAD_INFO(a2, a1)
 	l32i	a3, a2, TI_PRE_COUNT
 	addi	a3, a3, 1
Signature · 1 line
2.9.5
author maintainer
#11 lore raw

PATCH 11/15 drm/i915: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

---
 drivers/gpu/drm/i915/Kconfig.debug | 1 -
 drivers/gpu/drm/i915/i915_utils.h  | 3 +--
 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/gpu/drm/i915/Kconfig.debug +0 -1
index 1cb28c2..17d9b00 100644
@@ -20,7 +20,6 @@ config DRM_I915_DEBUG
 	bool "Enable additional driver debugging"
 	depends on DRM_I915
 	select DEBUG_FS
-	select PREEMPT_COUNT
 	select I2C_CHARDEV
 	select STACKDEPOT
 	select DRM_DP_AUX_CHARDEV
drivers/gpu/drm/i915/i915_utils.h +1 -2
index 5477337..ecfed86 100644
@@ -337,8 +337,7 @@ wait_remaining_ms_from_jiffies(unsigned long timestamp_jiffies, int to_wait_ms)
 						   (Wmax))
 #define wait_for(COND, MS)		_wait_for((COND), (MS) * 1000, 10, 1000)
 
-/* If CONFIG_PREEMPT_COUNT is disabled, in_atomic() always reports false. */
-#if defined(CONFIG_DRM_I915_DEBUG) && defined(CONFIG_PREEMPT_COUNT)
+#ifdef CONFIG_DRM_I915_DEBUG
 # define _WAIT_FOR_ATOMIC_CHECK(ATOMIC) WARN_ON_ONCE((ATOMIC) && !in_atomic())
 #else
 # define _WAIT_FOR_ATOMIC_CHECK(ATOMIC) do { } while (0)
Signature · 6 lines
2.9.5

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
maintainer Reviewed-by on v1 11/15
#12 lore raw

Re: [PATCH tip/core/rcu 11/15] drm/i915: Cleanup PREEMPT_COUNT leftovers

Quoting [email protected] (2020-09-29 02:30:58)

Quoted 4 lines

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Change looks fine:

Are you looking for us to merge or merge through another tree?

If us, did the base patch always enabling PREEMPT_COUNT go into 5.9 or is it heading to 5.10? We can queue this earliest for 5.11 as drm-next closed for 5.10 at week of -rc5.

Regards, Joonas

Quoted 43 lines
---
 drivers/gpu/drm/i915/Kconfig.debug | 1 -
 drivers/gpu/drm/i915/i915_utils.h  | 3 +--
 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/gpu/drm/i915/Kconfig.debug +0 -1
index 1cb28c2..17d9b00 100644
@@ -20,7 +20,6 @@ config DRM_I915_DEBUG
        bool "Enable additional driver debugging"
        depends on DRM_I915
        select DEBUG_FS
-       select PREEMPT_COUNT
        select I2C_CHARDEV
        select STACKDEPOT
        select DRM_DP_AUX_CHARDEV
drivers/gpu/drm/i915/i915_utils.h +1 -2
index 5477337..ecfed86 100644
@@ -337,8 +337,7 @@ wait_remaining_ms_from_jiffies(unsigned long timestamp_jiffies, int to_wait_ms)
                                                   (Wmax))
 #define wait_for(COND, MS)             _wait_for((COND), (MS) * 1000, 10, 1000)
 
-/* If CONFIG_PREEMPT_COUNT is disabled, in_atomic() always reports false. */
-#if defined(CONFIG_DRM_I915_DEBUG) && defined(CONFIG_PREEMPT_COUNT)
+#ifdef CONFIG_DRM_I915_DEBUG
 # define _WAIT_FOR_ATOMIC_CHECK(ATOMIC) WARN_ON_ONCE((ATOMIC) && !in_atomic())
 #else
 # define _WAIT_FOR_ATOMIC_CHECK(ATOMIC) do { } while (0)
Signature · 1 line
2.9.5
#13 lore raw

Re: [PATCH tip/core/rcu 11/15] drm/i915: Cleanup PREEMPT_COUNT leftovers

Quoted 6 lines · Joonas Lahtinen

Quoting [email protected] (2020-09-29 02:30:58)

Quoted 2 lines

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Change looks fine:

Quoted 7 lines

Are you looking for us to merge or merge through another tree?

If us, did the base patch always enabling PREEMPT_COUNT go into 5.9 or is it heading to 5.10? We can queue this earliest for 5.11 as drm-next closed for 5.10 at week of -rc5.

If at all it goes through rcu/tip because it depends on the earlier patches.

Thanks,

tglx

author maintainer Applied
#14 lore raw

Re: [PATCH tip/core/rcu 11/15] drm/i915: Cleanup PREEMPT_COUNT leftovers

Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Joonas Lahtinen

Quoting [email protected] (2020-09-29 02:30:58)

Quoted 2 lines

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Change looks fine:

Applied, thank you!

Quoted 7 lines
Quoted 5 lines

Are you looking for us to merge or merge through another tree?

If us, did the base patch always enabling PREEMPT_COUNT go into 5.9 or is it heading to 5.10? We can queue this earliest for 5.11 as drm-next closed for 5.10 at week of -rc5.

If at all it goes through rcu/tip because it depends on the earlier patches.

I was figuring on sending a pull request later today, Pacific Time.

Thanx, Paul

author maintainer
#15 lore raw

PATCH 12/15 rcutorture: Cleanup PREEMPT_COUNT leftovers

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

CONFIG_PREEMPT_COUNT is now unconditionally enabled and will be removed. Cleanup the leftovers before doing so.

Cc "Paul E. McKenney" <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Shuah Khan <[email protected]>
---
 tools/testing/selftests/rcutorture/configs/rcu/SRCU-t            | 1 -
 tools/testing/selftests/rcutorture/configs/rcu/SRCU-u            | 1 -
 tools/testing/selftests/rcutorture/configs/rcu/TINY01            | 1 -
 tools/testing/selftests/rcutorture/doc/TINY_RCU.txt              | 5 ++---
 tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt      | 1 -
 tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h | 1 -
 6 files changed, 2 insertions(+), 8 deletions(-)
tools/testing/selftests/rcutorture/configs/rcu/SRCU-t +0 -1
index 6c78022..553cf65 100644
@@ -7,4 +7,3 @@ CONFIG_RCU_TRACE=n
 CONFIG_DEBUG_LOCK_ALLOC=n
 CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
 CONFIG_DEBUG_ATOMIC_SLEEP=y
-#CHECK#CONFIG_PREEMPT_COUNT=y
tools/testing/selftests/rcutorture/configs/rcu/SRCU-u +0 -1
index c15ada8..99563da 100644
@@ -7,4 +7,3 @@ CONFIG_RCU_TRACE=n
 CONFIG_DEBUG_LOCK_ALLOC=y
 CONFIG_PROVE_LOCKING=y
 CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
-CONFIG_PREEMPT_COUNT=n
tools/testing/selftests/rcutorture/configs/rcu/TINY01 +0 -1
index 6db705e..9b22b8e 100644
@@ -10,4 +10,3 @@ CONFIG_RCU_TRACE=n
 #CHECK#CONFIG_RCU_STALL_COMMON=n
 CONFIG_DEBUG_LOCK_ALLOC=n
 CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
-CONFIG_PREEMPT_COUNT=n
tools/testing/selftests/rcutorture/doc/TINY_RCU.txt +2 -3
index a75b169..d30cedf 100644
@@ -3,11 +3,10 @@ This document gives a brief rationale for the TINY_RCU test cases.
 
 Kconfig Parameters:
 
-CONFIG_DEBUG_LOCK_ALLOC -- Do all three and none of the three.
-CONFIG_PREEMPT_COUNT
+CONFIG_DEBUG_LOCK_ALLOC -- Do both and none of the two.
 CONFIG_RCU_TRACE
 
-The theory here is that randconfig testing will hit the other six possible
+The theory here is that randconfig testing will hit the other two possible
 combinations of these parameters.
 
 
tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt +0 -1
index 1b96d68..cfdd48f 100644
@@ -43,7 +43,6 @@ CONFIG_64BIT
 
 	Used only to check CONFIG_RCU_FANOUT value, inspection suffices.
 
-CONFIG_PREEMPT_COUNT
 CONFIG_PREEMPT_RCU
 
 	Redundant with CONFIG_PREEMPT, ignore.
tools/testing/selftests/rcutorture/formal/srcu-cbmc/src/config.h +0 -1
index 283d710..d0d485d 100644
@@ -8,7 +8,6 @@
 #undef CONFIG_HOTPLUG_CPU
 #undef CONFIG_MODULES
 #undef CONFIG_NO_HZ_FULL_SYSIDLE
-#undef CONFIG_PREEMPT_COUNT
 #undef CONFIG_PREEMPT_RCU
 #undef CONFIG_PROVE_RCU
 #undef CONFIG_RCU_NOCB_CPU
Signature · 1 line
2.9.5
author maintainer
#16 lore raw

PATCH 13/15 preempt: Remove PREEMPT_COUNT from Kconfig

In reply to Paul E. McKenney

From: Thomas Gleixner <[email protected]>

All conditionals and irritations are gone.

---
 kernel/Kconfig.preempt | 3 ---
 1 file changed, 3 deletions(-)
kernel/Kconfig.preempt +0 -3
index 3f4712f..120b63f 100644
@@ -74,8 +74,5 @@ config PREEMPT_RT
 
 endchoice
 
-config PREEMPT_COUNT
-       def_bool y
-
 config PREEMPTION
        bool
Signature · 1 line
2.9.5
author maintainer
#17 lore raw

PATCH 14/15 rcu/tree: Allocate a page when caller is preemptible

In reply to Paul E. McKenney

From: "Uladzislau Rezki (Sony)" <[email protected]>

The current memory-allocation interface poses the following challenges:

a) In kernels built with CONFIG_PROVE_RAW_LOCK_NESTING, lockdep
complains ("BUG: Invalid wait context"). This complaint is due to the memory allocator acquiring non-raw spinlocks while a raw spinlocks is held. This problem can also arise if kvfree_rcu() is invoked while holding a raw spinlock.

b) In -rt kernels built with CONFIG_PREEMPT_RT, the situation
described in (a) above results in an attempt to acquire a sleeplock while holding a spinlock, which is of course forbidden. This can lead to "BUG: scheduling while atomic".

c) Please note that call_rcu() is invoked from raw atomic context,
so that kfree_rcu() and kvfree_rcu() are therefore also expected to be callable from atomic raw context as well.

However given that CONFIG_PREEMPT_COUNT is unconditionally enabled by the earlier commits in this series, the preemptible() macro now properly detects preempt-disable code regions even in kernels built with CONFIG_PREEMPT_NONE.

This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu(). If !preemptible(), then allocation is not possible, and kvfree_rcu() falls back to using the less cache-friendly rcu_head approach. Even when preemptible(), the caller might be involved in reclaim, so the GFP_ flags used by double-argument kvfree_rcu() must avoid invoking reclaim processing.

Note that single-argument kvfree_rcu() must be invoked in sleepable contexts, and that its fallback is the relatively high latency synchronize_rcu(). Single-argument kvfree_rcu() therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.

---
 kernel/rcu/tree.c | 70 ++++++++++++++++---------------------------------------
 1 file changed, 20 insertions(+), 50 deletions(-)
kernel/rcu/tree.c +20 -50
index 8ce77d9..cc998d7 100644
@@ -3166,7 +3166,7 @@ static void kfree_rcu_work(struct work_struct *work)
 			krc_this_cpu_unlock(krcp, flags);
 
 			if (bkvhead[i])
-				free_page((unsigned long) bkvhead[i]);
+				kfree(bkvhead[i]);
 
 			cond_resched_tasks_rcu_qs();
 		}
@@ -3291,43 +3291,28 @@ static void kfree_rcu_monitor(struct work_struct *work)
 }
 
 static inline bool
-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
+add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
+	unsigned long *flags, void *ptr, bool can_sleep)
 {
 	struct kvfree_rcu_bulk_data *bnode;
+	bool can_alloc_page = preemptible();
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;
 	int idx;
 
-	if (unlikely(!krcp->initialized))
+	*krcp = krc_this_cpu_lock(flags);
+	if (unlikely(!(*krcp)->initialized))
 		return false;
 
-	lockdep_assert_held(&krcp->lock);
 	idx = !!is_vmalloc_addr(ptr);
 
 	/* Check if a new block is required. */
-	if (!krcp->bkvhead[idx] ||
-			krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
-		bnode = get_cached_bnode(krcp);
-		if (!bnode) {
-			/*
-			 * To keep this path working on raw non-preemptible
-			 * sections, prevent the optional entry into the
-			 * allocator as it uses sleeping locks. In fact, even
-			 * if the caller of kfree_rcu() is preemptible, this
-			 * path still is not, as krcp->lock is a raw spinlock.
-			 * With additional page pre-allocation in the works,
-			 * hitting this return is going to be much less likely.
-			 */
-			if (IS_ENABLED(CONFIG_PREEMPT_RT))
-				return false;
-
-			/*
-			 * NOTE: For one argument of kvfree_rcu() we can
-			 * drop the lock and get the page in sleepable
-			 * context. That would allow to maintain an array
-			 * for the CONFIG_PREEMPT_RT as well if no cached
-			 * pages are available.
-			 */
-			bnode = (struct kvfree_rcu_bulk_data *)
-				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
+	if (!(*krcp)->bkvhead[idx] ||
+			(*krcp)->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
+		bnode = get_cached_bnode(*krcp);
+		if (!bnode && can_alloc_page) {
+			krc_this_cpu_unlock(*krcp, *flags);
+			bnode = kmalloc(PAGE_SIZE, gfp);
+			*krcp = krc_this_cpu_lock(flags);
 		}
 
 		/* Switch to emergency path. */
@@ -3336,15 +3321,15 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
 
 		/* Initialize the new block. */
 		bnode->nr_records = 0;
-		bnode->next = krcp->bkvhead[idx];
+		bnode->next = (*krcp)->bkvhead[idx];
 
 		/* Attach it to the head. */
-		krcp->bkvhead[idx] = bnode;
+		(*krcp)->bkvhead[idx] = bnode;
 	}
 
 	/* Finally insert. */
-	krcp->bkvhead[idx]->records
-		[krcp->bkvhead[idx]->nr_records++] = ptr;
+	(*krcp)->bkvhead[idx]->records
+		[(*krcp)->bkvhead[idx]->nr_records++] = ptr;
 
 	return true;
 }
@@ -3382,24 +3367,20 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 		ptr = (unsigned long *) func;
 	}
 
-	krcp = krc_this_cpu_lock(&flags);
-
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
 		WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
 			  __func__, head);
 
-		// Mark as success and leave.
-		success = true;
-		goto unlock_return;
+		return;
 	}
 
 	/*
 	 * Under high memory pressure GFP_NOWAIT can fail,
 	 * in that case the emergency path is maintained.
 	 */
-	success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
+	success = add_ptr_to_bulk_krc_lock(&krcp, &flags, ptr, !head);
 	if (!success) {
 		if (head == NULL)
 			// Inline if kvfree_rcu(one_arg) call.
@@ -4394,23 +4375,12 @@ static void __init kfree_rcu_batch_init(void)
 
 	for_each_possible_cpu(cpu) {
 		struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
-		struct kvfree_rcu_bulk_data *bnode;
 
 		for (i = 0; i < KFREE_N_BATCHES; i++) {
 			INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work);
 			krcp->krw_arr[i].krcp = krcp;
 		}
 
-		for (i = 0; i < rcu_min_cached_objs; i++) {
-			bnode = (struct kvfree_rcu_bulk_data *)
-				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
-
-			if (bnode)
-				put_cached_bnode(krcp, bnode);
-			else
-				pr_err("Failed to preallocate for %d CPU!\n", cpu);
-		}
-
 		INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor);
 		krcp->initialized = true;
 	}
Signature · 1 line
2.9.5
#18 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]

Quoted 2 lines

This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu().

This deserves a comment. Because GFP_ATOMIC is possible for many !preemptible() contexts. It is the raw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.

Quoted 5 lines

If !preemptible(),
then allocation is not possible, and kvfree_rcu() falls back to using the less cache-friendly rcu_head approach. Even when preemptible(), the caller might be involved in reclaim, so the GFP_ flags used by double-argument kvfree_rcu() must avoid invoking reclaim processing.

Could you be more specific? Is this about being called directly in the reclaim context and you want to prevent a recursion? If that is the case, do you really need to special case this in any way? Any memory reclaim will set PF_MEMALLOC so allocations called from that context will not perform reclaim. So if you are called from the reclaim directly then you might want to do GFP_KERNEL | __GFP_NOMEMALLOC | __GFP_NOWARN. That should handle both from-the-recalim and outside of reclaim contexts just fine (assuming you don't allocated from !preemptible()) context.

Quoted 5 lines

Note that single-argument kvfree_rcu() must be invoked in sleepable contexts, and that its fallback is the relatively high latency synchronize_rcu(). Single-argument kvfree_rcu() therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.

[...]

kernel/rcu/tree.c static void kfree_rcu_monitor(struct work_struct *work) · line 3299

Quoted 8 lines

static inline bool

kernel/rcu/tree.c +4 -1
-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
+add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
+	unsigned long *flags, void *ptr, bool can_sleep)
 {
 	struct kvfree_rcu_bulk_data *bnode;
+	bool can_alloc_page = preemptible();
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;

This is quite confusing IMHO. At least without a further explanation. can_sleep is not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?

Quoted 32 lines
kernel/rcu/tree.c +2 -23
 	int idx;
 
-	if (unlikely(!krcp->initialized))
+	*krcp = krc_this_cpu_lock(flags);
+	if (unlikely(!(*krcp)->initialized))
 		return false;
 
-	lockdep_assert_held(&krcp->lock);
 	idx = !!is_vmalloc_addr(ptr);
 
 	/* Check if a new block is required. */
-	if (!krcp->bkvhead[idx] ||
-			krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
-		bnode = get_cached_bnode(krcp);
-		if (!bnode) {
-			/*
-			 * To keep this path working on raw non-preemptible
-			 * sections, prevent the optional entry into the
-			 * allocator as it uses sleeping locks. In fact, even
-			 * if the caller of kfree_rcu() is preemptible, this
-			 * path still is not, as krcp->lock is a raw spinlock.
-			 * With additional page pre-allocation in the works,
-			 * hitting this return is going to be much less likely.
-			 */
-			if (IS_ENABLED(CONFIG_PREEMPT_RT))
-				return false;
-
-			/*
-			 * NOTE: For one argument of kvfree_rcu() we can
-			 * drop the lock and get the page in sleepable
-			 * context. That would allow to maintain an array
-			 * for the CONFIG_PREEMPT_RT as well if no cached

kernel/rcu/tree.c static void kfree_rcu_monitor(struct work_struct *work) · line 3314

Quoted 10 lines
kernel/rcu/tree.c +6 -4
-			 * pages are available.
-			 */
-			bnode = (struct kvfree_rcu_bulk_data *)
-				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
+	if (!(*krcp)->bkvhead[idx] ||
+			(*krcp)->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
+		bnode = get_cached_bnode(*krcp);
+		if (!bnode && can_alloc_page) {
+			krc_this_cpu_unlock(*krcp, *flags);
+			bnode = kmalloc(PAGE_SIZE, gfp);

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#19 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

Quoted 2 lines · Michal Hocko

On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]

Apologies for the delay, but today has not been boring.

Quoted 7 lines
Quoted 2 lines

This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu().

This deserves a comment. Because GFP_ATOMIC is possible for many !preemptible() contexts. It is the raw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.

Good point, and so I have added the following as a header comment to the add_ptr_to_bulk_krc_lock() function:

// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() // state specified by flags. If can_sleep is true, the caller must // be schedulable and not be holding any locks or mutexes that might be // acquired by the memory allocator or anything that it might invoke. // If !can_sleep, then if !preemptible() no allocation will be undertaken, // otherwise the allocation will use GFP_ATOMIC to avoid the remainder of // the aforementioned deadlock possibilities. Returns true iff ptr was // successfully recorded, else the caller must use a fallback.

The updated patch is shown below.

Of course, to Vlastimil's point, lockless access to the allocator would significantly reduce the level of confusion posed by this code. But that is a separate issue at the moment.

Quoted 6 lines
Quoted 5 lines

If !preemptible(),
then allocation is not possible, and kvfree_rcu() falls back to using the less cache-friendly rcu_head approach. Even when preemptible(), the caller might be involved in reclaim, so the GFP_ flags used by double-argument kvfree_rcu() must avoid invoking reclaim processing.

Quoted 8 lines

Could you be more specific? Is this about being called directly in the reclaim context and you want to prevent a recursion? If that is the case, do you really need to special case this in any way? Any memory reclaim will set PF_MEMALLOC so allocations called from that context will not perform reclaim. So if you are called from the reclaim directly then you might want to do GFP_KERNEL | __GFP_NOMEMALLOC | __GFP_NOWARN. That should handle both from-the-recalim and outside of reclaim contexts just fine (assuming you don't allocated from !preemptible()) context.

My thought was to point you at Daniel Vetter, but you already got in touch with him, so thank you for that!

Quoted 6 lines
Quoted 5 lines

Note that single-argument kvfree_rcu() must be invoked in sleepable contexts, and that its fallback is the relatively high latency synchronize_rcu(). Single-argument kvfree_rcu() therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.

[...]

Quoted 12 lines

Review

Quoted 8 lines

static inline bool

(quoted diff) +4 -1
-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
+add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
+	unsigned long *flags, void *ptr, bool can_sleep)
 {
 	struct kvfree_rcu_bulk_data *bnode;
+	bool can_alloc_page = preemptible();
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;

This is quite confusing IMHO. At least without a further explanation. can_sleep is not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?

No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.

The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().

Quoted 37 lines
Quoted 37 lines
(quoted diff) +3 -27
 	int idx;
 
-	if (unlikely(!krcp->initialized))
+	*krcp = krc_this_cpu_lock(flags);
+	if (unlikely(!(*krcp)->initialized))
 		return false;
 
-	lockdep_assert_held(&krcp->lock);
 	idx = !!is_vmalloc_addr(ptr);
 
 	/* Check if a new block is required. */
-	if (!krcp->bkvhead[idx] ||
-			krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
-		bnode = get_cached_bnode(krcp);
-		if (!bnode) {
-			/*
-			 * To keep this path working on raw non-preemptible
-			 * sections, prevent the optional entry into the
-			 * allocator as it uses sleeping locks. In fact, even
-			 * if the caller of kfree_rcu() is preemptible, this
-			 * path still is not, as krcp->lock is a raw spinlock.
-			 * With additional page pre-allocation in the works,
-			 * hitting this return is going to be much less likely.
-			 */
-			if (IS_ENABLED(CONFIG_PREEMPT_RT))
-				return false;
-
-			/*
-			 * NOTE: For one argument of kvfree_rcu() we can
-			 * drop the lock and get the page in sleepable
-			 * context. That would allow to maintain an array
-			 * for the CONFIG_PREEMPT_RT as well if no cached
-			 * pages are available.
-			 */
-			bnode = (struct kvfree_rcu_bulk_data *)
-				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
+	if (!(*krcp)->bkvhead[idx] ||
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+			(*krcp)->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
+		bnode = get_cached_bnode(*krcp);
+		if (!bnode && can_alloc_page) {
+			krc_this_cpu_unlock(*krcp, *flags);
+			bnode = kmalloc(PAGE_SIZE, gfp);

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

Again, please see below for the updated patch. The only change from the earlier verison is the addition of the header comment mentioned above.

Thanx, Paul

------------------------------------------------------------------------

commit 3a8e10ea2a2ecf59a33d7d80ad6c4372c0c27f78

Author: Uladzislau Rezki (Sony) <[email protected]>
Date:   Tue Sep 22 21:06:22 2020 +0200

rcu/tree: Allocate a page when caller is preemptible

The current memory-allocation interface poses the following challenges:

a) In kernels built with CONFIG_PROVE_RAW_LOCK_NESTING, lockdep
complains ("BUG: Invalid wait context"). This complaint is due to the memory allocator acquiring non-raw spinlocks while a raw spinlocks is held. This problem can also arise if kvfree_rcu() is invoked while holding a raw spinlock.

b) In -rt kernels built with CONFIG_PREEMPT_RT, the situation
described in (a) above results in an attempt to acquire a sleeplock while holding a spinlock, which is of course forbidden. This can lead to "BUG: scheduling while atomic".

c) Please note that call_rcu() is invoked from raw atomic context,
so that kfree_rcu() and kvfree_rcu() are therefore also expected to be callable from atomic raw context as well.

However given that CONFIG_PREEMPT_COUNT is unconditionally enabled by the earlier commits in this series, the preemptible() macro now properly detects preempt-disable code regions even in kernels built with CONFIG_PREEMPT_NONE.

This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu(). If !preemptible(), then allocation is not possible, and kvfree_rcu() falls back to using the less cache-friendly rcu_head approach. Even when preemptible(), the caller might be involved in reclaim, so the GFP_ flags used by double-argument kvfree_rcu() must avoid invoking reclaim processing.

Note that single-argument kvfree_rcu() must be invoked in sleepable contexts, and that its fallback is the relatively high latency synchronize_rcu(). Single-argument kvfree_rcu() therefore uses GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the memory allocator.

Fixes 3042f83f19be ("rcu: Support reclaim for head-less object")
[ paulmck: Add add_ptr_to_bulk_krc_lock header comment per Michal Hocko. ]
kernel/rcu/tree.c +28 -50
index 8ce77d9..39ac930 100644
@@ -3166,7 +3166,7 @@ static void kfree_rcu_work(struct work_struct *work)
 			krc_this_cpu_unlock(krcp, flags);
 
 			if (bkvhead[i])
-				free_page((unsigned long) bkvhead[i]);
+				kfree(bkvhead[i]);
 
 			cond_resched_tasks_rcu_qs();
 		}
@@ -3290,44 +3290,37 @@ static void kfree_rcu_monitor(struct work_struct *work)
 		raw_spin_unlock_irqrestore(&krcp->lock, flags);
 }
 
+// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock()
+// state specified by flags.  If can_sleep is true, the caller must
+// be schedulable and not be holding any locks or mutexes that might be
+// acquired by the memory allocator or anything that it might invoke.
+// If !can_sleep, then if !preemptible() no allocation will be undertaken,
+// otherwise the allocation will use GFP_ATOMIC to avoid the remainder of
+// the aforementioned deadlock possibilities.  Returns true iff ptr was
+// successfully recorded, else the caller must use a fallback.
 static inline bool
-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
+add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
+	unsigned long *flags, void *ptr, bool can_sleep)
 {
 	struct kvfree_rcu_bulk_data *bnode;
+	bool can_alloc_page = preemptible();
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;
 	int idx;
 
-	if (unlikely(!krcp->initialized))
+	*krcp = krc_this_cpu_lock(flags);
+	if (unlikely(!(*krcp)->initialized))
 		return false;
 
-	lockdep_assert_held(&krcp->lock);
 	idx = !!is_vmalloc_addr(ptr);
 
 	/* Check if a new block is required. */
-	if (!krcp->bkvhead[idx] ||
-			krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
-		bnode = get_cached_bnode(krcp);
-		if (!bnode) {
-			/*
-			 * To keep this path working on raw non-preemptible
-			 * sections, prevent the optional entry into the
-			 * allocator as it uses sleeping locks. In fact, even
-			 * if the caller of kfree_rcu() is preemptible, this
-			 * path still is not, as krcp->lock is a raw spinlock.
-			 * With additional page pre-allocation in the works,
-			 * hitting this return is going to be much less likely.
-			 */
-			if (IS_ENABLED(CONFIG_PREEMPT_RT))
-				return false;
-
-			/*
-			 * NOTE: For one argument of kvfree_rcu() we can
-			 * drop the lock and get the page in sleepable
-			 * context. That would allow to maintain an array
-			 * for the CONFIG_PREEMPT_RT as well if no cached
-			 * pages are available.
-			 */
-			bnode = (struct kvfree_rcu_bulk_data *)
-				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
+	if (!(*krcp)->bkvhead[idx] ||
+			(*krcp)->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
+		bnode = get_cached_bnode(*krcp);
+		if (!bnode && can_alloc_page) {
+			krc_this_cpu_unlock(*krcp, *flags);
+			bnode = kmalloc(PAGE_SIZE, gfp);
+			*krcp = krc_this_cpu_lock(flags);
 		}
 
 		/* Switch to emergency path. */
@@ -3336,15 +3329,15 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
 
 		/* Initialize the new block. */
 		bnode->nr_records = 0;
-		bnode->next = krcp->bkvhead[idx];
+		bnode->next = (*krcp)->bkvhead[idx];
 
 		/* Attach it to the head. */
-		krcp->bkvhead[idx] = bnode;
+		(*krcp)->bkvhead[idx] = bnode;
 	}
 
 	/* Finally insert. */
-	krcp->bkvhead[idx]->records
-		[krcp->bkvhead[idx]->nr_records++] = ptr;
+	(*krcp)->bkvhead[idx]->records
+		[(*krcp)->bkvhead[idx]->nr_records++] = ptr;
 
 	return true;
 }
@@ -3382,24 +3375,20 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 		ptr = (unsigned long *) func;
 	}
 
-	krcp = krc_this_cpu_lock(&flags);
-
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
 		WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
 			  __func__, head);
 
-		// Mark as success and leave.
-		success = true;
-		goto unlock_return;
+		return;
 	}
 
 	/*
 	 * Under high memory pressure GFP_NOWAIT can fail,
 	 * in that case the emergency path is maintained.
 	 */
-	success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
+	success = add_ptr_to_bulk_krc_lock(&krcp, &flags, ptr, !head);
 	if (!success) {
 		if (head == NULL)
 			// Inline if kvfree_rcu(one_arg) call.
@@ -4394,23 +4383,12 @@ static void __init kfree_rcu_batch_init(void)
 
 	for_each_possible_cpu(cpu) {
 		struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
-		struct kvfree_rcu_bulk_data *bnode;
 
 		for (i = 0; i < KFREE_N_BATCHES; i++) {
 			INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work);
 			krcp->krw_arr[i].krcp = krcp;
 		}
 
-		for (i = 0; i < rcu_min_cached_objs; i++) {
-			bnode = (struct kvfree_rcu_bulk_data *)
-				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
-
-			if (bnode)
-				put_cached_bnode(krcp, bnode);
-			else
-				pr_err("Failed to preallocate for %d CPU!\n", cpu);
-		}
-
 		INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor);
 		krcp->initialized = true;
 	}
#20 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

Quoted 17 lines · Paul E. McKenney
Quoted 2 lines · Michal Hocko

On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]

Apologies for the delay, but today has not been boring.

Quoted 7 lines
Quoted 2 lines

This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu().

This deserves a comment. Because GFP_ATOMIC is possible for many !preemptible() contexts. It is the raw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.

Good point, and so I have added the following as a header comment to the add_ptr_to_bulk_krc_lock() function:

Quoted 8 lines

// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() // state specified by flags. If can_sleep is true, the caller must // be schedulable and not be holding any locks or mutexes that might be // acquired by the memory allocator or anything that it might invoke. // If !can_sleep, then if !preemptible() no allocation will be undertaken, // otherwise the allocation will use GFP_ATOMIC to avoid the remainder of // the aforementioned deadlock possibilities. Returns true iff ptr was // successfully recorded, else the caller must use a fallback.

OK, not trivial to follow but at least verbose enough to understand the intention after some mulling. Definitely an improvement, thanks!

[...]

Quoted 19 lines
Quoted 11 lines

kernel/rcu/tree.c static void kfree_rcu_monitor(struct work_struct *work) · line 3307

Quoted 7 lines
kernel/rcu/tree.c +4 -1
-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
+add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
+	unsigned long *flags, void *ptr, bool can_sleep)
 {
 	struct kvfree_rcu_bulk_data *bnode;
+	bool can_alloc_page = preemptible();
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;

This is quite confusing IMHO. At least without a further explanation. can_sleep is not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?

No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.

Quoted 3 lines

The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().

OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need __GFP_NOMEMALLOC as well.

[...]

Quoted 6 lines
Quoted 2 lines

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.

Signature · 2 lines
Michal Hocko
SUSE Labs
maintainer
#21 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Quoted 8 lines
Quoted 3 lines

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.

I do not have a strong opinion here but i would tend to using of the page allocator directly. Because we operate on a page level where the page allocator is supposed to be used. From the other hand i saw a slightly better performance in case of SLAB only. I think that is because of
a) slab caches; b) more objects which can be cached. But i have one more concern about using of kmalloc(). That is about fragmentations it can cause with PAGE_SIZE requests.

As for SLUB it was identical whereas the SLOB i din not check.

--
Vlad Rezki

author maintainer
#22 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

In reply to Michal Hocko
Quoted 27 lines · Michal Hocko
Quoted 17 lines · Paul E. McKenney
Quoted 2 lines · Michal Hocko

On Mon 28-09-20 16:31:01, [email protected] wrote:
[...]

Apologies for the delay, but today has not been boring.

Quoted 7 lines
Quoted 2 lines

This commit therefore uses preemptible() to determine whether allocation is possible at all for double-argument kvfree_rcu().

This deserves a comment. Because GFP_ATOMIC is possible for many !preemptible() contexts. It is the raw_spin_lock, NMIs and likely few others that are a problem. You are taking a conservative approach which is fine but it would be good to articulate that explicitly.

Good point, and so I have added the following as a header comment to the add_ptr_to_bulk_krc_lock() function:

Quoted 8 lines

// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock() // state specified by flags. If can_sleep is true, the caller must // be schedulable and not be holding any locks or mutexes that might be // acquired by the memory allocator or anything that it might invoke. // If !can_sleep, then if !preemptible() no allocation will be undertaken, // otherwise the allocation will use GFP_ATOMIC to avoid the remainder of // the aforementioned deadlock possibilities. Returns true iff ptr was // successfully recorded, else the caller must use a fallback.

Quoted 2 lines

OK, not trivial to follow but at least verbose enough to understand the intention after some mulling. Definitely an improvement, thanks!

Glad it helped! With some luck, perhaps it will improve with time...

Quoted 20 lines

[...]

Quoted 19 lines
Quoted 11 lines

Review

Quoted 7 lines
(quoted diff) +4 -1
-kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
+add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
+	unsigned long *flags, void *ptr, bool can_sleep)
 {
 	struct kvfree_rcu_bulk_data *bnode;
+	bool can_alloc_page = preemptible();
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;

This is quite confusing IMHO. At least without a further explanation. can_sleep is not as much about sleeping as it is about the reclaim recursion AFAIU your changelog, right?

No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.

Quoted 7 lines
Quoted 3 lines

The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().

OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need __GFP_NOMEMALLOC as well.

Right now the restriction is that single-argument (AKA can_sleep==true) kvfree_rcu() cannot be invoked from memory reclaim context.

But would adding __GFP_NOMEMALLOC to the can_sleep==true GFP_ flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).

Quoted 5 lines

[...]

Quoted 3 lines
Quoted 2 lines

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Quoted 7 lines
Quoted 3 lines

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.

OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.

You mentioned alloc_pages(). I reverted to __get_free_pages(), but alloc_pages() of course looks nicer. What are the tradeoffs between __get_free_pages() and alloc_pages()?

Thanx, Paul

------------------------------------------------------------------------

commit 490b638d7c241ac06cee168ccf8688bb8b872478

Author: Paul E. McKenney <[email protected]>
Date:   Wed Sep 30 16:16:39 2020 -0700

kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.

The advantages of using kmalloc() and kfree() are a possible small speedup on CONFIG_SLAB=y systems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using __get_free_page() and free_page() are a possible reduction in fragmentation and direct access to the buddy allocator.

To help settle the question as to which to use, this commit switches from kmalloc() and kfree() to __get_free_page() and free_page().

kernel/rcu/tree.c +3 -2
index 2886e81..242f0f0 100644
@@ -3225,7 +3225,8 @@ static void kfree_rcu_work(struct work_struct *work)
 				bkvhead[i] = NULL;
 			krc_this_cpu_unlock(krcp, flags);
 
-			kfree(bkvhead[i]);
+			if (bkvhead[i])
+				free_page((unsigned long)bkvhead[i]);
 
 			cond_resched_tasks_rcu_qs();
 		}
@@ -3378,7 +3379,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 		bnode = get_cached_bnode(*krcp);
 		if (!bnode && can_alloc_page) {
 			krc_this_cpu_unlock(*krcp, *flags);
-			bnode = kmalloc(PAGE_SIZE, gfp);
+			bnode = (struct kvfree_rcu_bulk_data *)__get_free_page(gfp);
 			*krcp = krc_this_cpu_lock(flags);
 		}
 
#23 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

Quoted 2 lines · Paul E. McKenney
Quoted 1 line · Michal Hocko

On Tue 29-09-20 18:53:27, Paul E. McKenney wrote:

[...]

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines

No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.

Quoted 7 lines
Quoted 3 lines

The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().

OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need __GFP_NOMEMALLOC as well.

Quoted 7 lines

Right now the restriction is that single-argument (AKA can_sleep==true) kvfree_rcu() cannot be invoked from memory reclaim context.

But would adding __GFP_NOMEMALLOC to the can_sleep==true GFP_ flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).

It would be quite daring to claim it will be ok but it will certainly be less problematic. Adding the flag will not hurt in any case. As this is a shared called that might be called from many contexts I think it will be safer to have it there. The justification is that it will prevent consumption of memory reserves from MEMALLOC contexts.

Quoted 10 lines
Quoted 10 lines

[...]

Quoted 6 lines
Quoted 2 lines

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

Quoted 8 lines
Quoted 3 lines

alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.

OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.

It doesn't matter all that much. Both allocators will work. It is just a matter of using optimal tool for the specific purose.

Quoted 3 lines

You mentioned alloc_pages(). I reverted to __get_free_pages(), but alloc_pages() of course looks nicer. What are the tradeoffs between __get_free_pages() and alloc_pages()?

alloc_pages will return struct page but you need a kernel pointer. That is what __get_free_pages will give you (or you can call page_address directly).

Quoted 16 lines

Thanx, Paul

------------------------------------------------------------------------

commit 490b638d7c241ac06cee168ccf8688bb8b872478

Author: Paul E. McKenney <[email protected]>
Date:   Wed Sep 30 16:16:39 2020 -0700

kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.

The advantages of using kmalloc() and kfree() are a possible small speedup on CONFIG_SLAB=y systems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using __get_free_page() and free_page() are a possible reduction in fragmentation and direct access to the buddy allocator.

Quoted 6 lines

To help settle the question as to which to use, this commit switches from kmalloc() and kfree() to __get_free_page() and free_page().

Yes, looks good to me. I am not entirely sure about the fragmentation argument. It really depends on the SL.B allocator internals. The same applies for the potential speed up. I would be even surprised if the SLAB was faster in average considering it has to use the page allocator as well. So to me the primary motivation would be "use the right tool for the purpose".

Quoted 23 lines
kernel/rcu/tree.c +3 -2
index 2886e81..242f0f0 100644
@@ -3225,7 +3225,8 @@ static void kfree_rcu_work(struct work_struct *work)
 				bkvhead[i] = NULL;
 			krc_this_cpu_unlock(krcp, flags);
 
-			kfree(bkvhead[i]);
+			if (bkvhead[i])
+				free_page((unsigned long)bkvhead[i]);
 
 			cond_resched_tasks_rcu_qs();
 		}
@@ -3378,7 +3379,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 		bnode = get_cached_bnode(*krcp);
 		if (!bnode && can_alloc_page) {
 			krc_this_cpu_unlock(*krcp, *flags);
-			bnode = kmalloc(PAGE_SIZE, gfp);
+			bnode = (struct kvfree_rcu_bulk_data *)__get_free_page(gfp);
 			*krcp = krc_this_cpu_lock(flags);
 		}
 
Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#24 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

Quoted 27 lines · Michal Hocko
Quoted 2 lines · Paul E. McKenney
Quoted 1 line · Michal Hocko

On Tue 29-09-20 18:53:27, Paul E. McKenney wrote:

[...]

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines

No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.

Quoted 7 lines
Quoted 3 lines

The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().

OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need __GFP_NOMEMALLOC as well.

Quoted 7 lines

Right now the restriction is that single-argument (AKA can_sleep==true) kvfree_rcu() cannot be invoked from memory reclaim context.

But would adding __GFP_NOMEMALLOC to the can_sleep==true GFP_ flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).

Quoted 5 lines

It would be quite daring to claim it will be ok but it will certainly be less problematic. Adding the flag will not hurt in any case. As this is a shared called that might be called from many contexts I think it will be safer to have it there. The justification is that it will prevent consumption of memory reserves from MEMALLOC contexts.

Ah, so a different goal (and yes, I finally went over and read the relevant documentation). Agreed, the can_sleep path does not really need to be dipping into the emergency reserves. And it looks like the not-from-reclaim restriction is still at least partially in effect, but one step at a time.

The patch is shown below, which I have queued for a later release.

Quoted 21 lines
Quoted 9 lines
Quoted 9 lines

[...]

Quoted 6 lines
Quoted 2 lines

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

Quoted 8 lines
Quoted 3 lines

alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.

OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.

It doesn't matter all that much. Both allocators will work. It is just a matter of using optimal tool for the specific purose.

Quoted 7 lines
Quoted 3 lines

You mentioned alloc_pages(). I reverted to __get_free_pages(), but alloc_pages() of course looks nicer. What are the tradeoffs between __get_free_pages() and alloc_pages()?

alloc_pages will return struct page but you need a kernel pointer. That is what __get_free_pages will give you (or you can call page_address directly).

Thank you, looks like __get_free_pages() is the tool for this job.

Please see below for the aforementioned patch.

Thanx, Paul

------------------------------------------------------------------------

commit ea5c19d21233b5e8d3d06c0d4ecd6be9f2829dc3

Author: Paul E. McKenney <[email protected]>
Date:   Thu Oct 1 09:24:40 2020 -0700

kvfree_rcu: Use __GFP_NOMEMALLOC for single-argument kvfree_rcu()

This commit applies the __GFP_NOMEMALLOC gfp flag to memory allocations carried out by the single-argument variant of kvfree_rcu(), thus avoiding this can-sleep code path from dipping into the emergency reserves.

kernel/rcu/tree.c +2 -1
index 242f0f0..6132452 100644
@@ -3364,7 +3364,8 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 {
 	struct kvfree_rcu_bulk_data *bnode;
 	bool can_alloc_page = preemptible();
-	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOMEMALLOC
+			       : GFP_ATOMIC) | __GFP_NOWARN;
 	int idx;
 
 	*krcp = krc_this_cpu_lock(flags);
Acked-by
#25 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

On Thu 01-10-20 09:27:09, Paul E. McKenney wrote:
[...]

Quoted 4 lines

commit ea5c19d21233b5e8d3d06c0d4ecd6be9f2829dc3

Author: Paul E. McKenney <[email protected]>
Date:   Thu Oct 1 09:24:40 2020 -0700
Quoted 8 lines

kvfree_rcu: Use __GFP_NOMEMALLOC for single-argument kvfree_rcu()

This commit applies the __GFP_NOMEMALLOC gfp flag to memory allocations carried out by the single-argument variant of kvfree_rcu(), thus avoiding this can-sleep code path from dipping into the emergency reserves.

LGTM. At least for this one I feel competent to give you

Quoted 15 lines
kernel/rcu/tree.c +2 -1
index 242f0f0..6132452 100644
@@ -3364,7 +3364,8 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 {
 	struct kvfree_rcu_bulk_data *bnode;
 	bool can_alloc_page = preemptible();
-	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOMEMALLOC
+			       : GFP_ATOMIC) | __GFP_NOWARN;
 	int idx;
 
 	*krcp = krc_this_cpu_lock(flags);
Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#26 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

Quoted 12 lines · Michal Hocko

On Thu 01-10-20 09:27:09, Paul E. McKenney wrote:
[...]

Quoted 10 lines

commit ea5c19d21233b5e8d3d06c0d4ecd6be9f2829dc3

Author: Paul E. McKenney <[email protected]>
Date:   Thu Oct 1 09:24:40 2020 -0700

kvfree_rcu: Use __GFP_NOMEMALLOC for single-argument kvfree_rcu()

This commit applies the __GFP_NOMEMALLOC gfp flag to memory allocations carried out by the single-argument variant of kvfree_rcu(), thus avoiding this can-sleep code path from dipping into the emergency reserves.

Quoted 5 lines
Quoted 2 lines

LGTM. At least for this one I feel competent to give you

Thank you very much! I will apply this on the next rebase later today, Pacific Time.

Thanx, Paul

Quoted 18 lines
Quoted 14 lines
kernel/rcu/tree.c +2 -1
index 242f0f0..6132452 100644
@@ -3364,7 +3364,8 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 {
 	struct kvfree_rcu_bulk_data *bnode;
 	bool can_alloc_page = preemptible();
-	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL : GFP_ATOMIC) | __GFP_NOWARN;
+	gfp_t gfp = (can_sleep ? GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOMEMALLOC
+			       : GFP_ATOMIC) | __GFP_NOWARN;
 	int idx;
 
 	*krcp = krc_this_cpu_lock(flags);
Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#27 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

In reply to Michal Hocko
Quoted 1 line · Michal Hocko

On Wed 30-09-20 16:21:54, Paul E. McKenney wrote:

[ . . . ]

Hit "send" too soon, apologies...

Quoted 21 lines
Quoted 14 lines

------------------------------------------------------------------------

commit 490b638d7c241ac06cee168ccf8688bb8b872478

Author: Paul E. McKenney <[email protected]>
Date:   Wed Sep 30 16:16:39 2020 -0700

kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.

The advantages of using kmalloc() and kfree() are a possible small speedup on CONFIG_SLAB=y systems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using __get_free_page() and free_page() are a possible reduction in fragmentation and direct access to the buddy allocator.

Quoted 6 lines

To help settle the question as to which to use, this commit switches from kmalloc() and kfree() to __get_free_page() and free_page().

Quoted 6 lines

Yes, looks good to me. I am not entirely sure about the fragmentation argument. It really depends on the SL.B allocator internals. The same applies for the potential speed up. I would be even surprised if the SLAB was faster in average considering it has to use the page allocator as well. So to me the primary motivation would be "use the right tool for the purpose".

Very well, I will update the commit message, and thank you!

Thanx, Paul

Quoted 27 lines
Quoted 23 lines
kernel/rcu/tree.c +3 -2
index 2886e81..242f0f0 100644
@@ -3225,7 +3225,8 @@ static void kfree_rcu_work(struct work_struct *work)
 				bkvhead[i] = NULL;
 			krc_this_cpu_unlock(krcp, flags);
 
-			kfree(bkvhead[i]);
+			if (bkvhead[i])
+				free_page((unsigned long)bkvhead[i]);
 
 			cond_resched_tasks_rcu_qs();
 		}
@@ -3378,7 +3379,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 		bnode = get_cached_bnode(*krcp);
 		if (!bnode && can_alloc_page) {
 			krc_this_cpu_unlock(*krcp, *flags);
-			bnode = kmalloc(PAGE_SIZE, gfp);
+			bnode = (struct kvfree_rcu_bulk_data *)__get_free_page(gfp);
 			*krcp = krc_this_cpu_lock(flags);
 		}
 
Signature · 2 lines
Michal Hocko
SUSE Labs
maintainer
#28 lore raw

Re: [PATCH tip/core/rcu 14/15] rcu/tree: Allocate a page when caller is preemptible

In reply to Michal Hocko
Quoted 86 lines · Michal Hocko
Quoted 2 lines · Paul E. McKenney
Quoted 1 line · Michal Hocko

On Tue 29-09-20 18:53:27, Paul E. McKenney wrote:

[...]

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines

No argument on it being confusing, and I hope that the added header comment helps. But specifically, can_sleep==true is a promise by the caller to be schedulable and not to be holding any lock/mutex/whatever that might possibly be acquired by the memory allocator or by anything else that the memory allocator might invoke, to your point, including for but one example the reclaim logic.

Quoted 7 lines
Quoted 3 lines

The only way that can_sleep==true is if this function was invoked due to a call to single-argument kvfree_rcu(), which must be schedulable because its fallback is to invoke synchronize_rcu().

OK. I have to say that it is still not clear to me whether this call path can be called from the memory reclaim context. If yes then you need __GFP_NOMEMALLOC as well.

Quoted 7 lines

Right now the restriction is that single-argument (AKA can_sleep==true) kvfree_rcu() cannot be invoked from memory reclaim context.

But would adding __GFP_NOMEMALLOC to the can_sleep==true GFP_ flags allow us to remove this restriction? If so, I will queue a separate patch making this change. The improved ease of use would be well worth it, if I understand correctly (ha!!!).

It would be quite daring to claim it will be ok but it will certainly be less problematic. Adding the flag will not hurt in any case. As this is a shared called that might be called from many contexts I think it will be safer to have it there. The justification is that it will prevent consumption of memory reserves from MEMALLOC contexts.

Quoted 10 lines
Quoted 10 lines

[...]

Quoted 6 lines
Quoted 2 lines

What is the point of calling kmalloc for a PAGE_SIZE object? Wouldn't using the page allocator directly be better?

Well, you guys gave me considerable heat about abusing internal allocator interfaces, and kmalloc() and kfree() seem to be about as non-internal as you can get and still be invoking the allocator. ;-)

Quoted 8 lines
Quoted 3 lines

alloc_pages resp. __get_free_pages is a normal page allocator interface to use for page size granular allocations. kmalloc is for more fine grained allocations.

OK, in the short term, both work, but I have queued a separate patch making this change and recording the tradeoffs. This is not yet a promise to push this patch, but it is a promise not to lose this part of the picture. Please see below.

It doesn't matter all that much. Both allocators will work. It is just a matter of using optimal tool for the specific purose.

Quoted 3 lines

You mentioned alloc_pages(). I reverted to __get_free_pages(), but alloc_pages() of course looks nicer. What are the tradeoffs between __get_free_pages() and alloc_pages()?

alloc_pages will return struct page but you need a kernel pointer. That is what __get_free_pages will give you (or you can call page_address directly).

Quoted 16 lines

Thanx, Paul

------------------------------------------------------------------------

commit 490b638d7c241ac06cee168ccf8688bb8b872478

Author: Paul E. McKenney <[email protected]>
Date:   Wed Sep 30 16:16:39 2020 -0700

kvfree_rcu(): Switch from kmalloc/kfree to __get_free_page/free_page.

The advantages of using kmalloc() and kfree() are a possible small speedup on CONFIG_SLAB=y systems, avoiding the allocation-side cast, and use of more-familiar API members. The advantages of using __get_free_page() and free_page() are a possible reduction in fragmentation and direct access to the buddy allocator.

Quoted 6 lines

To help settle the question as to which to use, this commit switches from kmalloc() and kfree() to __get_free_page() and free_page().

Quoted 7 lines

Yes, looks good to me. I am not entirely sure about the fragmentation argument. It really depends on the SL.B allocator internals. The same applies for the potential speed up. I would be even surprised if the SLAB was faster in average considering it has to use the page allocator as well. So to me the primary motivation would be "use the right tool for the purpose".

As for raised a concern about fragmentation, i mostly was thinking about that SLAbs where not designed to do an efficient allocations for sizes which are >= than PAGE_SIZE. But it depends on three different implementations, actually it also a good argument to switch to the page allocator. I mean to get rid of such dependency.

Other side is, SLABs, at least SLAB and SLUB use slab-caches and sizes which they support include up to:

<snip>

kmalloc-8k           420    420   8192    4
kmalloc-4k          1372   1392   4096    8    8 : tunables    0    0

...
<snip>

I would no be surprised that SLAB is faster than using the page allocator in _some_ sense. If it is principle i can double check. I can explain it just in having dynamic caching that can grow based on demand, thus there is no need to go deeper to page allocator if the kmalloc-4k has extra objects. But the worst case of course will be slower :)

--
Vlad Rezki

author maintainer
#29 lore raw

PATCH 15/15 kvfree_rcu(): Fix ifnullfree.cocci warnings

In reply to Paul E. McKenney

From: kernel test robot <[email protected]>

NULL check before kfree is not needed.

Generated by: scripts/coccinelle/free/ifnullfree.cocci

Fixes 9f7e887e648c ("rcu/tree: Allocate a page when caller is preemptible")
---
 kernel/rcu/tree.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/rcu/tree.c +1 -2
index cc998d7..4a85ea2 100644
@@ -3165,8 +3165,7 @@ static void kfree_rcu_work(struct work_struct *work)
 				bkvhead[i] = NULL;
 			krc_this_cpu_unlock(krcp, flags);
 
-			if (bkvhead[i])
-				kfree(bkvhead[i]);
+			kfree(bkvhead[i]);
 
 			cond_resched_tasks_rcu_qs();
 		}
Signature · 1 line
2.9.5