debugobjects: add a new Kconfig for POOL_SIZE

PATCH Superseded Qian Cai 3 participants 11 messages Nov 18, 2018 – Nov 19, 2018

Subsystems (2)

Library code Debugobjects:

lore

Share & tools

lore

Qian Cai

author

2018-11-18 08:23 UTC #0 lore raw

PATCH 1/1 debugobjects: add a new Kconfig for POOL_SIZE

The current value of ODEBUG_POOL_SIZE is not big enough for large memory systems with timer or/and workqueue objects because during the early boot, timer objects needs at least the size equals to

No. CPUs x 2 (worker pool)

start_kernel
workqueue_init_early
init_worker_pool
init_timer_key
debug_object_init

puls, No. CPUs

start_kernel
sched_init
hrtimer_init
debug_object_init

Then, workqueue objects requires even more,

No. CPUs x 2 (worker pool) x 6 (workqueue)

start_kernel
workqueue_init_early
__alloc_workqueue_key
alloc_workqueue
init_pwq
debug_object_init

plus, No, CPUs x 2 (worker pool)

start_kernel
perf_event_init

    __init_srcu_struct
      init_srcu_struct_fields
        __init_work
          debug_object_init

As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".

Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.

Signed-off-by Qian Cai <[email protected]>

---
 lib/Kconfig.debug  | 12 ++++++++++++
 lib/debugobjects.c |  7 ++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

lib/Kconfig.debug +12 -0

index 1af29b8224fd..a4af837649df 100644
@@ -449,6 +449,18 @@ config DEBUG_OBJECTS
 	  kernel to track the life time of various objects and validate
 	  the operations on those objects.
 
+config DEBUG_OBJECTS_POOL_SIZE
+	int "Debug objects pool size"
+	depends on DEBUG_OBJECTS_TIMERS || DEBUG_OBJECTS_WORK
+	default 1024
+	help
+	  Some debug objects are allocated during the early boot. Enable some
+	  config options may requires the sizes below,
+
+	  DEBUG_OBJECTS_TIMERS: No. CPUs x 2 (worker pool) + No. CPUs (hrtimer)
+	  DEBUG_OBJECTS_WORK:   No. CPUs x 2 (worker pool) x 6 (workqueue) +
+				No. CPUs x 2 (worker pool)
+
 config DEBUG_OBJECTS_SELFTEST
 	bool "Debug objects selftest"
 	depends on DEBUG_OBJECTS

lib/debugobjects.c +6 -1

index 70935ed91125..eb8158538993 100644
@@ -23,7 +23,12 @@
 #define ODEBUG_HASH_BITS	14
 #define ODEBUG_HASH_SIZE	(1 << ODEBUG_HASH_BITS)
 
-#define ODEBUG_POOL_SIZE	1024
+#ifdef CONFIG_DEBUG_OBJECTS_POOL_SIZE
+#define ODEBUG_POOL_SIZE	CONFIG_DEBUG_OBJECTS_POOL_SIZE
+#else
+#define ODEBUG_POOL_SIZE        1024
+#endif
+
 #define ODEBUG_POOL_MIN_LEVEL	256
 
 #define ODEBUG_CHUNK_SHIFT	PAGE_SHIFT

Signature · 1 line

2.17.2 (Apple Git-113)

Thomas Gleixner

2018-11-18 18:21 UTC #1 lore raw

Qian,

Quoted 39 lines · Qian Cai

The current value of ODEBUG_POOL_SIZE is not big enough for large memory systems with timer or/and workqueue objects because during the early boot, timer objects needs at least the size equals to
No. CPUs x 2 (worker pool)
start_kernel
workqueue_init_early
init_worker_pool
init_timer_key
debug_object_init
puls, No. CPUs
start_kernel
sched_init
hrtimer_init
debug_object_init
Then, workqueue objects requires even more,
No. CPUs x 2 (worker pool) x 6 (workqueue)
start_kernel
workqueue_init_early
__alloc_workqueue_key
alloc_workqueue
init_pwq
debug_object_init
plus, No, CPUs x 2 (worker pool)
start_kernel
perf_event_init
    __init_srcu_struct
      init_srcu_struct_fields
        __init_work
          debug_object_init

Quoted 5 lines

As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.

why do we need a config option, when the required number can be deduced already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?

Thanks,

tglx

Qian Cai

author

2018-11-18 18:57 UTC #2 lore raw

Quoted 47 lines · Thomas Gleixner

Qian,
Quoted 39 lines · Qian Cai
The current value of ODEBUG_POOL_SIZE is not big enough for large memory systems with timer or/and workqueue objects because during the early boot, timer objects needs at least the size equals to
No. CPUs x 2 (worker pool)
start_kernel
workqueue_init_early
init_worker_pool
init_timer_key
debug_object_init
puls, No. CPUs
start_kernel
sched_init
hrtimer_init
debug_object_init
Then, workqueue objects requires even more,
No. CPUs x 2 (worker pool) x 6 (workqueue)
start_kernel
workqueue_init_early
__alloc_workqueue_key
alloc_workqueue
init_pwq
debug_object_init
plus, No, CPUs x 2 (worker pool)
start_kernel
perf_event_init
__init_srcu_struct
     init_srcu_struct_fields
       __init_work
         debug_object_init

Quoted 9 lines

Quoted 5 lines
As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.
why do we need a config option, when the required number can be deduced already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?

It because I am worry about the coupling between the implementation details of timers and workqueue objects, and the computation in the code you mentioned here. For example, people could change workqueue.c to have different number of worekqueues initialized during the early boot in the future which is going to affect the required pool size, and I am not sure if people are going to adjust the code in debugobjects.c here as well when they made changes like that.

Also, the computation could become so complex depends on lots of config options like perf, hrtimer, and combinations that I have not tested so far which is difficult to exhausted all the possibilities.

Hence, I feel like the Kconfig option is more flexible and less error-prone.

Thomas Gleixner

2018-11-19 08:09 UTC #3 lore raw

Qian,

Quoted 18 lines · Qian Cai

Quoted 11 lines · Thomas Gleixner
Quoted 5 lines · Qian Cai
As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.
why do we need a config option, when the required number can be deduced already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?
It because I am worry about the coupling between the implementation details of timers and workqueue objects, and the computation in the code you mentioned here. For example, people could change workqueue.c to have different number of worekqueues initialized during the early boot in the future which is going to affect the required pool size, and I am not sure if people are going to adjust the code in debugobjects.c here as well when they made changes like that.

Quoted 5 lines

Also, the computation could become so complex depends on lots of config options like perf, hrtimer, and combinations that I have not tested so far which is difficult to exhausted all the possibilities.
Hence, I feel like the Kconfig option is more flexible and less error-prone.

Quite the contrary. Config options are a pain and truly error-prone if you want to compile general purpose kernels as distributions do.

Its not really a problem to have a larger initial static pool which gets freed after init anyway. So we can size it generously depending on the config options and be done with it.

Thanks,

tglx

Qian Cai

author

2018-11-19 13:27 UTC #4 lore raw

Quoted 21 lines · Thomas Gleixner

Qian,
Quoted 18 lines · Qian Cai
Quoted 11 lines · Thomas Gleixner
Quoted 5 lines · Qian Cai
As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.
why do we need a config option, when the required number can be deduced already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?
It because I am worry about the coupling between the implementation details of timers and workqueue objects, and the computation in the code you mentioned here. For example, people could change workqueue.c to have different number of worekqueues initialized during the early boot in the future which is going to affect the required pool size, and I am not sure if people are going to adjust the code in debugobjects.c here as well when they made changes like that.

Quoted 9 lines

Quoted 5 lines
Also, the computation could become so complex depends on lots of config options like perf, hrtimer, and combinations that I have not tested so far which is difficult to exhausted all the possibilities.
Hence, I feel like the Kconfig option is more flexible and less error-prone.
Quite the contrary. Config options are a pain and truly error-prone if you want to compile general purpose kernels as distributions do.

Ah, I never thought distributions people would
enable those debugging options.

Quoted 4 lines

Its not really a problem to have a larger initial static pool which gets freed after init anyway. So we can size it generously depending on the config options and be done with it.

That’s a good point. I’ll send out a patch shortly.

Waiman Long

2018-11-19 14:51 UTC #5 lore raw

Quoted 26 lines · Qian Cai

Quoted 24 lines · Thomas Gleixner
Qian,
Quoted 21 lines · Qian Cai
Quoted 10 lines · Thomas Gleixner
Quoted 5 lines · Qian Cai
As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.
why do we need a config option, when the required number can be deduced already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?
It because I am worry about the coupling between the implementation details of timers and workqueue objects, and the computation in the code you mentioned here. For example, people could change workqueue.c to have different number of worekqueues initialized during the early boot in the future which is going to affect the required pool size, and I am not sure if people are going to adjust the code in debugobjects.c here as well when they made changes like that.
Also, the computation could become so complex depends on lots of config options like perf, hrtimer, and combinations that I have not tested so far which is difficult to exhausted all the possibilities.

Quoted 6 lines

Quoted 4 lines
Quoted 1 line
Hence, I feel like the Kconfig option is more flexible and less error-prone.
Quite the contrary. Config options are a pain and truly error-prone if you want to compile general purpose kernels as distributions do.
Ah, I never thought distributions people would
enable those debugging options.

Distros like RHEL usually ship two kernels - one for production and one for debug. The debug kernel does have debugobjects enabled.

Cheers,
Longman

Qian Cai

author

2018-11-19 15:18 UTC #6 lore raw

Quoted 46 lines · Waiman Long

Quoted 45 lines · Qian Cai
Quoted 42 lines · Thomas Gleixner
Qian,
Quoted 28 lines · Qian Cai
Quoted 7 lines
On Nov 18, 2018, at 1:21 PM, Thomas Gleixner <[email protected]> wrote:
Quoted 4 lines · Qian Cai
As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Quoted 8 lines
Quoted 3 lines
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE
accordingly if either timer or workqueue objects are selected.
why do we need a config option, when the required number can be deduced
already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?
It because I am worry about the coupling between the implementation details of
timers and workqueue objects, and the computation in the code you mentioned
here. For example, people could change workqueue.c to have different number
of worekqueues initialized during the early boot in the future which is going to
affect the required pool size, and I am not sure if people are going to adjust the
code in debugobjects.c here as well when they made changes like that.
Quoted 7 lines
Also, the computation could become so complex depends on lots of config options like perf, hrtimer, and combinations that I have not tested so far which is
difficult to exhausted all the possibilities.
Hence, I feel like the Kconfig option is more flexible and less error- prone.
Quite the contrary. Config options are a pain and truly error-prone if you want to compile general purpose kernels as distributions do.

Quoted 6 lines

Quoted 2 lines
Ah, I never thought distributions people would
enable those debugging options.
Distros like RHEL usually ship two kernels - one for production and one for debug. The debug kernel does have debugobjects enabled.

Right, I can remember that now . However, if I understand correctly, since the early static pool size needs to be determined during the compilation time, it depends on the No. CPUs are from the machines that built the distro kernels. Then, when users use those distro kernels, they are not going to have correct the pool size according to the No. CPUs on their test machines.

Waiman Long

2018-11-19 16:19 UTC #7 lore raw

Quoted 45 lines · Qian Cai

Quoted 44 lines · Waiman Long
Quoted 43 lines · Qian Cai
Quoted 34 lines · Thomas Gleixner
Qian,
Quoted 31 lines · Qian Cai
Quoted 7 lines
On Nov 18, 2018, at 1:21 PM, Thomas Gleixner <[email protected]> wrote:
Quoted 4 lines · Qian Cai
As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".
Quoted 7 lines
Quoted 3 lines
Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE
accordingly if either timer or workqueue objects are selected.
why do we need a config option, when the required number can be deduced
already from the active CONFIG_DEBUG_OBJECTS_* and NR_CPUS?
It because I am worry about the coupling between the implementation details of
timers and workqueue objects, and the computation in the code you mentioned
here. For example, people could change workqueue.c to have different number
of worekqueues initialized during the early boot in the future which is going to
affect the required pool size, and I am not sure if people are going to adjust the
code in debugobjects.c here as well when they made changes like that.
Also, the computation could become so complex depends on lots of config options like perf, hrtimer, and combinations that I have not tested so far which is
difficult to exhausted all the possibilities.
Quoted 5 lines
Quoted 2 lines
Hence, I feel like the Kconfig option is more flexible and less error- prone.
Quite the contrary. Config options are a pain and truly error-prone if you want to compile general purpose kernels as distributions do.
Ah, I never thought distributions people would
enable those debugging options.

Quoted 8 lines

Quoted 3 lines
Distros like RHEL usually ship two kernels - one for production and one for debug. The debug kernel does have debugobjects enabled.
Right, I can remember that now . However, if I understand correctly, since the early static pool size needs to be determined during the compilation time, it depends on the No. CPUs are from the machines that built the distro kernels. Then, when users use those distro kernels, they are not going to have correct the pool size according to the No. CPUs on their test machines.

I see your point. Perhaps you can make ODEBUG_POOL_SIZE scales with CONFIG_NR_CPUS like

#define ODEBUG_POOL_SIZE (1024 + CONFIG_NR_CPUS * 2)

CONFIG_NR_CPUS is usually set to a lot higher than the actual number of CPUs in a typical system. So you don't want to set the multiplier too high.

Cheers,
Longman

Thomas Gleixner

2018-11-19 16:26 UTC #8 lore raw

Quoted 7 lines · Waiman Long

Quoted 5 lines · Qian Cai
Right, I can remember that now . However, if I understand correctly, since the early static pool size needs to be determined during the compilation time, it depends on the No. CPUs are from the machines that built the distro kernels. Then, when users use those distro kernels, they are not going to have correct the pool size according to the No. CPUs on their test machines.

Quoted 7 lines

I see your point. Perhaps you can make ODEBUG_POOL_SIZE scales with CONFIG_NR_CPUS like
#define ODEBUG_POOL_SIZE (1024 + CONFIG_NR_CPUS * 2)
CONFIG_NR_CPUS is usually set to a lot higher than the actual number of CPUs in a typical system. So you don't want to set the multiplier too high.

The number of CPUs on the build machine is totally irrelevant and not influencing the build at all. The sizing solely depends on CONFIG_NR_CPUS.

And even if the initial pool is a bit oversized, it's init data and freed, so no real harm done.

Thanks,

tglx

Qian Cai

author

2018-11-19 16:37 UTC #9 lore raw

Quoted 17 lines · Thomas Gleixner

Quoted 16 lines · Waiman Long
Quoted 9 lines · Qian Cai
Right, I can remember that now . However, if I understand correctly, since the
early static pool size needs to be determined during the compilation time, it
depends on the No. CPUs are from the machines that built the distro kernels.
Then, when users use those distro kernels, they are not going to have correct
the pool size according to the No. CPUs on their test machines.
I see your point. Perhaps you can make ODEBUG_POOL_SIZE scales with CONFIG_NR_CPUS like
#define ODEBUG_POOL_SIZE (1024 + CONFIG_NR_CPUS * 2)

Quoted 9 lines

Quoted 2 lines
CONFIG_NR_CPUS is usually set to a lot higher than the actual number of CPUs in a typical system. So you don't want to set the multiplier too high.
The number of CPUs on the build machine is totally irrelevant and not influencing the build at all. The sizing solely depends on CONFIG_NR_CPUS.
And even if the initial pool is a bit oversized, it's init data and freed, so no real harm done.

Ah, I thought you meant the NR_CPUS macro. It is CONFIG_NR_CPUS, and we are on the same page.

Qian Cai

author

2018-11-18 22:09 UTC #10 lore raw

PATCH v2 1/1 debugobjects: add a new Kconfig for POOL_SIZE

In reply to Qian Cai

The current value of ODEBUG_POOL_SIZE is not big enough for systems with large number of CPUs with timer or/and workqueue objects because during the early boot, timer objects needs the size equals to

No. CPUs x 2 (worker pool)

start_kernel
workqueue_init_early
init_worker_pool
init_timer_key
debug_object_init

puls, No. CPUs (if have the hrtimer option)

start_kernel
sched_init
hrtick_rq_init
hrtimer_init
debug_object_init

Then, workqueue objects requires even more,

No. CPUs x 6 (workqueue)

start_kernel
workqueue_init_early
alloc_workqueue
__alloc_workqueue_key
alloc_and_link_pwqs
init_pwq
debug_object_init

plus, No, CPUs x 2 (worker pool) (if have the perf option)

start_kernel
perf_event_init

    __init_srcu_struct
      init_srcu_struct_fields
        __init_work
          debug_object_init

As the results, systems have 60+ CPUs with both timer and workqueue objects enabled could trigger "ODEBUG: Out of memory. ODEBUG disabled".

Hence, add a new Kconfig option so users could adjust ODEBUG_POOL_SIZE accordingly if either timer or workqueue objects are selected.

Signed-off-by Qian Cai <[email protected]>

---

Changes since v1:
* Fixed a few typos.
* Added missing call stacks.
* Removed implementation details from the Kconfig help.

 lib/Kconfig.debug  | 11 +++++++++++
 lib/debugobjects.c |  4 ++++
 2 files changed, 15 insertions(+)

lib/Kconfig.debug +11 -0

index 1af29b8224fd..0749dc627eb5 100644
@@ -449,6 +449,17 @@ config DEBUG_OBJECTS
 	  kernel to track the life time of various objects and validate
 	  the operations on those objects.
 
+config DEBUG_OBJECTS_POOL_SIZE
+	int "Debug objects pool size"
+	depends on DEBUG_OBJECTS_TIMERS || DEBUG_OBJECTS_WORK
+	default 1024
+	help
+	  Some debug objects are allocated during the early boot. Enabling some
+	  options below like timers or workqueue objects may increase the size
+	  required significantly with large number of CPUs. If the kernel
+	  reports "ODEBUG: Out of memory. ODEBUG disabled", please increase
+	  this value.
+
 config DEBUG_OBJECTS_SELFTEST
 	bool "Debug objects selftest"
 	depends on DEBUG_OBJECTS

lib/debugobjects.c +4 -0

index 70935ed91125..b0e8e90f6ad1 100644
@@ -23,7 +23,11 @@
 #define ODEBUG_HASH_BITS	14
 #define ODEBUG_HASH_SIZE	(1 << ODEBUG_HASH_BITS)
 
+#ifdef CONFIG_DEBUG_OBJECTS_POOL_SIZE
+#define ODEBUG_POOL_SIZE	CONFIG_DEBUG_OBJECTS_POOL_SIZE
+#else
 #define ODEBUG_POOL_SIZE	1024
+#endif
 #define ODEBUG_POOL_MIN_LEVEL	256
 
 #define ODEBUG_CHUNK_SHIFT	PAGE_SHIFT

Signature · 1 line

2.17.2 (Apple Git-113)