Skip to contentSkip to conversation

CPU hotplug awareness in percpu allocator

RFC PATCH In review Bharata B Rao 3 participants 10 messages Jun 1, 2021 – Jun 4, 2021

Subsystems (10)
lore
Share & tools
lore
#0 lore raw

cover CPU hotplug awareness in percpu allocator

Hi,

This is an attempt to make the percpu allocator CPU hotplug aware. Currently the percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and when they become online.

This early RFC work shows some good memory savings for a powerpc KVM guest that is booted with 16 online and 1024 possible CPUs. Here is the comparision of Percpu memory consumption from /proc/meminfo before and after creating 1000 memcgs.

W/o patch W/ patch
Before 1441792 kB 22528 kB
After 1000 memcgs 4390912 kB 68608 kB

Note that the Percpu reporting in meminfo has been changed in the patchset to reflect the allocation for online CPUs only.

More details about the approach are present in the patch descriptions.

Bharata B Rao (3):
percpu: CPU hotplug support for alloc_percpu()
percpu: Limit percpu allocator to online cpus
percpu: Avoid using percpu ptrs of non-existing cpus

 fs/namespace.c             |   4 +-
 include/linux/cpuhotplug.h |   2 +
 include/linux/percpu.h     |  15 +++
 kernel/cgroup/rstat.c      |  20 +++-
 kernel/sched/cpuacct.c     |  10 +-
 kernel/sched/psi.c         |  14 ++-
 lib/percpu-refcount.c      |   4 +-
 lib/percpu_counter.c       |   2 +-
 mm/percpu-internal.h       |   9 ++
 mm/percpu-vm.c             | 211 +++++++++++++++++++++++++++++++++-
 mm/percpu.c                | 229 +++++++++++++++++++++++++++++++++++--
 net/ipv4/fib_semantics.c   |   2 +-
 net/ipv6/route.c           |   6 +-
 13 files changed, 490 insertions(+), 38 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

RFC PATCH 1/3 percpu: CPU hotplug support for alloc_percpu()

The percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and they become online. Essentially the population of the chunk which involves allocating the pages for the chunk unit that corresponding to the CPU and mapping them to the vmalloc range can be delayed to the CPU hotplug time.

To achieve this, add CPU hotplug callback support to the percpu allocator and let it setup the percpu allocation corresponding to the newly coming up CPU at hotplug time. The vmalloc range is allocated for all the possible CPUs upfront, but during hotplug time, only the populated pages from the chunk are setup (allocated and mapped) for the unit that corresponds to the new CPUs. The same is undone (unit pages unmapped and freed) at unplug time.

This itself isn't sufficient because some callers of alloc_percpu() would expect the percpu variables/pointers for all the possible CPUs to have been initialized at allocation time itself. Hence allow them to register a callback via alloc_percpu() variants that would be called back during hotpug time for any necessary initialization of percpu variables.

This is very much an experimental patch with major unsolved and unaddressed aspects listed below:

- Memcg charging has been changed to account for online CPUs, however the growing and removing of charge corresponding to the hotplugged CPU hasn't been done yet.
- The CPU hotplug support has been added only to vmalloc based percpu allocator.
- All the callers of alloc_percpu() who need initialization callbacks haven't been changed to use the new variants. I have changed only those callers whom I ran into when booting a minimal powerpc KVM guest in my environment.
- Yet to audit all the callers of alloc_percpu() and verify if the approach taken here would fit their use of percpu memory and if they are in a position to handle the required initialization during CPU hotplug time.
- The patches may break git blame, the intention right now is just to make it easy to illustrate the approach taken.

---
 include/linux/cpuhotplug.h |   2 +
 include/linux/percpu.h     |  15 +++
 mm/percpu-internal.h       |   9 ++
 mm/percpu-vm.c             | 199 +++++++++++++++++++++++++++++++++++
 mm/percpu.c                | 209 ++++++++++++++++++++++++++++++++++++-
 5 files changed, 430 insertions(+), 4 deletions(-)
include/linux/cpuhotplug.h +2 -0
index 4a62b3980642..ae20a14967eb 100644
@@ -36,6 +36,8 @@ enum cpuhp_state {
 	CPUHP_X86_MCE_DEAD,
 	CPUHP_VIRT_NET_DEAD,
 	CPUHP_SLUB_DEAD,
+	CPUHP_PERCPU_SETUP,
+	CPUHP_PERCPU_ALLOC,
 	CPUHP_DEBUG_OBJ_DEAD,
 	CPUHP_MM_WRITEBACK_DEAD,
 	CPUHP_MM_VMSTAT_DEAD,
include/linux/percpu.h +15 -0
index 5e76af742c80..145fdb9318d1 100644
@@ -100,6 +100,7 @@ typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size,
 typedef void (*pcpu_fc_free_fn_t)(void *ptr, size_t size);
 typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr);
 typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to);
+typedef int (*pcpu_cpuhp_fn_t)(void __percpu *ptr, unsigned int cpu, void *data);
 
 extern struct pcpu_alloc_info * __init pcpu_alloc_alloc_info(int nr_groups,
 							     int nr_units);
@@ -133,6 +134,11 @@ extern void __init setup_per_cpu_areas(void);
 
 extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp);
 extern void __percpu *__alloc_percpu(size_t size, size_t align);
+extern void __percpu *__alloc_percpu_gfp_cb(size_t size, size_t align,
+					    gfp_t gfp, pcpu_cpuhp_fn_t fn,
+					    void *data);
+extern void __percpu *__alloc_percpu_cb(size_t size, size_t align,
+					pcpu_cpuhp_fn_t fn, void *data);
 extern void free_percpu(void __percpu *__pdata);
 extern phys_addr_t per_cpu_ptr_to_phys(void *addr);
 
@@ -143,6 +149,15 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr);
 	(typeof(type) __percpu *)__alloc_percpu(sizeof(type),		\
 						__alignof__(type))
 
+#define alloc_percpu_gfp_cb(type, gfp, fn, data)			\
+	(typeof(type) __percpu *)__alloc_percpu_gfp_cb(sizeof(type),	\
+						__alignof__(type), gfp,	\
+						fn, data)
+#define alloc_percpu_cb(type, fn, data)					\
+	(typeof(type) __percpu *)__alloc_percpu_cb(sizeof(type),	\
+						__alignof__(type),	\
+						fn, data)
+
 extern unsigned long pcpu_nr_pages(void);
 
 #endif /* __LINUX_PERCPU_H */
mm/percpu-internal.h +9 -0
index ae26b118e246..8064e7c43b9f 100644
@@ -57,6 +57,8 @@ struct pcpu_chunk {
 #endif
 
 	struct list_head	list;		/* linked to pcpu_slot lists */
+	struct list_head	cpuhp;		/* list of registered cpu hotplug
+						   notifiers */
 	int			free_bytes;	/* free bytes in the chunk */
 	struct pcpu_block_md	chunk_md;
 	void			*base_addr;	/* base address of this chunk */
@@ -282,4 +284,11 @@ static inline void pcpu_stats_chunk_dealloc(void)
 
 #endif /* !CONFIG_PERCPU_STATS */
 
+struct percpu_cpuhp_notifier {
+	void __percpu *ptr;
+	void *data;
+	pcpu_cpuhp_fn_t cb;
+	struct list_head list;
+};
+
 #endif
mm/percpu-vm.c +199 -0

229 lines not on this page · raw message

mm/percpu.c +205 -4

287 lines not on this page · raw message

Signature · 1 line
2.31.1
#2 lore raw

Re: [RFC PATCH v0 1/3] percpu: CPU hotplug support for alloc_percpu()

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Bharata,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on linux/master linus/master v5.13-rc4 next-20210528] [cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git a8ea6fc9b089156d9230bfeef964dd9be101a4a9
config: i386-tinyconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/56c0782cce05ec8801eac57b90a531eb01e025c0
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307
        git checkout 56c0782cce05ec8801eac57b90a531eb01e025c0
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

mm/percpu.c: In function '__pcpu_cpuhp_setup':

>> mm/percpu.c:1698:5: error: implicit declaration of function 'pcpu_populate_chunk_cpu'; did you mean 'pcpu_populate_chunk'? [-Werror=implicit-function-declaration]
    1698 |     pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
         |     ^~~~~~~~~~~~~~~~~~~~~~~
         |     pcpu_populate_chunk
   mm/percpu.c: In function '__pcpu_cpuhp_destroy':
>> mm/percpu.c:1738:5: error: implicit declaration of function 'pcpu_depopulate_chunk_cpu'; did you mean 'pcpu_depopulate_chunk'? [-Werror=implicit-function-declaration]
    1738 |     pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re);
         |     ^~~~~~~~~~~~~~~~~~~~~~~~~
         |     pcpu_depopulate_chunk
   mm/percpu.c: At top level:
>> mm/percpu.c:2124:16: warning: no previous prototype for '__alloc_reserved_percpu_cb' [-Wmissing-prototypes]

2124 | void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors

vim +1698 mm/percpu.c

1682
1683 static void __pcpu_cpuhp_setup(enum pcpu_chunk_type type, unsigned int cpu)
1684 {
1685 int slot;
1686 struct list_head *pcpu_slot = pcpu_chunk_list(type);
1687 struct pcpu_chunk *chunk;
1688
1689 for (slot = 0; slot < pcpu_nr_slots; slot++) {
1690 list_for_each_entry(chunk, &pcpu_slot[slot], list) {
1691 unsigned int rs, re;
1692
1693 if (chunk == pcpu_first_chunk)
1694 continue;
1695
1696 bitmap_for_each_set_region(chunk->populated, rs, re, 0,
1697 chunk->nr_pages)

> 1698					pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
  1699								GFP_KERNEL);
  1700			}
  1701		}
  1702	}
  1703	
  1704	/**
  1705	 * cpu hotplug callback for percpu allocator
  1706	 * @cpu: cpu that is being hotplugged
  1707	 *
  1708	 * Allocates and maps the pages that corresponds to @cpu's unit
  1709	 * in all chunks.
  1710	 */
  1711	static int percpu_cpuhp_setup(unsigned int cpu)
  1712	{
  1713		enum pcpu_chunk_type type;
  1714	
  1715		mutex_lock(&pcpu_alloc_mutex);
  1716		for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++)
  1717			__pcpu_cpuhp_setup(type, cpu);
  1718		mutex_unlock(&pcpu_alloc_mutex);
  1719	
  1720		return 0;
  1721	}
  1722	
  1723	static void __pcpu_cpuhp_destroy(enum pcpu_chunk_type type, unsigned int cpu)
  1724	{
  1725		int slot;
  1726		struct list_head *pcpu_slot = pcpu_chunk_list(type);
  1727		struct pcpu_chunk *chunk;
  1728	
  1729		for (slot = 0; slot < pcpu_nr_slots; slot++) {
  1730			list_for_each_entry(chunk, &pcpu_slot[slot], list) {
  1731				unsigned int rs, re;
  1732	
  1733				if (chunk == pcpu_first_chunk)
  1734					continue;
  1735	
  1736				bitmap_for_each_set_region(chunk->populated, rs, re, 0,
  1737							   chunk->nr_pages)
> 1738					pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re);

1739 }
1740 }
1741 }
1742

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#3 lore raw

Re: [RFC PATCH v0 1/3] percpu: CPU hotplug support for alloc_percpu()

In reply to Bharata B Rao
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Bharata,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on linux/master linus/master v5.13-rc4 next-20210528] [cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git a8ea6fc9b089156d9230bfeef964dd9be101a4a9
config: x86_64-randconfig-a001-20210601 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project db26cd30b6dd65e88d786e97a1e453af5cd48966)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/56c0782cce05ec8801eac57b90a531eb01e025c0
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 git checkout 56c0782cce05ec8801eac57b90a531eb01e025c0
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> mm/percpu.c:1698:5: error: implicit declaration of function 'pcpu_populate_chunk_cpu' [-Werror,-Wimplicit-function-declaration]
                                   pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
                                   ^
   mm/percpu.c:1698:5: note: did you mean 'pcpu_populate_chunk'?
   mm/percpu-km.c:35:12: note: 'pcpu_populate_chunk' declared here
   static int pcpu_populate_chunk(struct pcpu_chunk *chunk,
              ^
>> mm/percpu.c:1738:5: error: implicit declaration of function 'pcpu_depopulate_chunk_cpu' [-Werror,-Wimplicit-function-declaration]
                                   pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re);
                                   ^
   mm/percpu.c:1738:5: note: did you mean 'pcpu_depopulate_chunk'?
   mm/percpu-km.c:41:13: note: 'pcpu_depopulate_chunk' declared here
   static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk,
               ^
   mm/percpu.c:2124:16: warning: no previous prototype for function '__alloc_reserved_percpu_cb' [-Wmissing-prototypes]
   void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
                  ^

mm/percpu.c:2124:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
^
static
1 warning and 2 errors generated.

vim +/pcpu_populate_chunk_cpu +1698 mm/percpu.c

1682
1683 static void __pcpu_cpuhp_setup(enum pcpu_chunk_type type, unsigned int cpu)
1684 {
1685 int slot;
1686 struct list_head *pcpu_slot = pcpu_chunk_list(type);
1687 struct pcpu_chunk *chunk;
1688
1689 for (slot = 0; slot < pcpu_nr_slots; slot++) {
1690 list_for_each_entry(chunk, &pcpu_slot[slot], list) {
1691 unsigned int rs, re;
1692
1693 if (chunk == pcpu_first_chunk)
1694 continue;
1695
1696 bitmap_for_each_set_region(chunk->populated, rs, re, 0,
1697 chunk->nr_pages)

> 1698					pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
  1699								GFP_KERNEL);
  1700			}
  1701		}
  1702	}
  1703	
  1704	/**
  1705	 * cpu hotplug callback for percpu allocator
  1706	 * @cpu: cpu that is being hotplugged
  1707	 *
  1708	 * Allocates and maps the pages that corresponds to @cpu's unit
  1709	 * in all chunks.
  1710	 */
  1711	static int percpu_cpuhp_setup(unsigned int cpu)
  1712	{
  1713		enum pcpu_chunk_type type;
  1714	
  1715		mutex_lock(&pcpu_alloc_mutex);
  1716		for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++)
  1717			__pcpu_cpuhp_setup(type, cpu);
  1718		mutex_unlock(&pcpu_alloc_mutex);
  1719	
  1720		return 0;
  1721	}
  1722	
  1723	static void __pcpu_cpuhp_destroy(enum pcpu_chunk_type type, unsigned int cpu)
  1724	{
  1725		int slot;
  1726		struct list_head *pcpu_slot = pcpu_chunk_list(type);
  1727		struct pcpu_chunk *chunk;
  1728	
  1729		for (slot = 0; slot < pcpu_nr_slots; slot++) {
  1730			list_for_each_entry(chunk, &pcpu_slot[slot], list) {
  1731				unsigned int rs, re;
  1732	
  1733				if (chunk == pcpu_first_chunk)
  1734					continue;
  1735	
  1736				bitmap_for_each_set_region(chunk->populated, rs, re, 0,
  1737							   chunk->nr_pages)
> 1738					pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re);

1739 }
1740 }
1741 }
1742

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#4 lore raw

Re: [RFC PATCH v0 1/3] percpu: CPU hotplug support for alloc_percpu()

In reply to Bharata B Rao
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Bharata,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on linux/master linus/master v5.13-rc4 next-20210601] [cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git a8ea6fc9b089156d9230bfeef964dd9be101a4a9
config: x86_64-randconfig-s021-20210601 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce:
        # apt-get install sparse
        # sparse version: v0.6.3-341-g8af24329-dirty
        # https://github.com/0day-ci/linux/commit/56c0782cce05ec8801eac57b90a531eb01e025c0
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307
        git checkout 56c0782cce05ec8801eac57b90a531eb01e025c0
        # save the attached .config to linux build tree
        make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate

sparse warnings: (new ones prefixed by >>)

>> mm/percpu.c:2124:15: sparse: sparse: symbol '__alloc_reserved_percpu_cb' was not declared. Should it be static?

Please review and possibly fold the followup patch.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#5 lore raw

[RFC PATCH] percpu: __alloc_reserved_percpu_cb() can be static

In reply to Bharata B Rao
mm/percpu.c:2124:15: warning: symbol '__alloc_reserved_percpu_cb' was not declared. Should it be static?

mm/percpu.c:2124:15: warning: symbol '__alloc_reserved_percpu_cb' was not declared. Should it be static?

---
 percpu.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/percpu.c +2 -2
index ca8ca541bede86..1af5c1e9f5707e 100644
@@ -2121,8 +2121,8 @@ void __percpu *__alloc_percpu_cb(size_t size, size_t align, pcpu_cpuhp_fn_t cb,
 }
 EXPORT_SYMBOL_GPL(__alloc_percpu_cb);
 
-void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
-					  pcpu_cpuhp_fn_t cb, void *data)
+static void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
+						 pcpu_cpuhp_fn_t cb, void *data)
 {
 	return pcpu_alloc(size, align, true, GFP_KERNEL, cb, data);
 }
#6 lore raw

RFC PATCH 2/3 percpu: Limit percpu allocator to online cpus

In reply to Bharata B Rao

Now that percpu allocator supports growing of memory for newly coming up CPU at hotplug time, limit the allocation, mapping and memcg charging of memory to online CPUs.

Also change the Percpu memory reporting in /proc/meminfo to reflect the populated pages of only online CPUs.

TODO: Address percpu memcg charging and uncharging from CPU hotplug callback.

---
 mm/percpu-vm.c | 12 ++++++------
 mm/percpu.c    | 20 +++++++++++++-------
 2 files changed, 19 insertions(+), 13 deletions(-)
mm/percpu-vm.c +6 -6
index 3250e1c9aeaf..79ce104c963a 100644
@@ -118,7 +118,7 @@ static void pcpu_free_pages(struct pcpu_chunk *chunk,
 	unsigned int cpu;
 	int i;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		for (i = page_start; i < page_end; i++) {
 			struct page *page = pages[pcpu_page_idx(cpu, i)];
 
@@ -149,7 +149,7 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk,
 
 	gfp |= __GFP_HIGHMEM;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		for (i = page_start; i < page_end; i++) {
 			struct page **pagep = &pages[pcpu_page_idx(cpu, i)];
 
@@ -164,7 +164,7 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk,
 	while (--i >= page_start)
 		__free_page(pages[pcpu_page_idx(cpu, i)]);
 
-	for_each_possible_cpu(tcpu) {
+	for_each_online_cpu(tcpu) {
 		if (tcpu == cpu)
 			break;
 		for (i = page_start; i < page_end; i++)
@@ -248,7 +248,7 @@ static void pcpu_unmap_pages(struct pcpu_chunk *chunk,
 	unsigned int cpu;
 	int i;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		for (i = page_start; i < page_end; i++) {
 			struct page *page;
 
@@ -344,7 +344,7 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk,
 	unsigned int cpu, tcpu;
 	int i, err;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		err = __pcpu_map_pages(pcpu_chunk_addr(chunk, cpu, page_start),
 				       &pages[pcpu_page_idx(cpu, page_start)],
 				       page_end - page_start);
@@ -357,7 +357,7 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk,
 	}
 	return 0;
 err:
-	for_each_possible_cpu(tcpu) {
+	for_each_online_cpu(tcpu) {
 		if (tcpu == cpu)
 			break;
 		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
mm/percpu.c +13 -7
index ca8ca541bede..83b6bcfcfa80 100644
@@ -1594,7 +1594,7 @@ static enum pcpu_chunk_type pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp,
 	if (!objcg)
 		return PCPU_CHUNK_ROOT;
 
-	if (obj_cgroup_charge(objcg, gfp, size * num_possible_cpus())) {
+	if (obj_cgroup_charge(objcg, gfp, size * num_online_cpus())) {
 		obj_cgroup_put(objcg);
 		return PCPU_FAIL_ALLOC;
 	}
@@ -1615,10 +1615,10 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg,
 
 		rcu_read_lock();
 		mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B,
-				size * num_possible_cpus());
+				size * num_online_cpus());
 		rcu_read_unlock();
 	} else {
-		obj_cgroup_uncharge(objcg, size * num_possible_cpus());
+		obj_cgroup_uncharge(objcg, size * num_online_cpus());
 		obj_cgroup_put(objcg);
 	}
 }
@@ -1633,11 +1633,11 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size)
 	objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT];
 	chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL;
 
-	obj_cgroup_uncharge(objcg, size * num_possible_cpus());
+	obj_cgroup_uncharge(objcg, size * num_online_cpus());
 
 	rcu_read_lock();
 	mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B,
-			-(size * num_possible_cpus()));
+			-(size * num_online_cpus()));
 	rcu_read_unlock();
 
 	obj_cgroup_put(objcg);
@@ -1680,6 +1680,9 @@ static void pcpu_cpuhp_deregister(struct pcpu_chunk *chunk,
 		}
 }
 
+/*
+ * TODO: Grow the memcg charge
+ */
 static void __pcpu_cpuhp_setup(enum pcpu_chunk_type type, unsigned int cpu)
 {
 	int slot;
@@ -1720,6 +1723,9 @@ static int percpu_cpuhp_setup(unsigned int cpu)
 	return 0;
 }
 
+/*
+ * TODO: Reduce the memcg charge
+ */
 static void __pcpu_cpuhp_destroy(enum pcpu_chunk_type type, unsigned int cpu)
 {
 	int slot;
@@ -2000,7 +2006,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 		pcpu_schedule_balance_work();
 
 	/* clear the areas and return address relative to base address */
-	for_each_possible_cpu(cpu)
+	for_each_online_cpu(cpu)
 		memset((void *)pcpu_chunk_addr(chunk, cpu, 0) + off, 0, size);
 
 	ptr = __addr_to_pcpu_ptr(chunk->base_addr + off);
@@ -3372,7 +3378,7 @@ void __init setup_per_cpu_areas(void)
  */
 unsigned long pcpu_nr_pages(void)
 {
-	return pcpu_nr_populated * pcpu_nr_units;
+	return pcpu_nr_populated * num_online_cpus();
 }
 
 /*
Signature · 1 line
2.31.1
#7 lore raw

RFC PATCH 3/3 percpu: Avoid using percpu ptrs of non-existing cpus

In reply to Bharata B Rao

Prevent the callers of alloc_percpu() from using the percpu pointer of non-existing CPUs. Also switch those callers who require initialization of percpu data for onlined CPU to use the new variant alloc_percpu_cb()

Note: Not all callers have been modified here

---
 fs/namespace.c           |  4 ++--
 kernel/cgroup/rstat.c    | 20 ++++++++++++++++----
 kernel/sched/cpuacct.c   | 10 +++++-----
 kernel/sched/psi.c       | 14 +++++++++++---
 lib/percpu-refcount.c    |  4 ++--
 lib/percpu_counter.c     |  2 +-
 net/ipv4/fib_semantics.c |  2 +-
 net/ipv6/route.c         |  6 +++---
 8 files changed, 41 insertions(+), 21 deletions(-)
fs/namespace.c +2 -2
index c3f1a78ba369..b6ea584b99e5 100644
@@ -182,7 +182,7 @@ int mnt_get_count(struct mount *mnt)
 	int count = 0;
 	int cpu;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		count += per_cpu_ptr(mnt->mnt_pcp, cpu)->mnt_count;
 	}
 
@@ -294,7 +294,7 @@ static unsigned int mnt_get_writers(struct mount *mnt)
 	unsigned int count = 0;
 	int cpu;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		count += per_cpu_ptr(mnt->mnt_pcp, cpu)->mnt_writers;
 	}
 
kernel/cgroup/rstat.c +16 -4
index cee265cb535c..b25c59138c0b 100644
@@ -152,7 +152,7 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep)
 
 	lockdep_assert_held(&cgroup_rstat_lock);
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock,
 						       cpu);
 		struct cgroup *pos = NULL;
@@ -245,19 +245,31 @@ void cgroup_rstat_flush_release(void)
 	spin_unlock_irq(&cgroup_rstat_lock);
 }
 
+static int cgroup_rstat_cpuhp_handler(void __percpu *ptr, unsigned int cpu, void *data)
+{
+	struct cgroup *cgrp = (struct cgroup *)data;
+	struct cgroup_rstat_cpu *rstatc = per_cpu_ptr(ptr, cpu);
+
+	rstatc->updated_children = cgrp;
+	u64_stats_init(&rstatc->bsync);
+	return 0;
+}
+
 int cgroup_rstat_init(struct cgroup *cgrp)
 {
 	int cpu;
 
 	/* the root cgrp has rstat_cpu preallocated */
 	if (!cgrp->rstat_cpu) {
-		cgrp->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu);
+		cgrp->rstat_cpu = alloc_percpu_cb(struct cgroup_rstat_cpu,
+						  cgroup_rstat_cpuhp_handler,
+						  cgrp);
 		if (!cgrp->rstat_cpu)
 			return -ENOMEM;
 	}
 
 	/* ->updated_children list is self terminated */
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu);
 
 		rstatc->updated_children = cgrp;
@@ -274,7 +286,7 @@ void cgroup_rstat_exit(struct cgroup *cgrp)
 	cgroup_rstat_flush(cgrp);
 
 	/* sanity check */
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu);
 
 		if (WARN_ON_ONCE(rstatc->updated_children != cgrp) ||
kernel/sched/cpuacct.c +5 -5
index 104a1bade14f..81dd53387ba5 100644
@@ -160,7 +160,7 @@ static u64 __cpuusage_read(struct cgroup_subsys_state *css,
 	u64 totalcpuusage = 0;
 	int i;
 
-	for_each_possible_cpu(i)
+	for_each_online_cpu(i)
 		totalcpuusage += cpuacct_cpuusage_read(ca, i, index);
 
 	return totalcpuusage;
@@ -195,7 +195,7 @@ static int cpuusage_write(struct cgroup_subsys_state *css, struct cftype *cft,
 	if (val)
 		return -EINVAL;
 
-	for_each_possible_cpu(cpu)
+	for_each_online_cpu(cpu)
 		cpuacct_cpuusage_write(ca, cpu, 0);
 
 	return 0;
@@ -208,7 +208,7 @@ static int __cpuacct_percpu_seq_show(struct seq_file *m,
 	u64 percpu;
 	int i;
 
-	for_each_possible_cpu(i) {
+	for_each_online_cpu(i) {
 		percpu = cpuacct_cpuusage_read(ca, i, index);
 		seq_printf(m, "%llu ", (unsigned long long) percpu);
 	}
@@ -242,7 +242,7 @@ static int cpuacct_all_seq_show(struct seq_file *m, void *V)
 		seq_printf(m, " %s", cpuacct_stat_desc[index]);
 	seq_puts(m, "\n");
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		struct cpuacct_usage *cpuusage = per_cpu_ptr(ca->cpuusage, cpu);
 
 		seq_printf(m, "%d", cpu);
@@ -275,7 +275,7 @@ static int cpuacct_stats_show(struct seq_file *sf, void *v)
 	int stat;
 
 	memset(val, 0, sizeof(val));
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		u64 *cpustat = per_cpu_ptr(ca->cpustat, cpu)->cpustat;
 
 		val[CPUACCT_STAT_USER]   += cpustat[CPUTIME_USER];
kernel/sched/psi.c +11 -3
index cc25a3cff41f..228977aa4780 100644
@@ -186,7 +186,7 @@ static void group_init(struct psi_group *group)
 {
 	int cpu;
 
-	for_each_possible_cpu(cpu)
+	for_each_online_cpu(cpu)
 		seqcount_init(&per_cpu_ptr(group->pcpu, cpu)->seq);
 	group->avg_last_update = sched_clock();
 	group->avg_next_update = group->avg_last_update + psi_period;
@@ -321,7 +321,7 @@ static void collect_percpu_times(struct psi_group *group,
 	 * the sampling period. This eliminates artifacts from uneven
 	 * loading, or even entirely idle CPUs.
 	 */
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		u32 times[NR_PSI_STATES];
 		u32 nonidle;
 		u32 cpu_changed_states;
@@ -935,12 +935,20 @@ void psi_memstall_leave(unsigned long *flags)
 }
 
 #ifdef CONFIG_CGROUPS
+static int psi_cpuhp_handler(void __percpu *ptr, unsigned int cpu, void *unused)
+{
+	struct psi_group_cpu *groupc = per_cpu_ptr(ptr, cpu);
+
+	seqcount_init(&groupc->seq);
+	return 0;
+}
+
 int psi_cgroup_alloc(struct cgroup *cgroup)
 {
 	if (static_branch_likely(&psi_disabled))
 		return 0;
 
-	cgroup->psi.pcpu = alloc_percpu(struct psi_group_cpu);
+	cgroup->psi.pcpu = alloc_percpu_cb(struct psi_group_cpu, psi_cpuhp_handler, NULL);
 	if (!cgroup->psi.pcpu)
 		return -ENOMEM;
 	group_init(&cgroup->psi);
lib/percpu-refcount.c +2 -2
index a1071cdefb5a..aeba43c33600 100644
@@ -173,7 +173,7 @@ static void percpu_ref_switch_to_atomic_rcu(struct rcu_head *rcu)
 	unsigned long count = 0;
 	int cpu;
 
-	for_each_possible_cpu(cpu)
+	for_each_online_cpu(cpu)
 		count += *per_cpu_ptr(percpu_count, cpu);
 
 	pr_debug("global %lu percpu %lu\n",
@@ -253,7 +253,7 @@ static void __percpu_ref_switch_to_percpu(struct percpu_ref *ref)
 	 * zeroing is visible to all percpu accesses which can see the
 	 * following __PERCPU_REF_ATOMIC clearing.
 	 */
-	for_each_possible_cpu(cpu)
+	for_each_online_cpu(cpu)
 		*per_cpu_ptr(percpu_count, cpu) = 0;
 
 	smp_store_release(&ref->percpu_count_ptr,
lib/percpu_counter.c +1 -1
index ed610b75dc32..db40abc6f0f5 100644
@@ -63,7 +63,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount)
 	unsigned long flags;
 
 	raw_spin_lock_irqsave(&fbc->lock, flags);
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		s32 *pcount = per_cpu_ptr(fbc->counters, cpu);
 		*pcount = 0;
 	}
net/ipv4/fib_semantics.c +1 -1
index a632b66bc13a..dbfd14b0077f 100644
@@ -194,7 +194,7 @@ static void rt_fibinfo_free_cpus(struct rtable __rcu * __percpu *rtp)
 	if (!rtp)
 		return;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		struct rtable *rt;
 
 		rt = rcu_dereference_protected(*per_cpu_ptr(rtp, cpu), 1);
net/ipv6/route.c +3 -3
index a22822bdbf39..e7db3a5fe5c5 100644
@@ -165,7 +165,7 @@ static void rt6_uncached_list_flush_dev(struct net *net, struct net_device *dev)
 	if (dev == loopback_dev)
 		return;
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		struct uncached_list *ul = per_cpu_ptr(&rt6_uncached_list, cpu);
 		struct rt6_info *rt;
 
@@ -3542,7 +3542,7 @@ void fib6_nh_release(struct fib6_nh *fib6_nh)
 	if (fib6_nh->rt6i_pcpu) {
 		int cpu;
 
-		for_each_possible_cpu(cpu) {
+		for_each_online_cpu(cpu) {
 			struct rt6_info **ppcpu_rt;
 			struct rt6_info *pcpu_rt;
 
@@ -6569,7 +6569,7 @@ int __init ip6_route_init(void)
 #endif
 #endif
 
-	for_each_possible_cpu(cpu) {
+	for_each_online_cpu(cpu) {
 		struct uncached_list *ul = per_cpu_ptr(&rt6_uncached_list, cpu);
 
 		INIT_LIST_HEAD(&ul->head);
Signature · 1 line
2.31.1
maintainer
#8 lore raw
In reply to Bharata B Rao

Hello,

Quoted 30 lines · Bharata B Rao

Hi,

This is an attempt to make the percpu allocator CPU hotplug aware. Currently the percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and when they become online.

This early RFC work shows some good memory savings for a powerpc KVM guest that is booted with 16 online and 1024 possible CPUs. Here is the comparision of Percpu memory consumption from /proc/meminfo before and after creating 1000 memcgs.

W/o patch W/ patch
Before 1441792 kB 22528 kB
After 1000 memcgs 4390912 kB 68608 kB

Note that the Percpu reporting in meminfo has been changed in the patchset to reflect the allocation for online CPUs only.

More details about the approach are present in the patch descriptions.

Bharata B Rao (3):
percpu: CPU hotplug support for alloc_percpu()
percpu: Limit percpu allocator to online cpus
percpu: Avoid using percpu ptrs of non-existing cpus

Quoted 18 lines
 fs/namespace.c             |   4 +-
 include/linux/cpuhotplug.h |   2 +
 include/linux/percpu.h     |  15 +++
 kernel/cgroup/rstat.c      |  20 +++-
 kernel/sched/cpuacct.c     |  10 +-
 kernel/sched/psi.c         |  14 ++-
 lib/percpu-refcount.c      |   4 +-
 lib/percpu_counter.c       |   2 +-
 mm/percpu-internal.h       |   9 ++
 mm/percpu-vm.c             | 211 +++++++++++++++++++++++++++++++++-
 mm/percpu.c                | 229 +++++++++++++++++++++++++++++++++++--
 net/ipv4/fib_semantics.c   |   2 +-
 net/ipv6/route.c           |   6 +-
 13 files changed, 490 insertions(+), 38 deletions(-)
Signature · 1 line
2.31.1

I have thought about this for a day now and to be honest my thoughts haven't really changed since the last discussion in [1].

I struggle here for a few reasons:
1. We're intertwining cpu and memory for hotplug.
- What does it mean if we don't have enough memory?
- How hard do we try to reclaim memory?
- Partially allocated cpus? Do we free it all and try again?
2. We're now blocking the whole system on the percpu mutex which can cause terrible side effects. If there is a large amount of percpu memory already in use, this means we've accumulated a substantial number of callbacks.
3. While I did mention a callback approach would work. I'm not thrilled by the additional complexity of it as it can be error prone.

Beyond the above. I still don't believe it's the most well motivated problem. I struggle to see a world where it makes sense to let someone scale from 16 cpus to 1024. As in my mind you would also need to scale memory to some degree too (not necessarily linearly but a 1024 core machine with say like 16 gigs of ram would be pretty funny).

Would it be that bad to use cold migration points and eat a little bit of overhead for what I understand to be a relatively uncommon use case?

[1] https://lore.kernel.org/linux-mm/[email protected]/

Thanks,
Dennis

#9 lore raw
Quoted 23 lines · Dennis Zhou

Hello,

Quoted 10 lines · Bharata B Rao

Hi,

This is an attempt to make the percpu allocator CPU hotplug aware. Currently the percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and when they become online.

Quoted 9 lines

This early RFC work shows some good memory savings for a powerpc KVM guest that is booted with 16 online and 1024 possible CPUs. Here is the comparision of Percpu memory consumption from /proc/meminfo before and after creating 1000 memcgs.

W/o patch W/ patch
Before 1441792 kB 22528 kB
After 1000 memcgs 4390912 kB 68608 kB

Quoted 6 lines

I have thought about this for a day now and to be honest my thoughts haven't really changed since the last discussion in [1].

I struggle here for a few reasons:
1. We're intertwining cpu and memory for hotplug.
- What does it mean if we don't have enough memory?

That means CPU hotplug will fail, but...

Quoted 2 lines

- How hard do we try to reclaim memory?
- Partially allocated cpus? Do we free it all and try again?

... yes these are some difficult questions. We should check if roll back can be done cleanly and efficiently. You can see that I am registering separate hotplug callbacks for the hotplug core and for init routines of alloc_percpu() callers. Rolling back the former should be fairly straight forward, but have to see how desirable and feasible it is to undo the entire CPU hotplug when one of the alloc_percpu callbacks fails, especially if there are hundreds of registered alloc_percpu callbacks.

Quoted 4 lines

2. We're now blocking the whole system on the percpu mutex which can cause terrible side effects. If there is a large amount of percpu memory already in use, this means we've accumulated a substantial number of callbacks.

I am yet to look at each caller in detail and see which of them really need init/free callbacks and which can do without it. After this we will have to measure the overhead all this is putting on the hotplug path. Given that hotplug is a slow path, I wonder if some overhead is tolerable here.

CPU hotplug already happens with cpu_hotplug_lock held, so when you mention that this callback holding percpu mutex can have terrible effects, are you specifically worried about blocking all the percpu allocation requests during hotplug? Or is it something else?

Quoted 2 lines

3. While I did mention a callback approach would work. I'm not thrilled by the additional complexity of it as it can be error prone.

Fair enough, the callback for the percpu allocator core seems fine to me but since I haven't yet looked at all callers in detail, I don't know if we would run into some issues/dependencies in any specific callback handlers that increases the overall complexity.

Other than the callbacks, I am also bit worried about the complexity and the overhead involved in memcg charging and uncharging at CPU hotplug time. In my environment (powerpc kvm guest), I see that each chunk can have a maximum of 180224 obj_cgroups. Now checking for the valid/used one out of that, determining the allocation size and charging/uncharging to the right memcg could be an expensive task.

Quoted 6 lines

Beyond the above. I still don't believe it's the most well motivated problem. I struggle to see a world where it makes sense to let someone scale from 16 cpus to 1024. As in my mind you would also need to scale memory to some degree too (not necessarily linearly but a 1024 core machine with say like 16 gigs of ram would be pretty funny).

Well the platform here provides the capability of scaling and until that scaling happens, why consume memory for not-present CPUs is the motivation here. But as you note, it definetely is a question of whether any real application is making use of this scaling now and the associated complexity.

Even if we consider the scaling from 16 to 1024 CPUs as unrealistic for now, the usecase and the numbers from the production scenario that Alexey mentioned in [1] (2 to 128 CPUs) is certainly a good motivator?

Alexey - You did mention about creating a huge number of memcgs and observing VMs consuming 16-20 GB percpu memory in production. So how any memcgs are we talking about here?

Quoted 3 lines

Would it be that bad to use cold migration points and eat a little bit of overhead for what I understand to be a relatively uncommon use case?

[1] https://lore.kernel.org/linux-mm/[email protected]/

Regards,
Bharata.