CPU hotplug awareness in percpu allocator
Subsystems (10)
cover CPU hotplug awareness in percpu allocator
Hi,
This is an attempt to make the percpu allocator CPU hotplug aware. Currently the percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and when they become online.
This early RFC work shows some good memory savings for a powerpc KVM guest that is booted with 16 online and 1024 possible CPUs. Here is the comparision of Percpu memory consumption from /proc/meminfo before and after creating 1000 memcgs.
W/o patch W/ patch
Before 1441792 kB 22528 kB
After 1000 memcgs 4390912 kB 68608 kB
Note that the Percpu reporting in meminfo has been changed in the patchset to reflect the allocation for online CPUs only.
More details about the approach are present in the patch descriptions.
Bharata B Rao (3):
percpu: CPU hotplug support for alloc_percpu()
percpu: Limit percpu allocator to online cpus
percpu: Avoid using percpu ptrs of non-existing cpus
fs/namespace.c | 4 +- include/linux/cpuhotplug.h | 2 + include/linux/percpu.h | 15 +++ kernel/cgroup/rstat.c | 20 +++- kernel/sched/cpuacct.c | 10 +- kernel/sched/psi.c | 14 ++- lib/percpu-refcount.c | 4 +- lib/percpu_counter.c | 2 +- mm/percpu-internal.h | 9 ++ mm/percpu-vm.c | 211 +++++++++++++++++++++++++++++++++- mm/percpu.c | 229 +++++++++++++++++++++++++++++++++++-- net/ipv4/fib_semantics.c | 2 +- net/ipv6/route.c | 6 +- 13 files changed, 490 insertions(+), 38 deletions(-)
Signature · 1 line
2.31.1
RFC PATCH 1/3 percpu: CPU hotplug support for alloc_percpu()
The percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and they become online. Essentially the population of the chunk which involves allocating the pages for the chunk unit that corresponding to the CPU and mapping them to the vmalloc range can be delayed to the CPU hotplug time.
To achieve this, add CPU hotplug callback support to the percpu allocator and let it setup the percpu allocation corresponding to the newly coming up CPU at hotplug time. The vmalloc range is allocated for all the possible CPUs upfront, but during hotplug time, only the populated pages from the chunk are setup (allocated and mapped) for the unit that corresponds to the new CPUs. The same is undone (unit pages unmapped and freed) at unplug time.
This itself isn't sufficient because some callers of alloc_percpu() would expect the percpu variables/pointers for all the possible CPUs to have been initialized at allocation time itself. Hence allow them to register a callback via alloc_percpu() variants that would be called back during hotpug time for any necessary initialization of percpu variables.
This is very much an experimental patch with major unsolved and unaddressed aspects listed below:
- Memcg charging has been changed to account for online CPUs, however the growing and removing of charge corresponding to the hotplugged CPU hasn't been done yet.
- The CPU hotplug support has been added only to vmalloc based percpu allocator.
- All the callers of alloc_percpu() who need initialization callbacks haven't been changed to use the new variants. I have changed only those callers whom I ran into when booting a minimal powerpc KVM guest in my environment.
- Yet to audit all the callers of alloc_percpu() and verify if the approach taken here would fit their use of percpu memory and if they are in a position to handle the required initialization during CPU hotplug time.
- The patches may break git blame, the intention right now is just to make it easy to illustrate the approach taken.
--- include/linux/cpuhotplug.h | 2 + include/linux/percpu.h | 15 +++ mm/percpu-internal.h | 9 ++ mm/percpu-vm.c | 199 +++++++++++++++++++++++++++++++++++ mm/percpu.c | 209 ++++++++++++++++++++++++++++++++++++- 5 files changed, 430 insertions(+), 4 deletions(-)
include/linux/cpuhotplug.h +2 -0
@@ -36,6 +36,8 @@ enum cpuhp_state {CPUHP_X86_MCE_DEAD ,CPUHP_VIRT_NET_DEAD ,CPUHP_SLUB_DEAD , +CPUHP_PERCPU_SETUP , +CPUHP_PERCPU_ALLOC ,CPUHP_DEBUG_OBJ_DEAD ,CPUHP_MM_WRITEBACK_DEAD , CPUHP_MM_VMSTAT_DEAD,
include/linux/percpu.h +15 -0
@@ -100,6 +100,7 @@ typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size,typedef void (*pcpu_fc_free_fn_t )(void *ptr,size_t size);typedef void (*pcpu_fc_populate_pte_fn_t )(unsigned long addr);typedef int (pcpu_fc_cpu_distance_fn_t )(unsigned int from,unsigned int to); +typedef int (*pcpu_cpuhp_fn_t )(void __percpu *ptr,unsigned int cpu,void *data);extern struct pcpu_alloc_info *__init pcpu_alloc_alloc_info (int nr_groups,int nr_units);
@@ -133,6 +134,11 @@ extern void __init setup_per_cpu_areas(void);extern void __percpu *__alloc_percpu_gfp (size_t size,size_t align,gfp_t gfp);extern void __percpu *__alloc_percpu (size_t size,size_t align); +extern void __percpu *__alloc_percpu_gfp_cb (size_t size,size_t align, +gfp_t gfp,pcpu_cpuhp_fn_t fn, +void *data); +extern void __percpu *__alloc_percpu_cb (size_t size,size_t align, +pcpu_cpuhp_fn_t fn,void *data);extern void free_percpu (void __percpu *__pdata);extern phys_addr_t per_cpu_ptr_to_phys (void *addr);
@@ -143,6 +149,15 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +#define alloc_percpu_gfp_cb (type, gfp, fn, data) \ + (typeof(type) __percpu *)__alloc_percpu_gfp_cb(sizeof(type), \ + __alignof__(type), gfp, \ + fn, data) +#define alloc_percpu_cb (type, fn, data) \ + (typeof(type) __percpu *)__alloc_percpu_cb(sizeof(type), \ + __alignof__(type), \ + fn, data) +extern unsigned long pcpu_nr_pages (void );#endif /* __LINUX_PERCPU_H */
mm/percpu-internal.h +9 -0
@@ -57,6 +57,8 @@ struct pcpu_chunk {#endif struct list_head list;/* linked to pcpu_slot lists */ +struct list_head cpuhp;/* list of registered cpu hotplug +notifiers */ int free_bytes;/* free bytes in the chunk */ struct pcpu_block_md chunk_md;void *base_addr;/* base address of this chunk */
@@ -282,4 +284,11 @@ static inline void pcpu_stats_chunk_dealloc(void)#endif /* !CONFIG_PERCPU_STATS */ +struct percpu_cpuhp_notifier { +void __percpu *ptr; +void *data; +pcpu_cpuhp_fn_t cb; +struct list_head list; +}; +#endif
mm/percpu-vm.c +199 -0
229 lines not on this page · raw message
mm/percpu.c +205 -4
287 lines not on this page · raw message
Signature · 1 line
2.31.1
kernel test robot
build botRe: [RFC PATCH v0 1/3] percpu: CPU hotplug support for alloc_percpu()
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Bharata,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on linux/master linus/master v5.13-rc4 next-20210528] [cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git a8ea6fc9b089156d9230bfeef964dd9be101a4a9 config: i386-tinyconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/56c0782cce05ec8801eac57b90a531eb01e025c0 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 git checkout 56c0782cce05ec8801eac57b90a531eb01e025c0 # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
mm/percpu.c: In function '__pcpu_cpuhp_setup':
>> mm/percpu.c:1698:5: error: implicit declaration of function 'pcpu_populate_chunk_cpu'; did you mean 'pcpu_populate_chunk'? [-Werror=implicit-function-declaration]
1698 | pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
| ^~~~~~~~~~~~~~~~~~~~~~~
| pcpu_populate_chunk
mm/percpu.c: In function '__pcpu_cpuhp_destroy':
>> mm/percpu.c:1738:5: error: implicit declaration of function 'pcpu_depopulate_chunk_cpu'; did you mean 'pcpu_depopulate_chunk'? [-Werror=implicit-function-declaration]
1738 | pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re);
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| pcpu_depopulate_chunk
mm/percpu.c: At top level:
>> mm/percpu.c:2124:16: warning: no previous prototype for '__alloc_reserved_percpu_cb' [-Wmissing-prototypes] 2124 | void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +1698 mm/percpu.c
1682
1683 static void __pcpu_cpuhp_setup(enum pcpu_chunk_type type, unsigned int cpu)
1684 {
1685 int slot;
1686 struct list_head *pcpu_slot = pcpu_chunk_list(type);
1687 struct pcpu_chunk *chunk;
1688
1689 for (slot = 0; slot < pcpu_nr_slots; slot++) {
1690 list_for_each_entry(chunk, &pcpu_slot[slot], list) {
1691 unsigned int rs, re;
1692
1693 if (chunk == pcpu_first_chunk)
1694 continue;
1695
1696 bitmap_for_each_set_region(chunk->populated, rs, re, 0,
1697 chunk->nr_pages)
> 1698 pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
1699 GFP_KERNEL);
1700 }
1701 }
1702 }
1703
1704 /**
1705 * cpu hotplug callback for percpu allocator
1706 * @cpu: cpu that is being hotplugged
1707 *
1708 * Allocates and maps the pages that corresponds to @cpu's unit
1709 * in all chunks.
1710 */
1711 static int percpu_cpuhp_setup(unsigned int cpu)
1712 {
1713 enum pcpu_chunk_type type;
1714
1715 mutex_lock(&pcpu_alloc_mutex);
1716 for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++)
1717 __pcpu_cpuhp_setup(type, cpu);
1718 mutex_unlock(&pcpu_alloc_mutex);
1719
1720 return 0;
1721 }
1722
1723 static void __pcpu_cpuhp_destroy(enum pcpu_chunk_type type, unsigned int cpu)
1724 {
1725 int slot;
1726 struct list_head *pcpu_slot = pcpu_chunk_list(type);
1727 struct pcpu_chunk *chunk;
1728
1729 for (slot = 0; slot < pcpu_nr_slots; slot++) {
1730 list_for_each_entry(chunk, &pcpu_slot[slot], list) {
1731 unsigned int rs, re;
1732
1733 if (chunk == pcpu_first_chunk)
1734 continue;
1735
1736 bitmap_for_each_set_region(chunk->populated, rs, re, 0,
1737 chunk->nr_pages)
> 1738 pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re); 1739 }
1740 }
1741 }
1742
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH v0 1/3] percpu: CPU hotplug support for alloc_percpu()
In reply to Bharata B Rao[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Bharata,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on linux/master linus/master v5.13-rc4 next-20210528] [cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git a8ea6fc9b089156d9230bfeef964dd9be101a4a9 config: x86_64-randconfig-a001-20210601 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project db26cd30b6dd65e88d786e97a1e453af5cd48966)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/56c0782cce05ec8801eac57b90a531eb01e025c0
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 git checkout 56c0782cce05ec8801eac57b90a531eb01e025c0
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
>> mm/percpu.c:1698:5: error: implicit declaration of function 'pcpu_populate_chunk_cpu' [-Werror,-Wimplicit-function-declaration]
pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
^
mm/percpu.c:1698:5: note: did you mean 'pcpu_populate_chunk'?
mm/percpu-km.c:35:12: note: 'pcpu_populate_chunk' declared here
static int pcpu_populate_chunk(struct pcpu_chunk *chunk,
^
>> mm/percpu.c:1738:5: error: implicit declaration of function 'pcpu_depopulate_chunk_cpu' [-Werror,-Wimplicit-function-declaration]
pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re);
^
mm/percpu.c:1738:5: note: did you mean 'pcpu_depopulate_chunk'?
mm/percpu-km.c:41:13: note: 'pcpu_depopulate_chunk' declared here
static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk,
^
mm/percpu.c:2124:16: warning: no previous prototype for function '__alloc_reserved_percpu_cb' [-Wmissing-prototypes]
void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
^ mm/percpu.c:2124:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void __percpu *__alloc_reserved_percpu_cb(size_t size, size_t align,
^
static
1 warning and 2 errors generated.
vim +/pcpu_populate_chunk_cpu +1698 mm/percpu.c
1682
1683 static void __pcpu_cpuhp_setup(enum pcpu_chunk_type type, unsigned int cpu)
1684 {
1685 int slot;
1686 struct list_head *pcpu_slot = pcpu_chunk_list(type);
1687 struct pcpu_chunk *chunk;
1688
1689 for (slot = 0; slot < pcpu_nr_slots; slot++) {
1690 list_for_each_entry(chunk, &pcpu_slot[slot], list) {
1691 unsigned int rs, re;
1692
1693 if (chunk == pcpu_first_chunk)
1694 continue;
1695
1696 bitmap_for_each_set_region(chunk->populated, rs, re, 0,
1697 chunk->nr_pages)
> 1698 pcpu_populate_chunk_cpu(cpu, chunk, rs, re,
1699 GFP_KERNEL);
1700 }
1701 }
1702 }
1703
1704 /**
1705 * cpu hotplug callback for percpu allocator
1706 * @cpu: cpu that is being hotplugged
1707 *
1708 * Allocates and maps the pages that corresponds to @cpu's unit
1709 * in all chunks.
1710 */
1711 static int percpu_cpuhp_setup(unsigned int cpu)
1712 {
1713 enum pcpu_chunk_type type;
1714
1715 mutex_lock(&pcpu_alloc_mutex);
1716 for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++)
1717 __pcpu_cpuhp_setup(type, cpu);
1718 mutex_unlock(&pcpu_alloc_mutex);
1719
1720 return 0;
1721 }
1722
1723 static void __pcpu_cpuhp_destroy(enum pcpu_chunk_type type, unsigned int cpu)
1724 {
1725 int slot;
1726 struct list_head *pcpu_slot = pcpu_chunk_list(type);
1727 struct pcpu_chunk *chunk;
1728
1729 for (slot = 0; slot < pcpu_nr_slots; slot++) {
1730 list_for_each_entry(chunk, &pcpu_slot[slot], list) {
1731 unsigned int rs, re;
1732
1733 if (chunk == pcpu_first_chunk)
1734 continue;
1735
1736 bitmap_for_each_set_region(chunk->populated, rs, re, 0,
1737 chunk->nr_pages)
> 1738 pcpu_depopulate_chunk_cpu(cpu, chunk, rs, re); 1739 }
1740 }
1741 }
1742
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH v0 1/3] percpu: CPU hotplug support for alloc_percpu()
In reply to Bharata B Rao[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Bharata,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on linux/master linus/master v5.13-rc4 next-20210601] [cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git a8ea6fc9b089156d9230bfeef964dd9be101a4a9 config: x86_64-randconfig-s021-20210601 (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.3-341-g8af24329-dirty # https://github.com/0day-ci/linux/commit/56c0782cce05ec8801eac57b90a531eb01e025c0 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Bharata-B-Rao/CPU-hotplug-awareness-in-percpu-allocator/20210601-145307 git checkout 56c0782cce05ec8801eac57b90a531eb01e025c0 # save the attached .config to linux build tree make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
sparse warnings: (new ones prefixed by >>)
>> mm/percpu.c:2124:15: sparse: sparse: symbol '__alloc_reserved_percpu_cb' was not declared. Should it be static?
Please review and possibly fold the followup patch.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build bot[RFC PATCH] percpu: __alloc_reserved_percpu_cb() can be static
In reply to Bharata B Raomm/percpu.c:2124:15: warning: symbol '__alloc_reserved_percpu_cb' was not declared. Should it be static?
mm/percpu.c:2124:15: warning: symbol '__alloc_reserved_percpu_cb' was not declared. Should it be static?
--- percpu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/percpu.c +2 -2
@@ -2121,8 +2121,8 @@ void __percpu *__alloc_percpu_cb(size_t size, size_t align, pcpu_cpuhp_fn_t cb, }EXPORT_SYMBOL_GPL (__alloc_percpu_cb); -void __percpu *__alloc_reserved_percpu_cb (size_t size,size_t align, -pcpu_cpuhp_fn_t cb,void *data) +static void __percpu *__alloc_reserved_percpu_cb (size_t size,size_t align, +pcpu_cpuhp_fn_t cb,void *data) {return pcpu_alloc (size, align, true,GFP_KERNEL , cb, data); }
RFC PATCH 2/3 percpu: Limit percpu allocator to online cpus
In reply to Bharata B RaoNow that percpu allocator supports growing of memory for newly coming up CPU at hotplug time, limit the allocation, mapping and memcg charging of memory to online CPUs.
Also change the Percpu memory reporting in /proc/meminfo to reflect the populated pages of only online CPUs.
TODO: Address percpu memcg charging and uncharging from CPU hotplug callback.
--- mm/percpu-vm.c | 12 ++++++------ mm/percpu.c | 20 +++++++++++++------- 2 files changed, 19 insertions(+), 13 deletions(-)
mm/percpu-vm.c +6 -6
@@ -118,7 +118,7 @@ static void pcpu_free_pages(struct pcpu_chunk *chunk,unsigned int cpu;int i; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {for (i = page_start; i < page_end; i++) {struct page *page = pages[pcpu_page_idx (cpu, i)];
@@ -149,7 +149,7 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, gfp |= __GFP_HIGHMEM; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {for (i = page_start; i < page_end; i++) {struct page **pagep = &pages[pcpu_page_idx (cpu, i)];
@@ -164,7 +164,7 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk,while (--i >= page_start)__free_page (pages[pcpu_page_idx (cpu, i)]); -for_each_possible_cpu (tcpu) { +for_each_online_cpu (tcpu) {if (tcpu == cpu)break ;for (i = page_start; i < page_end; i++)
@@ -248,7 +248,7 @@ static void pcpu_unmap_pages(struct pcpu_chunk *chunk,unsigned int cpu;int i; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {for (i = page_start; i < page_end; i++) {struct page *page;
@@ -344,7 +344,7 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk,unsigned int cpu, tcpu;int i, err; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) { err =__pcpu_map_pages (pcpu_chunk_addr (chunk, cpu, page_start), &pages[pcpu_page_idx (cpu, page_start)], page_end - page_start);
@@ -357,7 +357,7 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, }return 0 ; err: -for_each_possible_cpu (tcpu) { +for_each_online_cpu (tcpu) {if (tcpu == cpu)break ; __pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
mm/percpu.c +13 -7
@@ -1594,7 +1594,7 @@ static enum pcpu_chunk_type pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp,if (!objcg)return PCPU_CHUNK_ROOT ; -if (obj_cgroup_charge (objcg, gfp, size *num_possible_cpus ())) { +if (obj_cgroup_charge (objcg, gfp, size *num_online_cpus ())) {obj_cgroup_put (objcg);return PCPU_FAIL_ALLOC ; }
@@ -1615,10 +1615,10 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg,rcu_read_lock ();mod_memcg_state (obj_cgroup_memcg (objcg),MEMCG_PERCPU_B , - size *num_possible_cpus ()); + size *num_online_cpus ());rcu_read_unlock (); }else { -obj_cgroup_uncharge (objcg, size *num_possible_cpus ()); +obj_cgroup_uncharge (objcg, size *num_online_cpus ());obj_cgroup_put (objcg); } }
@@ -1633,11 +1633,11 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) objcg = chunk->obj_cgroups[off >>PCPU_MIN_ALLOC_SHIFT ]; chunk->obj_cgroups[off >>PCPU_MIN_ALLOC_SHIFT ] =NULL ; -obj_cgroup_uncharge (objcg, size *num_possible_cpus ()); +obj_cgroup_uncharge (objcg, size *num_online_cpus ());rcu_read_lock ();mod_memcg_state (obj_cgroup_memcg (objcg),MEMCG_PERCPU_B , - -(size *num_possible_cpus ())); + -(size *num_online_cpus ()));rcu_read_unlock ();obj_cgroup_put (objcg);
@@ -1680,6 +1680,9 @@ static void pcpu_cpuhp_deregister(struct pcpu_chunk *chunk, } } +/* +* TODO: Grow the memcg charge +*/ static void __pcpu_cpuhp_setup (enum pcpu_chunk_type type,unsigned int cpu) {int slot;
@@ -1720,6 +1723,9 @@ static int percpu_cpuhp_setup(unsigned int cpu)return 0 ; } +/* +* TODO: Reduce the memcg charge +*/ static void __pcpu_cpuhp_destroy (enum pcpu_chunk_type type,unsigned int cpu) {int slot;
@@ -2000,7 +2006,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,pcpu_schedule_balance_work ();/* clear the areas and return address relative to base address */ -for_each_possible_cpu (cpu) +for_each_online_cpu (cpu)memset ((void *)pcpu_chunk_addr (chunk, cpu,0 ) + off,0 , size); ptr =__addr_to_pcpu_ptr (chunk->base_addr + off);
@@ -3372,7 +3378,7 @@ void __init setup_per_cpu_areas(void)*/ unsigned long pcpu_nr_pages (void ) { -return pcpu_nr_populated * pcpu_nr_units; +return pcpu_nr_populated *num_online_cpus (); }/*
Signature · 1 line
2.31.1
RFC PATCH 3/3 percpu: Avoid using percpu ptrs of non-existing cpus
In reply to Bharata B RaoPrevent the callers of alloc_percpu() from using the percpu pointer of non-existing CPUs. Also switch those callers who require initialization of percpu data for onlined CPU to use the new variant alloc_percpu_cb()
Note: Not all callers have been modified here
--- fs/namespace.c | 4 ++-- kernel/cgroup/rstat.c | 20 ++++++++++++++++---- kernel/sched/cpuacct.c | 10 +++++----- kernel/sched/psi.c | 14 +++++++++++--- lib/percpu-refcount.c | 4 ++-- lib/percpu_counter.c | 2 +- net/ipv4/fib_semantics.c | 2 +- net/ipv6/route.c | 6 +++--- 8 files changed, 41 insertions(+), 21 deletions(-)
fs/namespace.c +2 -2
@@ -182,7 +182,7 @@ int mnt_get_count(struct mount *mnt)int count =0 ;int cpu; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) { count +=per_cpu_ptr (mnt->mnt_pcp, cpu)->mnt_count; }
@@ -294,7 +294,7 @@ static unsigned int mnt_get_writers(struct mount *mnt)unsigned int count =0 ;int cpu; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) { count +=per_cpu_ptr (mnt->mnt_pcp, cpu)->mnt_writers; }
kernel/cgroup/rstat.c +16 -4
@@ -152,7 +152,7 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep)lockdep_assert_held (&cgroup_rstat_lock); -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {raw_spinlock_t *cpu_lock =per_cpu_ptr (&cgroup_rstat_cpu_lock, cpu);struct cgroup *pos =NULL ;
@@ -245,19 +245,31 @@ void cgroup_rstat_flush_release(void)spin_unlock_irq (&cgroup_rstat_lock); } +static int cgroup_rstat_cpuhp_handler (void __percpu *ptr,unsigned int cpu,void *data) +{ +struct cgroup *cgrp = (struct cgroup *)data; +struct cgroup_rstat_cpu *rstatc =per_cpu_ptr (ptr, cpu); + + rstatc->updated_children = cgrp; +u64_stats_init (&rstatc->bsync); +return 0 ; +} +int cgroup_rstat_init (struct cgroup *cgrp) {int cpu;/* the root cgrp has rstat_cpu preallocated */ if (!cgrp->rstat_cpu) { - cgrp->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); + cgrp->rstat_cpu = alloc_percpu_cb(struct cgroup_rstat_cpu, + cgroup_rstat_cpuhp_handler, + cgrp);if (!cgrp->rstat_cpu)return -ENOMEM ; }/* ->updated_children list is self terminated */ -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct cgroup_rstat_cpu *rstatc =cgroup_rstat_cpu (cgrp, cpu); rstatc->updated_children = cgrp;
@@ -274,7 +286,7 @@ void cgroup_rstat_exit(struct cgroup *cgrp)cgroup_rstat_flush (cgrp);/* sanity check */ -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct cgroup_rstat_cpu *rstatc =cgroup_rstat_cpu (cgrp, cpu); if (WARN_ON_ONCE(rstatc->updated_children != cgrp) ||
kernel/sched/cpuacct.c +5 -5
@@ -160,7 +160,7 @@ static u64 __cpuusage_read(struct cgroup_subsys_state *css,u64 totalcpuusage =0 ;int i; -for_each_possible_cpu (i) +for_each_online_cpu (i) totalcpuusage +=cpuacct_cpuusage_read (ca, i, index);return totalcpuusage;
@@ -195,7 +195,7 @@ static int cpuusage_write(struct cgroup_subsys_state *css, struct cftype *cft,if (val)return -EINVAL ; -for_each_possible_cpu (cpu) +for_each_online_cpu (cpu)cpuacct_cpuusage_write (ca, cpu,0 );return 0 ;
@@ -208,7 +208,7 @@ static int __cpuacct_percpu_seq_show(struct seq_file *m,u64 percpu;int i; -for_each_possible_cpu (i) { +for_each_online_cpu (i) { percpu =cpuacct_cpuusage_read (ca, i, index);seq_printf (m,"%llu " , (unsigned long long ) percpu); }
@@ -242,7 +242,7 @@ static int cpuacct_all_seq_show(struct seq_file *m, void *V)seq_printf (m," %s" , cpuacct_stat_desc[index]);seq_puts (m,"\n" ); -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct cpuacct_usage *cpuusage =per_cpu_ptr (ca->cpuusage, cpu);seq_printf (m,"%d" , cpu);
@@ -275,7 +275,7 @@ static int cpuacct_stats_show(struct seq_file *sf, void *v)int stat;memset (val,0 ,sizeof (val)); -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {u64 *cpustat =per_cpu_ptr (ca->cpustat, cpu)->cpustat; val[CPUACCT_STAT_USER ] += cpustat[CPUTIME_USER ];
kernel/sched/psi.c +11 -3
@@ -186,7 +186,7 @@ static void group_init(struct psi_group *group) {int cpu; -for_each_possible_cpu (cpu) +for_each_online_cpu (cpu)seqcount_init (&per_cpu_ptr (group->pcpu, cpu)->seq); group->avg_last_update =sched_clock (); group->avg_next_update = group->avg_last_update + psi_period;
@@ -321,7 +321,7 @@ static void collect_percpu_times(struct psi_group *group,* the sampling period. This eliminates artifacts from uneven * loading, or even entirely idle CPUs. */ -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {u32 times[NR_PSI_STATES ];u32 nonidle;u32 cpu_changed_states;
@@ -935,12 +935,20 @@ void psi_memstall_leave(unsigned long *flags) }#ifdef CONFIG_CGROUPS +static int psi_cpuhp_handler (void __percpu *ptr,unsigned int cpu,void *unused) +{ +struct psi_group_cpu *groupc =per_cpu_ptr (ptr, cpu); + +seqcount_init (&groupc->seq); +return 0 ; +} +int psi_cgroup_alloc (struct cgroup *cgroup) {if (static_branch_likely (&psi_disabled))return 0 ; - cgroup->psi.pcpu = alloc_percpu(struct psi_group_cpu); + cgroup->psi.pcpu = alloc_percpu_cb(struct psi_group_cpu, psi_cpuhp_handler, NULL);if (!cgroup->psi.pcpu)return -ENOMEM ;group_init (&cgroup->psi);
lib/percpu-refcount.c +2 -2
@@ -173,7 +173,7 @@ static void percpu_ref_switch_to_atomic_rcu(struct rcu_head *rcu)unsigned long count =0 ;int cpu; -for_each_possible_cpu (cpu) +for_each_online_cpu (cpu) count += *per_cpu_ptr (percpu_count, cpu); pr_debug("global %lu percpu %lu\n",
@@ -253,7 +253,7 @@ static void __percpu_ref_switch_to_percpu(struct percpu_ref *ref)* zeroing is visible to all percpu accesses which can see the * following __PERCPU_REF_ATOMIC clearing. */ -for_each_possible_cpu (cpu) +for_each_online_cpu (cpu) *per_cpu_ptr (percpu_count, cpu) =0 ; smp_store_release(&ref->percpu_count_ptr,
lib/percpu_counter.c +1 -1
@@ -63,7 +63,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount)unsigned long flags;raw_spin_lock_irqsave (&fbc->lock, flags); -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {s32 *pcount =per_cpu_ptr (fbc->counters, cpu); *pcount =0 ; }
net/ipv4/fib_semantics.c +1 -1
@@ -194,7 +194,7 @@ static void rt_fibinfo_free_cpus(struct rtable __rcu * __percpu *rtp)if (!rtp)return ; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct rtable *rt; rt =rcu_dereference_protected (*per_cpu_ptr (rtp, cpu),1 );
net/ipv6/route.c +3 -3
@@ -165,7 +165,7 @@ static void rt6_uncached_list_flush_dev(struct net *net, struct net_device *dev)if (dev == loopback_dev)return ; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct uncached_list *ul =per_cpu_ptr (&rt6_uncached_list, cpu);struct rt6_info *rt;
@@ -3542,7 +3542,7 @@ void fib6_nh_release(struct fib6_nh *fib6_nh)if (fib6_nh->rt6i_pcpu) {int cpu; -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct rt6_info **ppcpu_rt;struct rt6_info *pcpu_rt;
@@ -6569,7 +6569,7 @@ int __init ip6_route_init(void)#endif #endif -for_each_possible_cpu (cpu) { +for_each_online_cpu (cpu) {struct uncached_list *ul =per_cpu_ptr (&rt6_uncached_list, cpu);INIT_LIST_HEAD (&ul->head);
Signature · 1 line
2.31.1
Dennis Zhou
maintainerHello,
Quoted 30 lines · Bharata B Rao
Hi,
This is an attempt to make the percpu allocator CPU hotplug aware. Currently the percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and when they become online.
This early RFC work shows some good memory savings for a powerpc KVM guest that is booted with 16 online and 1024 possible CPUs. Here is the comparision of Percpu memory consumption from
/proc/meminfobefore and after creating 1000 memcgs.W/o patch W/ patch
Before 1441792 kB 22528 kB
After 1000 memcgs 4390912 kB 68608 kBNote that the Percpu reporting in meminfo has been changed in the patchset to reflect the allocation for online CPUs only.
More details about the approach are present in the patch descriptions.
Bharata B Rao (3):
percpu: CPU hotplug support foralloc_percpu()
percpu: Limit percpu allocator to online cpus
percpu: Avoid using percpu ptrs of non-existing cpus
Quoted 18 lines
fs/namespace.c | 4 +- include/linux/cpuhotplug.h | 2 + include/linux/percpu.h | 15 +++ kernel/cgroup/rstat.c | 20 +++- kernel/sched/cpuacct.c | 10 +- kernel/sched/psi.c | 14 ++- lib/percpu-refcount.c | 4 +- lib/percpu_counter.c | 2 +- mm/percpu-internal.h | 9 ++ mm/percpu-vm.c | 211 +++++++++++++++++++++++++++++++++- mm/percpu.c | 229 +++++++++++++++++++++++++++++++++++-- net/ipv4/fib_semantics.c | 2 +- net/ipv6/route.c | 6 +- 13 files changed, 490 insertions(+), 38 deletions(-)Signature · 1 line
2.31.1
I have thought about this for a day now and to be honest my thoughts haven't really changed since the last discussion in [1].
I struggle here for a few reasons:
1. We're intertwining cpu and memory for hotplug.
- What does it mean if we don't have enough memory?
- How hard do we try to reclaim memory?
- Partially allocated cpus? Do we free it all and try again?
2. We're now blocking the whole system on the percpu mutex which can cause terrible side effects. If there is a large amount of percpu memory already in use, this means we've accumulated a substantial number of callbacks.
3. While I did mention a callback approach would work. I'm not thrilled by the additional complexity of it as it can be error prone.
Beyond the above. I still don't believe it's the most well motivated problem. I struggle to see a world where it makes sense to let someone scale from 16 cpus to 1024. As in my mind you would also need to scale memory to some degree too (not necessarily linearly but a 1024 core machine with say like 16 gigs of ram would be pretty funny).
Would it be that bad to use cold migration points and eat a little bit of overhead for what I understand to be a relatively uncommon use case?
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Dennis
Quoted 23 lines · Dennis Zhou
Hello,
Quoted 10 lines · Bharata B Rao
Hi,
This is an attempt to make the percpu allocator CPU hotplug aware. Currently the percpu allocator allocates memory for all the possible CPUs. This can lead to wastage of memory when possible number of CPUs is significantly higher than the number of online CPUs. This can be avoided if the percpu allocator were to allocate only for the online CPUs and extend the allocation for other CPUs as and when they become online.
Quoted 9 lines
This early RFC work shows some good memory savings for a powerpc KVM guest that is booted with 16 online and 1024 possible CPUs. Here is the comparision of Percpu memory consumption from
/proc/meminfobefore and after creating 1000 memcgs.W/o patch W/ patch
Before 1441792 kB 22528 kB
After 1000 memcgs 4390912 kB 68608 kB
Quoted 6 lines
I have thought about this for a day now and to be honest my thoughts haven't really changed since the last discussion in [1].
I struggle here for a few reasons:
1. We're intertwining cpu and memory for hotplug.
- What does it mean if we don't have enough memory?
That means CPU hotplug will fail, but...
Quoted 2 lines
- How hard do we try to reclaim memory?
- Partially allocated cpus? Do we free it all and try again?
... yes these are some difficult questions. We should check if roll back can be done cleanly and efficiently. You can see that I am registering separate hotplug callbacks for the hotplug core and for init routines of alloc_percpu() callers. Rolling back the former should be fairly straight forward, but have to see how desirable and feasible it is to undo the entire CPU hotplug when one of the alloc_percpu callbacks fails, especially if there are hundreds of registered alloc_percpu callbacks.
Quoted 4 lines
2. We're now blocking the whole system on the percpu mutex which can cause terrible side effects. If there is a large amount of percpu memory already in use, this means we've accumulated a substantial number of callbacks.
I am yet to look at each caller in detail and see which of them really need init/free callbacks and which can do without it. After this we will have to measure the overhead all this is putting on the hotplug path. Given that hotplug is a slow path, I wonder if some overhead is tolerable here.
CPU hotplug already happens with cpu_hotplug_lock held, so when you mention that this callback holding percpu mutex can have terrible effects, are you specifically worried about blocking all the percpu allocation requests during hotplug? Or is it something else?
Quoted 2 lines
3. While I did mention a callback approach would work. I'm not thrilled by the additional complexity of it as it can be error prone.
Fair enough, the callback for the percpu allocator core seems fine to me but since I haven't yet looked at all callers in detail, I don't know if we would run into some issues/dependencies in any specific callback handlers that increases the overall complexity.
Other than the callbacks, I am also bit worried about the complexity and the overhead involved in memcg charging and uncharging at CPU hotplug time. In my environment (powerpc kvm guest), I see that each chunk can have a maximum of 180224 obj_cgroups. Now checking for the valid/used one out of that, determining the allocation size and charging/uncharging to the right memcg could be an expensive task.
Quoted 6 lines
Beyond the above. I still don't believe it's the most well motivated problem. I struggle to see a world where it makes sense to let someone scale from 16 cpus to 1024. As in my mind you would also need to scale memory to some degree too (not necessarily linearly but a 1024 core machine with say like 16 gigs of ram would be pretty funny).
Well the platform here provides the capability of scaling and until that scaling happens, why consume memory for not-present CPUs is the motivation here. But as you note, it definetely is a question of whether any real application is making use of this scaling now and the associated complexity.
Even if we consider the scaling from 16 to 1024 CPUs as unrealistic for now, the usecase and the numbers from the production scenario that Alexey mentioned in [1] (2 to 128 CPUs) is certainly a good motivator?
Alexey - You did mention about creating a huge number of memcgs and observing VMs consuming 16-20 GB percpu memory in production. So how any memcgs are we talking about here?
Quoted 3 lines
Would it be that bad to use cold migration points and eat a little bit of overhead for what I understand to be a relatively uncommon use case?
[1] https://lore.kernel.org/linux-mm/[email protected]/
Regards,
Bharata.