mm/slub: Optimize slub memory usage
Subsystems (3)
RFC PATCH v3 1/1 mm/slub: Optimize slub memory usage
In the current implementation of the slub memory allocator, the slab order selection process follows these criteria:
1) Determine the minimum order required to serve the minimum number of objects (min_objects). This calculation is based on the formula (order = min_objects * object_size / PAGE_SIZE). 2) If the minimum order is greater than the maximum allowed order (slub_max_order), set slub_max_order as the order for this slab. 3) If the minimum order is less than the slub_max_order, iterate through a loop from minimum order to slub_max_order and check if the condition (rem <= slab_size / fract_leftover) holds true. Here, slab_size is calculated as (PAGE_SIZE << order), rem is (slab_size % object_size), and fract_leftover can have values of 16, 8, or 4. If the condition is true, select that order for the slab.
However, in point 3, when calculating the fraction left over, it can result in a large range of values (like 1 Kb to 256 bytes on 4K page size & 4 Kb to 16 Kb on 64K page size with order 0 and goes on increasing with higher order) when compared to the remainder (rem). This can lead to the selection of an order that results in more memory wastage. To mitigate such wastage, we have modified point 3 as follows:
To adjust the value of fract_leftover based on the page size, while retaining the current value as the default for a 4K page size.
Test results are as follows:
1) On 160 CPUs with 64K Page size
+-----------------+----------------+----------------+ | Total wastage in slub memory | +-----------------+----------------+----------------+ | | After Boot |After Hackbench | | Normal | 932 Kb | 1812 Kb | | With Patch | 729 Kb | 1636 Kb | | Wastage reduce | ~22% | ~10% | +-----------------+----------------+----------------+
+-----------------+----------------+----------------+ | Total slub memory | +-----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 1855296 | 2944576 | | With Patch | 1544576 | 2692032 | | Memory reduce | ~17% | ~9% | +-----------------+----------------+----------------+
hackbench-process-sockets
+-------+-----+----------+----------+-----------+ | Amean | 1 | 1.2727 | 1.2450 | ( 2.22%) | | Amean | 4 | 1.6063 | 1.5810 | ( 1.60%) | | Amean | 7 | 2.4190 | 2.3983 | ( 0.86%) | | Amean | 12 | 3.9730 | 3.9347 | ( 0.97%) | | Amean | 21 | 6.9823 | 6.8957 | ( 1.26%) | | Amean | 30 | 10.1867 | 10.0600 | ( 1.26%) | | Amean | 48 | 16.7490 | 16.4853 | ( 1.60%) | | Amean | 79 | 28.1870 | 27.8673 | ( 1.15%) | | Amean | 110 | 39.8363 | 39.3793 | ( 1.16%) | | Amean | 141 | 51.5277 | 51.4907 | ( 0.07%) | | Amean | 172 | 62.9700 | 62.7300 | ( 0.38%) | | Amean | 203 | 74.5037 | 74.0630 | ( 0.59%) | | Amean | 234 | 85.6560 | 85.3587 | ( 0.35%) | | Amean | 265 | 96.9883 | 96.3770 | ( 0.63%) | | Amean | 296 | 108.6893 | 108.0870 | ( 0.56%) | +-------+-----+----------+----------+-----------+
2) On 16 CPUs with 64K Page size
+----------------+----------------+----------------+ | Total wastage in slub memory | +----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 273 Kb | 544 Kb | | With Patch | 260 Kb | 500 Kb | | Wastage reduce | ~5% | ~9% | +----------------+----------------+----------------+
+-----------------+----------------+----------------+ | Total slub memory | +-----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 275840 | 412480 | | With Patch | 272768 | 406208 | | Memory reduce | ~1% | ~2% | +-----------------+----------------+----------------+
hackbench-process-sockets
+-------+----+---------+---------+-----------+ | Amean | 1 | 0.9513 | 0.9250 | ( 2.77%) | | Amean | 4 | 2.9630 | 2.9570 | ( 0.20%) | | Amean | 7 | 5.1780 | 5.1763 | ( 0.03%) | | Amean | 12 | 8.8833 | 8.8817 | ( 0.02%) | | Amean | 21 | 15.7577 | 15.6883 | ( 0.44%) | | Amean | 30 | 22.2063 | 22.2843 | ( -0.35%) | | Amean | 48 | 36.0587 | 36.1390 | ( -0.22%) | | Amean | 64 | 49.7803 | 49.3457 | ( 0.87%) | +-------+----+---------+---------+-----------+
---
Changes from V2
1) removed all page order selection logic for slab cache base on wastage.
2) Increasing fraction size base on page size (keeping current value as default to 4K page)
Changes from V1
1) If min_objects * object_size > PAGE_ALLOC_COSTLY_ORDER, then it will return with PAGE_ALLOC_COSTLY_ORDER.
2) Similarly, if min_objects * object_size < PAGE_SIZE, then it will return with slub_min_order.
3) Additionally, I changed slub_max_order to 2. There is no specific reason for using the value 2, but it provided the best results in terms of performance without any noticeable impact.
arch/powerpc/include/asm/page.h | 2 ++ mm/slub.c | 15 +++++---------- 2 files changed, 7 insertions(+), 10 deletions(-)
arch/powerpc/include/asm/page.h +2 -0
@@ -22,6 +22,8 @@*/ #define PAGE_SHIFT CONFIG_PPC_PAGE_SHIFT#define PAGE_SIZE (ASM_CONST(1) << PAGE_SHIFT) +#define PAGE_SHIFT_4K 12 +#define PAGE_SIZE_4K (1 << PAGE_SHIFT_4K)#ifndef __ASSEMBLY__#ifndef CONFIG_HUGETLB_PAGE
mm/slub.c +5 -10
@@ -4117,6 +4117,7 @@ static inline int calculate_order(unsigned int size)unsigned int min_objects;unsigned int max_objects;unsigned int nr_cpus; +unsigned int page_frac;/* * Attempt to find best configuration for a slab. This
@@ -4145,10 +4146,12 @@ static inline int calculate_order(unsigned int size) max_objects =order_objects (slub_max_order, size); min_objects =min (min_objects, max_objects); -while (min_objects >1 ) { + page_frac = ((PAGE_SIZE /PAGE_SIZE_4K ) ==1 ) ?0 :PAGE_SIZE /PAGE_SIZE_4K ; + +while (min_objects >=1 ) {unsigned int fraction; - fraction =16 ; + fraction =16 + page_frac;while (fraction >=4 ) { order =calc_slab_order (size, min_objects, slub_max_order, fraction);
@@ -4159,14 +4162,6 @@ static inline int calculate_order(unsigned int size) min_objects--; } -/* -* We were unable to place multiple objects in a slab. Now -* lets see if we can place a single object there. -*/ - order =calc_slab_order (size,1 , slub_max_order,1 ); -if (order <= slub_max_order) -return order; -/* * Doh this slab cannot be placed using slub_max_order. */
Signature · 1 line
2.39.1
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Jay,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Jay-Patel/mm-slub-Optimize-slub-memory-usage/20230719-182356 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230719102104.1954891-1-jaypatel%40linux.ibm.com patch subject: [RFC PATCH v3] mm/slub: Optimize slub memory usage config: i386-randconfig-r024-20230718 (https://download.01.org/0day-ci/archive/20230719/[email protected]/config) compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a) reproduce: (https://download.01.org/0day-ci/archive/20230719/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> mm/slub.c:4164:26: error: use of undeclared identifier 'PAGE_SIZE_4K'
page_frac = ((PAGE_SIZE/PAGE_SIZE_4K) == 1) ? 0 : PAGE_SIZE/PAGE_SIZE_4K;
^
mm/slub.c:4164:62: error: use of undeclared identifier 'PAGE_SIZE_4K'
page_frac = ((PAGE_SIZE/PAGE_SIZE_4K) == 1) ? 0 : PAGE_SIZE/PAGE_SIZE_4K;
^2 errors generated.
vim +/PAGE_SIZE_4K +4164 mm/slub.c
4128
4129 static inline int calculate_order(unsigned int size)
4130 {
4131 unsigned int order;
4132 unsigned int min_objects;
4133 unsigned int max_objects;
4134 unsigned int nr_cpus;
4135 unsigned int page_frac;
4136
4137 /*
4138 * Attempt to find best configuration for a slab. This
4139 * works by first attempting to generate a layout with
4140 * the best configuration and backing off gradually.
4141 *
4142 * First we increase the acceptable waste in a slab. Then
4143 * we reduce the minimum objects required in a slab.
4144 */
4145 min_objects = slub_min_objects;
4146 if (!min_objects) {
4147 /*
4148 * Some architectures will only update present cpus when
4149 * onlining them, so don't trust the number if it's just 1. But
4150 * we also don't want to use nr_cpu_ids always, as on some other
4151 * architectures, there can be many possible cpus, but never
4152 * onlined. Here we compromise between trying to avoid too high
4153 * order on systems that appear larger than they are, and too
4154 * low order on systems that appear smaller than they are.
4155 */
4156 nr_cpus = num_present_cpus();
4157 if (nr_cpus <= 1)
4158 nr_cpus = nr_cpu_ids;
4159 min_objects = 4 * (fls(nr_cpus) + 1);
4160 }
4161 max_objects = order_objects(slub_max_order, size);
4162 min_objects = min(min_objects, max_objects);
4163
> 4164 page_frac = ((PAGE_SIZE/PAGE_SIZE_4K) == 1) ? 0 : PAGE_SIZE/PAGE_SIZE_4K;
4165
4166 while (min_objects >= 1) {
4167 unsigned int fraction;
4168
4169 fraction = 16 + page_frac;
4170 while (fraction >= 4) {
4171 order = calc_slab_order(size, min_objects,
4172 slub_max_order, fraction);
4173 if (order <= slub_max_order)
4174 return order;
4175 fraction /= 2;
4176 }
4177 min_objects--;
4178 }
4179
4180 /*
4181 * Doh this slab cannot be placed using slub_max_order.
4182 */
4183 order = calc_slab_order(size, 1, MAX_ORDER, 1);
4184 if (order <= MAX_ORDER)
4185 return order;
4186 return -ENOSYS;
4187 }
4188
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Jay,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Jay-Patel/mm-slub-Optimize-slub-memory-usage/20230719-182356 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230719102104.1954891-1-jaypatel%40linux.ibm.com patch subject: [RFC PATCH v3] mm/slub: Optimize slub memory usage config: alpha-randconfig-r012-20230718 (https://download.01.org/0day-ci/archive/20230720/[email protected]/config) compiler: alpha-linux-gcc (GCC) 12.3.0 reproduce: (https://download.01.org/0day-ci/archive/20230720/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/slub.c: In function 'calculate_order':
>> mm/slub.c:4164:33: error: 'PAGE_SIZE_4K' undeclared (first use in this function); did you mean 'PAGE_SIZE'?
4164 | page_frac = ((PAGE_SIZE/PAGE_SIZE_4K) == 1) ? 0 : PAGE_SIZE/PAGE_SIZE_4K;
| ^~~~~~~~~~~~
| PAGE_SIZE mm/slub.c:4164:33: note: each undeclared identifier is reported only once for each function it appears in
vim +4164 mm/slub.c
4128
4129 static inline int calculate_order(unsigned int size)
4130 {
4131 unsigned int order;
4132 unsigned int min_objects;
4133 unsigned int max_objects;
4134 unsigned int nr_cpus;
4135 unsigned int page_frac;
4136
4137 /*
4138 * Attempt to find best configuration for a slab. This
4139 * works by first attempting to generate a layout with
4140 * the best configuration and backing off gradually.
4141 *
4142 * First we increase the acceptable waste in a slab. Then
4143 * we reduce the minimum objects required in a slab.
4144 */
4145 min_objects = slub_min_objects;
4146 if (!min_objects) {
4147 /*
4148 * Some architectures will only update present cpus when
4149 * onlining them, so don't trust the number if it's just 1. But
4150 * we also don't want to use nr_cpu_ids always, as on some other
4151 * architectures, there can be many possible cpus, but never
4152 * onlined. Here we compromise between trying to avoid too high
4153 * order on systems that appear larger than they are, and too
4154 * low order on systems that appear smaller than they are.
4155 */
4156 nr_cpus = num_present_cpus();
4157 if (nr_cpus <= 1)
4158 nr_cpus = nr_cpu_ids;
4159 min_objects = 4 * (fls(nr_cpus) + 1);
4160 }
4161 max_objects = order_objects(slub_max_order, size);
4162 min_objects = min(min_objects, max_objects);
4163
> 4164 page_frac = ((PAGE_SIZE/PAGE_SIZE_4K) == 1) ? 0 : PAGE_SIZE/PAGE_SIZE_4K;
4165
4166 while (min_objects >= 1) {
4167 unsigned int fraction;
4168
4169 fraction = 16 + page_frac;
4170 while (fraction >= 4) {
4171 order = calc_slab_order(size, min_objects,
4172 slub_max_order, fraction);
4173 if (order <= slub_max_order)
4174 return order;
4175 fraction /= 2;
4176 }
4177 min_objects--;
4178 }
4179
4180 /*
4181 * Doh this slab cannot be placed using slub_max_order.
4182 */
4183 order = calc_slab_order(size, 1, MAX_ORDER, 1);
4184 if (order <= MAX_ORDER)
4185 return order;
4186 return -ENOSYS;
4187 }
4188
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki