Memory allocation profiling
Subsystems (83)
Suren Baghdasaryan
maintainerv4 cover Memory allocation profiling
Overview:
Low overhead [1] per-callsite memory allocation profiling. Not just for debug kernels, overhead low enough to be deployed in production.
Example output:
root@moria-kvm:~# sort -rn /proc/allocinfo
127664128 31168 mm/page_ext.c:270 func:alloc_page_ext
56373248 4737 mm/slub.c:2259 func:alloc_slab_page
14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded
14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash
13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs
11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio
9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node
4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable
4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start
3940352 962 mm/memory.c:4214 func:alloc_anon_folio
2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node
...Since v3:
- Dropped patch changing string_get_size() [2] as not needed
- Dropped patch modifying xfs allocators [3] as non needed, per Dave Chinner
- Added Reviewed-by, per Kees Cook
- Moved prepare_slab_obj_exts_hook() and alloc_slab_obj_exts() where they are used, per Vlastimil Babka
- Fixed SLAB_NO_OBJ_EXT definition to use unused bit, per Vlastimil Babka
- Refactored patch [4] into other patches, per Vlastimil Babka
- Replaced snprintf() with seq_buf_printf(), per Kees Cook
- Changed output to report bytes, per Andrew Morton and Pasha Tatashin
- Changed output to report [module] only for loadable modules, per Vlastimil Babka
- Moved mem_alloc_profiling_enabled() check earlier, per Vlastimil Babka
- Changed the code to handle page splitting to be more understandable, per Vlastimil Babka
- Moved alloc_tagging_slab_free_hook(), mark_objexts_empty(), mark_failed_objexts_alloc() and handle_failed_objexts_alloc(), per Vlastimil Babka
- Fixed loss of __alloc_size(1, 2) in kvmalloc functions,
per Vlastimil Babka
- Refactored the code in show_mem() to avoid memory allocations, per Michal Hocko
- Changed to trylock in show_mem() to avoid blocking in atomic context, per Tetsuo Handa
- Added mm mailing list into MAINTAINERS, per Kees Cook
- Added base commit SHA, per Andy Shevchenko
- Added a patch with documentation, per Jani Nikula
- Fixed 0day bugs
- Added benchmark results [5], per Steven Rostedt
- Rebased over Linux 6.8-rc5
Items not yet addressed:
- An early_boot option to prevent pageext overhead. We are looking into ways for using the same sysctr instead of adding additional early boot parameter.
Usage:
kconfig options:
- CONFIG_MEM_ALLOC_PROFILING
- CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT
- CONFIG_MEM_ALLOC_PROFILING_DEBUG
adds warnings for allocations that weren't accounted because of a missing annotation
sysctl:
/proc/sys/vm/mem_profiling
Runtime info:
/proc/allocinfo
Notes:
[1]: Overhead
To measure the overhead we are comparing the following configurations:
(1) Baseline with CONFIG_MEMCG_KMEM=n
(2) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=y && CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n)
(3) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=y && CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=y)
(4) Enabled at runtime (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n && /proc/sys/vm/mem_profiling=1)
(5) Baseline with CONFIG_MEMCG_KMEM=y && allocating with __GFP_ACCOUNT
(6) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n) && CONFIG_MEMCG_KMEM=y
(7) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=y) && CONFIG_MEMCG_KMEM=yPerformance overhead:
To evaluate performance we implemented an in-kernel test executing multiple get_free_page/free_page and kmalloc/kfree calls with allocation sizes growing from 8 to 240 bytes with CPU frequency set to max and CPU affinity set to a specific CPU to minimize the noise. Below are results from running the test on Ubuntu 22.04.2 LTS with 6.8.0-rc1 kernel on 56 core Intel Xeon:
kmalloc pgalloc (1 baseline) 6.764s 16.902s (2 default disabled) 6.793s (+0.43%) 17.007s (+0.62%) (3 default enabled) 7.197s (+6.40%) 23.666s (+40.02%) (4 runtime enabled) 7.405s (+9.48%) 23.901s (+41.41%) (5 memcg) 13.388s (+97.94%) 48.460s (+186.71%) (6 def disabled+memcg) 13.332s (+97.10%) 48.105s (+184.61%) (7 def enabled+memcg) 13.446s (+98.78%) 54.963s (+225.18%)
Memory overhead:
Kernel size:
text data bss dec diff
(1) 26515311 18890222 17018880 62424413 (2) 26524728 19423818 16740352 62688898 264485 (3) 26524724 19423818 16740352 62688894 264481 (4) 26524728 19423818 16740352 62688898 264485 (5) 26541782 18964374 16957440 62463596 39183
Memory consumption on a 56 core Intel CPU with 125GB of memory:
Code tags: 192 kB PageExts: 262144 kB (256MB) SlabExts: 9876 kB (9.6MB) PcpuExts: 512 kB (0.5MB)
Total overhead is 0.2% of total memory.
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] Benchmarks:
Hackbench tests run 100 times:
hackbench -s 512 -l 200 -g 15 -f 25 -P
baseline disabled profiling enabled profiling avg 0.3543 0.3559 (+0.0016) 0.3566 (+0.0023) stdev 0.0137 0.0188 0.0077
hackbench -l 10000
baseline disabled profiling enabled profiling avg 6.4218 6.4306 (+0.0088) 6.5077 (+0.0859) stdev 0.0933 0.0286 0.0489
stress-ng tests:
stress-ng --class memory --seq 4 -t 60
stress-ng --class cpu --seq 4 -t 60
Results posted at: https://evilpiepirate.org/~kent/memalloc_prof_v4_stress-ng/
Kent Overstreet (13):
fix missing vmalloc.h includes
asm-generic/io.h: Kill vmalloc.h dependency
mm/slub: Mark slab_free_freelist_hook() __always_inline scripts/kallysms: Always include __start and __stop symbols
fs: Convert alloc_inode_sb() to a macro
rust: Add a rust helper for krealloc()
mempool: Hook up to memory allocation profiling
mm: percpu: Introduce pcpuobj_ext
mm: percpu: Add codetag reference into pcpuobj_ext
mm: vmalloc: Enable memory allocation profiling
rhashtable: Plumb through alloc tag
MAINTAINERS: Add entries for code tagging and memory allocation
profiling
memprofiling: Documentation
Suren Baghdasaryan (23):
mm: enumerate all gfp flags
mm: introduce slabobj_ext to support slab object extensions
mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext
creation
mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation
slab: objext: introduce objext_flags as extension to
page_memcg_data_flags
lib: code tagging framework
lib: code tagging module support
lib: prevent module unloading if memory is not freed
lib: add allocation tagging support for memory allocation profiling
lib: introduce support for page allocation tagging
mm: percpu: increase PERCPU_MODULE_RESERVE to accommodate allocation
tags
change alloc_pages name in dma_map_ops to avoid name conflicts
mm: enable page allocation tagging
mm: create new codetag references during page splitting mm/page_ext: enable early_page_ext when
CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
lib: add codetag reference into slabobj_ext
mm/slab: add allocation accounting into slab allocation and free paths mm/slab: enable slab allocation tagging for kmalloc and friends
mm: percpu: enable per-cpu allocation tagging
lib: add memory allocations report in show_mem()
codetag: debug: skip objext checking when it's for objext itself
codetag: debug: mark codetags for reserved pages as empty
codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext
allocations
Documentation/admin-guide/sysctl/vm.rst | 16 + Documentation/filesystems/proc.rst | 29 ++ Documentation/mm/allocation-profiling.rst | 86 ++++++ MAINTAINERS | 17 ++ arch/alpha/kernel/pci_iommu.c | 2 +- arch/alpha/lib/checksum.c | 1 + arch/alpha/lib/fpreg.c | 1 + arch/alpha/lib/memcpy.c | 1 + arch/arm/kernel/irq.c | 1 + arch/arm/kernel/traps.c | 1 + arch/arm64/kernel/efi.c | 1 + arch/loongarch/include/asm/kfence.h | 1 + arch/mips/jazz/jazzdma.c | 2 +- arch/powerpc/kernel/dma-iommu.c | 2 +- arch/powerpc/kernel/iommu.c | 1 + arch/powerpc/mm/mem.c | 1 + arch/powerpc/platforms/ps3/system-bus.c | 4 +- arch/powerpc/platforms/pseries/vio.c | 2 +- arch/riscv/kernel/elf_kexec.c | 1 + arch/riscv/kernel/probes/kprobes.c | 1 + arch/s390/kernel/cert_store.c | 1 + arch/s390/kernel/ipl.c | 1 + arch/x86/include/asm/io.h | 1 + arch/x86/kernel/amd_gart_64.c | 2 +- arch/x86/kernel/cpu/sgx/main.c | 1 + arch/x86/kernel/irq_64.c | 1 + arch/x86/mm/fault.c | 1 + drivers/accel/ivpu/ivpu_mmu_context.c | 1 + drivers/gpu/drm/gma500/mmu.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_pages.c | 1 + .../gpu/drm/i915/gem/selftests/mock_dmabuf.c | 1 + drivers/gpu/drm/i915/gt/shmem_utils.c | 1 + drivers/gpu/drm/i915/gvt/firmware.c | 1 + drivers/gpu/drm/i915/gvt/gtt.c | 1 + drivers/gpu/drm/i915/gvt/handlers.c | 1 + drivers/gpu/drm/i915/gvt/mmio.c | 1 + drivers/gpu/drm/i915/gvt/vgpu.c | 1 + drivers/gpu/drm/i915/intel_gvt.c | 1 + drivers/gpu/drm/imagination/pvr_vm_mips.c | 1 + drivers/gpu/drm/mediatek/mtk_drm_gem.c | 1 + drivers/gpu/drm/omapdrm/omap_gem.c | 1 + drivers/gpu/drm/v3d/v3d_bo.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_binding.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c | 1 + drivers/gpu/drm/xen/xen_drm_front_gem.c | 1 + drivers/hwtracing/coresight/coresight-trbe.c | 1 + drivers/iommu/dma-iommu.c | 2 +- .../marvell/octeon_ep/octep_pfvf_mbox.c | 1 + .../net/ethernet/microsoft/mana/hw_channel.c | 1 + drivers/parisc/ccio-dma.c | 2 +- drivers/parisc/sba_iommu.c | 2 +- drivers/platform/x86/uv_sysfs.c | 1 + drivers/scsi/mpi3mr/mpi3mr_transport.c | 2 + drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- drivers/vfio/pci/pds/dirty.c | 1 + drivers/virt/acrn/mm.c | 1 + drivers/virtio/virtio_mem.c | 1 + drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/asm-generic/codetag.lds.h | 14 + include/asm-generic/io.h | 1 - include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 195 ++++++++++++ include/linux/codetag.h | 81 +++++ include/linux/dma-map-ops.h | 2 +- include/linux/fortify-string.h | 5 +- include/linux/fs.h | 6 +- include/linux/gfp.h | 126 +++++--- include/linux/gfp_types.h | 101 +++++-- include/linux/memcontrol.h | 56 +++- include/linux/mempool.h | 73 +++-- include/linux/mm.h | 9 + include/linux/mm_types.h | 4 +- include/linux/page_ext.h | 1 - include/linux/pagemap.h | 9 +- include/linux/pds/pds_common.h | 2 + include/linux/percpu.h | 27 +- include/linux/pgalloc_tag.h | 110 +++++++ include/linux/rhashtable-types.h | 11 +- include/linux/sched.h | 24 ++ include/linux/slab.h | 175 +++++------ include/linux/string.h | 4 +- include/linux/vmalloc.h | 60 +++- include/rdma/rdmavt_qp.h | 1 + init/Kconfig | 4 + kernel/dma/mapping.c | 4 +- kernel/kallsyms_selftest.c | 2 +- kernel/module/main.c | 25 +- lib/Kconfig.debug | 31 ++ lib/Makefile | 3 + lib/alloc_tag.c | 204 +++++++++++++ lib/codetag.c | 283 ++++++++++++++++++ lib/rhashtable.c | 28 +- mm/compaction.c | 7 +- mm/debug_vm_pgtable.c | 1 + mm/filemap.c | 6 +- mm/huge_memory.c | 2 + mm/kfence/core.c | 14 +- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 +--- mm/mempolicy.c | 52 ++-- mm/mempool.c | 36 +-- mm/mm_init.c | 13 +- mm/nommu.c | 64 ++-- mm/page_alloc.c | 66 ++-- mm/page_ext.c | 13 + mm/page_owner.c | 2 +- mm/percpu-internal.h | 26 +- mm/percpu.c | 120 +++----- mm/show_mem.c | 26 ++ mm/slab.h | 126 ++++++-- mm/slab_common.c | 6 +- mm/slub.c | 244 +++++++++++---- mm/util.c | 44 +-- mm/vmalloc.c | 88 +++--- rust/helpers.c | 8 + scripts/kallsyms.c | 13 + scripts/module.lds.S | 7 + sound/pci/hda/cs35l41_hda.c | 1 + 123 files changed, 2269 insertions(+), 682 deletions(-) create mode 100644 Documentation/mm/allocation-profiling.rst create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 include/linux/codetag.h create mode 100644 include/linux/pgalloc_tag.h create mode 100644 lib/alloc_tag.c create mode 100644 lib/codetag.c
base-commit: 39133352cbed6626956d38ed72012f49b0421e7b
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 1/36 fix missing vmalloc.h includes
From: Kent Overstreet <[email protected]>
The next patch drops vmalloc.h from a system header in order to fix a circular dependency; this adds it to all the files that were pulling it in implicitly.
--- arch/alpha/lib/checksum.c | 1 + arch/alpha/lib/fpreg.c | 1 + arch/alpha/lib/memcpy.c | 1 + arch/arm/kernel/irq.c | 1 + arch/arm/kernel/traps.c | 1 + arch/arm64/kernel/efi.c | 1 + arch/loongarch/include/asm/kfence.h | 1 + arch/powerpc/kernel/iommu.c | 1 + arch/powerpc/mm/mem.c | 1 + arch/riscv/kernel/elf_kexec.c | 1 + arch/riscv/kernel/probes/kprobes.c | 1 + arch/s390/kernel/cert_store.c | 1 + arch/s390/kernel/ipl.c | 1 + arch/x86/include/asm/io.h | 1 + arch/x86/kernel/cpu/sgx/main.c | 1 + arch/x86/kernel/irq_64.c | 1 + arch/x86/mm/fault.c | 1 + drivers/accel/ivpu/ivpu_mmu_context.c | 1 + drivers/gpu/drm/gma500/mmu.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_pages.c | 1 + drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c | 1 + drivers/gpu/drm/i915/gt/shmem_utils.c | 1 + drivers/gpu/drm/i915/gvt/firmware.c | 1 + drivers/gpu/drm/i915/gvt/gtt.c | 1 + drivers/gpu/drm/i915/gvt/handlers.c | 1 + drivers/gpu/drm/i915/gvt/mmio.c | 1 + drivers/gpu/drm/i915/gvt/vgpu.c | 1 + drivers/gpu/drm/i915/intel_gvt.c | 1 + drivers/gpu/drm/imagination/pvr_vm_mips.c | 1 + drivers/gpu/drm/mediatek/mtk_drm_gem.c | 1 + drivers/gpu/drm/omapdrm/omap_gem.c | 1 + drivers/gpu/drm/v3d/v3d_bo.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_binding.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c | 1 + drivers/gpu/drm/xen/xen_drm_front_gem.c | 1 + drivers/hwtracing/coresight/coresight-trbe.c | 1 + drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c | 1 + drivers/net/ethernet/microsoft/mana/hw_channel.c | 1 + drivers/platform/x86/uv_sysfs.c | 1 + drivers/scsi/mpi3mr/mpi3mr_transport.c | 2 ++ drivers/vfio/pci/pds/dirty.c | 1 + drivers/virt/acrn/mm.c | 1 + drivers/virtio/virtio_mem.c | 1 + include/linux/pds/pds_common.h | 2 ++ include/rdma/rdmavt_qp.h | 1 + mm/debug_vm_pgtable.c | 1 + sound/pci/hda/cs35l41_hda.c | 1 + 51 files changed, 53 insertions(+)
arch/alpha/lib/checksum.c +1 -0
@@ -14,6 +14,7 @@#include <linux/string.h> #include <asm/byteorder.h> +#include <asm/checksum.h> static inline unsigned short from64to16 (unsigned long x) {
arch/alpha/lib/fpreg.c +1 -0
@@ -8,6 +8,7 @@#include <linux/compiler.h> #include <linux/export.h> #include <linux/preempt.h> +#include <asm/fpu.h> #include <asm/thread_info.h> #if defined(CONFIG_ALPHA_EV6 ) || defined(CONFIG_ALPHA_EV67 )
arch/alpha/lib/memcpy.c +1 -0
@@ -18,6 +18,7 @@#include <linux/types.h> #include <linux/export.h> +#include <linux/string.h> /* * This should be done in one go with ldq_u*2/mask/stq_u. Do it
arch/arm/kernel/irq.c +1 -0
@@ -32,6 +32,7 @@#include <linux/kallsyms.h> #include <linux/proc_fs.h> #include <linux/export.h> +#include <linux/vmalloc.h> #include <asm/hardware/cache-l2x0.h> #include <asm/hardware/cache-uniphier.h>
arch/arm/kernel/traps.c +1 -0
@@ -26,6 +26,7 @@#include <linux/sched/debug.h> #include <linux/sched/task_stack.h> #include <linux/irq.h> +#include <linux/vmalloc.h> #include <linux/atomic.h> #include <asm/cacheflush.h>
arch/arm64/kernel/efi.c +1 -0
@@ -10,6 +10,7 @@#include <linux/efi.h> #include <linux/init.h> #include <linux/screen_info.h> +#include <linux/vmalloc.h> #include <asm/efi.h> #include <asm/stacktrace.h>
arch/loongarch/include/asm/kfence.h +1 -0
@@ -10,6 +10,7 @@#define _ASM_LOONGARCH_KFENCE_H#include <linux/kfence.h> +#include <linux/vmalloc.h> #include <asm/pgtable.h> #include <asm/tlb.h>
arch/powerpc/kernel/iommu.c +1 -0
@@ -26,6 +26,7 @@#include <linux/iommu.h> #include <linux/sched.h> #include <linux/debugfs.h> +#include <linux/vmalloc.h> #include <asm/io.h> #include <asm/iommu.h> #include <asm/pci-bridge.h>
arch/powerpc/mm/mem.c +1 -0
@@ -16,6 +16,7 @@#include <linux/highmem.h> #include <linux/suspend.h> #include <linux/dma-direct.h> +#include <linux/vmalloc.h> #include <asm/swiotlb.h> #include <asm/machdep.h>
arch/riscv/kernel/elf_kexec.c +1 -0
@@ -19,6 +19,7 @@#include <linux/libfdt.h> #include <linux/types.h> #include <linux/memblock.h> +#include <linux/vmalloc.h> #include <asm/setup.h> int arch_kimage_file_post_load_cleanup (struct kimage *image)
arch/riscv/kernel/probes/kprobes.c +1 -0
@@ -6,6 +6,7 @@#include <linux/extable.h> #include <linux/slab.h> #include <linux/stop_machine.h> +#include <linux/vmalloc.h> #include <asm/ptrace.h> #include <linux/uaccess.h> #include <asm/sections.h>
arch/s390/kernel/cert_store.c +1 -0
@@ -21,6 +21,7 @@#include <linux/seq_file.h> #include <linux/slab.h> #include <linux/sysfs.h> +#include <linux/vmalloc.h> #include <crypto/sha2.h> #include <keys/user-type.h> #include <asm/debug.h>
arch/s390/kernel/ipl.c +1 -0
@@ -20,6 +20,7 @@#include <linux/gfp.h> #include <linux/crash_dump.h> #include <linux/debug_locks.h> +#include <linux/vmalloc.h> #include <asm/asm-extable.h> #include <asm/diag.h> #include <asm/ipl.h>
arch/x86/include/asm/io.h +1 -0
@@ -42,6 +42,7 @@#include <asm/early_ioremap.h> #include <asm/pgtable_types.h> #include <asm/shared/io.h> +#include <asm/special_insns.h> #define build_mmio_read (name, size, type, reg, barrier) \ static inline type name(const volatile void __iomem *addr) \
arch/x86/kernel/cpu/sgx/main.c +1 -0
@@ -13,6 +13,7 @@#include <linux/sched/signal.h> #include <linux/slab.h> #include <linux/sysfs.h> +#include <linux/vmalloc.h> #include <asm/sgx.h> #include "driver.h" #include "encl.h"
arch/x86/kernel/irq_64.c +1 -0
@@ -18,6 +18,7 @@#include <linux/uaccess.h> #include <linux/smp.h> #include <linux/sched/task_stack.h> +#include <linux/vmalloc.h> #include <asm/cpu_entry_area.h> #include <asm/softirq_stack.h>
arch/x86/mm/fault.c +1 -0
@@ -20,6 +20,7 @@#include <linux/efi.h> /* efi_crash_gracefully_on_page_fault()*/ #include <linux/mm_types.h> #include <linux/mm.h> /* find_and_lock_vma() */ +#include <linux/vmalloc.h> #include <asm/cpufeature.h> /* boot_cpu_has, ... */ #include <asm/traps.h> /* dotraplinkage, ... */
drivers/accel/ivpu/ivpu_mmu_context.c +1 -0
@@ -6,6 +6,7 @@#include <linux/bitfield.h> #include <linux/highmem.h> #include <linux/set_memory.h> +#include <linux/vmalloc.h> #include <drm/drm_cache.h>
drivers/gpu/drm/gma500/mmu.c +1 -0
@@ -5,6 +5,7 @@**************************************************************************/ #include <linux/highmem.h> +#include <linux/vmalloc.h> #include "mmu.h" #include "psb_drv.h"
drivers/gpu/drm/i915/gem/i915_gem_pages.c +1 -0
@@ -5,6 +5,7 @@*/ #include <drm/drm_cache.h> +#include <linux/vmalloc.h> #include "gt/intel_gt.h" #include "gt/intel_tlb.h"
drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c +1 -0
@@ -4,6 +4,7 @@* Copyright © 2016 Intel Corporation */ +#include <linux/vmalloc.h> #include "mock_dmabuf.h" static struct sg_table *mock_map_dma_buf(struct dma_buf_attachment *attachment,
drivers/gpu/drm/i915/gt/shmem_utils.c +1 -0
@@ -7,6 +7,7 @@#include <linux/mm.h> #include <linux/pagemap.h> #include <linux/shmem_fs.h> +#include <linux/vmalloc.h> #include "i915_drv.h" #include "gem/i915_gem_object.h"
drivers/gpu/drm/i915/gvt/firmware.c +1 -0
@@ -30,6 +30,7 @@#include <linux/firmware.h> #include <linux/crc32.h> +#include <linux/vmalloc.h> #include "i915_drv.h" #include "gvt.h"
drivers/gpu/drm/i915/gvt/gtt.c +1 -0
@@ -39,6 +39,7 @@#include "trace.h" #include "gt/intel_gt_regs.h" +#include <linux/vmalloc.h> #if defined(VERBOSE_DEBUG ) #define gvt_vdbg_mm(fmt, args...) gvt_dbg_mm(fmt, ##args)
drivers/gpu/drm/i915/gvt/handlers.c +1 -0
@@ -52,6 +52,7 @@#include "display/skl_watermark_regs.h" #include "display/vlv_dsi_pll_regs.h" #include "gt/intel_gt_regs.h" +#include <linux/vmalloc.h> /* XXX FIXME i915 has changed PP_XXX definition */ #define PCH_PP_STATUS _MMIO(0xc7200)
drivers/gpu/drm/i915/gvt/mmio.c +1 -0
@@ -33,6 +33,7 @@* */ +#include <linux/vmalloc.h> #include "i915_drv.h" #include "i915_reg.h" #include "gvt.h"
drivers/gpu/drm/i915/gvt/vgpu.c +1 -0
@@ -34,6 +34,7 @@#include "i915_drv.h" #include "gvt.h" #include "i915_pvinfo.h" +#include <linux/vmalloc.h> void populate_pvinfo_page (struct intel_vgpu *vgpu) {
drivers/gpu/drm/i915/intel_gvt.c +1 -0
@@ -28,6 +28,7 @@#include "gt/intel_context.h" #include "gt/intel_ring.h" #include "gt/shmem_utils.h" +#include <linux/vmalloc.h> /** * DOC: Intel GVT-g host support
drivers/gpu/drm/imagination/pvr_vm_mips.c +1 -0
@@ -14,6 +14,7 @@#include <linux/err.h> #include <linux/slab.h> #include <linux/types.h> +#include <linux/vmalloc.h> /** * pvr_vm_mips_init() - Initialise MIPS FW pagetable
drivers/gpu/drm/mediatek/mtk_drm_gem.c +1 -0
@@ -4,6 +4,7 @@*/ #include <linux/dma-buf.h> +#include <linux/vmalloc.h> #include <drm/drm.h> #include <drm/drm_device.h>
drivers/gpu/drm/omapdrm/omap_gem.c +1 -0
@@ -9,6 +9,7 @@#include <linux/shmem_fs.h> #include <linux/spinlock.h> #include <linux/pfn_t.h> +#include <linux/vmalloc.h> #include <drm/drm_prime.h> #include <drm/drm_vma_manager.h>
drivers/gpu/drm/v3d/v3d_bo.c +1 -0
@@ -21,6 +21,7 @@#include <linux/dma-buf.h> #include <linux/pfn_t.h> +#include <linux/vmalloc.h> #include "v3d_drv.h" #include "uapi/drm/v3d_drm.h"
drivers/gpu/drm/vmwgfx/vmwgfx_binding.c +1 -0
@@ -54,6 +54,7 @@#include "vmwgfx_drv.h" #include "vmwgfx_binding.h" #include "device_include/svga3d_reg.h" +#include <linux/vmalloc.h> #define VMW_BINDING_RT_BIT 0#define VMW_BINDING_PS_BIT 1
drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c +1 -0
@@ -31,6 +31,7 @@#include <drm/ttm/ttm_placement.h> #include <linux/sched/signal.h> +#include <linux/vmalloc.h> bool vmw_supports_3d (struct vmw_private *dev_priv) {
drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c +1 -0
@@ -25,6 +25,7 @@* **************************************************************************/ +#include <linux/vmalloc.h> #include "vmwgfx_devcaps.h" #include "vmwgfx_drv.h"
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +1 -0
@@ -53,6 +53,7 @@#include <linux/module.h> #include <linux/pci.h> #include <linux/version.h> +#include <linux/vmalloc.h> #define VMWGFX_DRIVER_DESC "Linux drm driver for VMware graphics devices"
drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c +1 -0
@@ -35,6 +35,7 @@#include <linux/sync_file.h> #include <linux/hashtable.h> +#include <linux/vmalloc.h> /* * Helper macro to get dx_ctx_node if available otherwise print an error
drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c +1 -0
@@ -31,6 +31,7 @@#include <drm/vmwgfx_drm.h> #include <linux/pci.h> +#include <linux/vmalloc.h> int vmw_getparam_ioctl (struct drm_device *dev,void *data,struct drm_file *file_priv)
drivers/gpu/drm/xen/xen_drm_front_gem.c +1 -0
@@ -11,6 +11,7 @@#include <linux/dma-buf.h> #include <linux/scatterlist.h> #include <linux/shmem_fs.h> +#include <linux/vmalloc.h> #include <drm/drm_gem.h> #include <drm/drm_prime.h>
drivers/hwtracing/coresight/coresight-trbe.c +1 -0
@@ -17,6 +17,7 @@#include <asm/barrier.h> #include <asm/cpufeature.h> +#include <linux/vmalloc.h> #include "coresight-self-hosted-trace.h" #include "coresight-trbe.h"
drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c +1 -0
@@ -15,6 +15,7 @@#include <linux/io.h> #include <linux/pci.h> #include <linux/etherdevice.h> +#include <linux/vmalloc.h> #include "octep_config.h" #include "octep_main.h"
drivers/net/ethernet/microsoft/mana/hw_channel.c +1 -0
@@ -3,6 +3,7 @@#include <net/mana/gdma.h> #include <net/mana/hw_channel.h> +#include <linux/vmalloc.h> static int mana_hwc_get_msg_index (struct hw_channel_context *hwc,u16 *msg_id) {
drivers/platform/x86/uv_sysfs.c +1 -0
@@ -11,6 +11,7 @@#include <linux/device.h> #include <linux/slab.h> #include <linux/kobject.h> +#include <linux/vmalloc.h> #include <asm/uv/bios.h> #include <asm/uv/uv.h> #include <asm/uv/uv_hub.h>
drivers/scsi/mpi3mr/mpi3mr_transport.c +2 -0
@@ -7,6 +7,8 @@* */ +#include <linux/vmalloc.h> +#include "mpi3mr.h" /**
drivers/vfio/pci/pds/dirty.c +1 -0
@@ -3,6 +3,7 @@#include <linux/interval_tree.h> #include <linux/vfio.h> +#include <linux/vmalloc.h> #include <linux/pds/pds_common.h> #include <linux/pds/pds_core_if.h>
drivers/virt/acrn/mm.c +1 -0
@@ -12,6 +12,7 @@#include <linux/io.h> #include <linux/mm.h> #include <linux/slab.h> +#include <linux/vmalloc.h> #include "acrn_drv.h"
drivers/virtio/virtio_mem.c +1 -0
@@ -21,6 +21,7 @@#include <linux/bitmap.h> #include <linux/lockdep.h> #include <linux/log2.h> +#include <linux/vmalloc.h> #include <acpi/acpi_numa.h>
include/linux/pds/pds_common.h +2 -0
@@ -4,6 +4,8 @@#ifndef _PDS_COMMON_H_#define _PDS_COMMON_H_ +#include <linux/notifier.h> +#define PDS_CORE_DRV_NAME "pds_core"/* the device's internal addressing uses up to 52 bits */
include/rdma/rdmavt_qp.h +1 -0
@@ -11,6 +11,7 @@#include <rdma/ib_verbs.h> #include <rdma/rdmavt_cq.h> #include <rdma/rvt-abi.h> +#include <linux/vmalloc.h> /* * Atomic bit definitions for r_aflags. */
mm/debug_vm_pgtable.c +1 -0
@@ -30,6 +30,7 @@#include <linux/start_kernel.h> #include <linux/sched/mm.h> #include <linux/io.h> +#include <linux/vmalloc.h> #include <asm/cacheflush.h> #include <asm/pgalloc.h>
sound/pci/hda/cs35l41_hda.c +1 -0
@@ -13,6 +13,7 @@#include <sound/soc.h> #include <linux/pm_runtime.h> #include <linux/spi/spi.h> +#include <linux/vmalloc.h> #include "hda_local.h" #include "hda_auto_parser.h" #include "hda_jack.h"
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 01/36] fix missing vmalloc.h includes
Quoted 9 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
The next patch drops
vmalloc.hfrom a system header in order to fix a circular dependency; this adds it to all the files that were pulling it in implicitly.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Suren Baghdasaryan
maintainerPATCH v4 2/36 asm-generic/io.h: Kill vmalloc.h dependency
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
Needed to avoid a new circular dependency with the memory allocation profiling series.
Naturally, a whole bunch of files needed to include vmalloc.h that were previously getting it implicitly.
--- include/asm-generic/io.h | 1 - 1 file changed, 1 deletion(-)
include/asm-generic/io.h +0 -1
@@ -991,7 +991,6 @@ static inline void iowrite64_rep(volatile void __iomem *addr,#ifdef __KERNEL__ -#include <linux/vmalloc.h> #define __io_virt (x) ((void __force *)(x))/*
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 02/36] asm-generic/io.h: Kill vmalloc.h dependency
Quoted 10 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
Needed to avoid a new circular dependency with the memory allocation profiling series.
Naturally, a whole bunch of files needed to include
vmalloc.hthat were previously getting it implicitly.Signed-off-by Kent Overstreet <[email protected]>
Suren Baghdasaryan
maintainerPATCH v4 3/36 mm/slub: Mark slab_free_freelist_hook() __always_inline
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only.
--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slub.c +1 -1
@@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init)return !kasan_slab_free (s, x, init); } -static inline bool slab_free_freelist_hook (struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s,void **head,void **tail,int *cnt) {
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 03/36] mm/slub: Mark slab_free_freelist_hook() __always_inline
Quoted 18 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only.
Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/slub.c +0
-0
mm/slub.c bool slab_free_hook(struct kmem_cache *s, void *x, bool init) · line 2124
Quoted 5 lines
mm/slub.c +1
-1@@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init)return !kasan_slab_free (s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s,
__fastpath_inline seems to me more appropriate here. It prioritizes memory vs performance.
Quoted 6 lines
mm/slub.c +0
-1void **head, void **tail, int *cnt) { --2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerRe: [PATCH v4 03/36] mm/slub: Mark slab_free_freelist_hook() __always_inline
Quoted 23 lines · Pasha Tatashin
Quoted 21 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only.
Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/slub.c +0
-0@@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init)return !kasan_slab_free (s, x, init); }
Quoted 5 lines
mm/slub.c bool slab_free_hook(struct kmem_cache *s, void *x, bool init) · line 2124
Quoted 2 lines
mm/slub.c +1
-1-static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s,
__fastpath_inlineseems to me more appropriate here. It prioritizes memory vs performance.
Hmm. AFAIKT this function is used only in one place and we do not add any additional users, so I don't think changing to __fastpath_inline here would gain us anything.
Quoted 7 lines
Quoted 7 lines
mm/slub.c +0
-1void **head, void **tail, int *cnt) { --2.44.0.rc0.258.g7320e95886-goog
Vlastimil Babka
Reviewed-byRe: [PATCH v4 03/36] mm/slub: Mark slab_free_freelist_hook() __always_inline
Quoted 31 lines · Suren Baghdasaryan
Quoted 23 lines · Pasha Tatashin
Quoted 21 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only.
Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/slub.c +0
-0@@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init)return !kasan_slab_free (s, x, init); }Quoted 5 lines
mm/slub.c bool slab_free_hook(struct kmem_cache *s, void *x, bool init) · line 2124
Quoted 2 lines
mm/slub.c +1
-1-static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s,
__fastpath_inlineseems to me more appropriate here. It prioritizes memory vs performance.
Quoted 3 lines
Hmm. AFAIKT this function is used only in one place and we do not add any additional users, so I don't think changing to
__fastpath_inlinehere would gain us anything.
It would have been more future-proof and self-documenting. But I don't insist.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
mm/slub.c +0
-1void **head, void **tail, int *cnt) { --2.44.0.rc0.258.g7320e95886-goog
Pasha Tatashin
Reviewed-byRe: [PATCH v4 03/36] mm/slub: Mark slab_free_freelist_hook() __always_inline
Quoted 25 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
Quoted 2 lines · Pasha Tatashin
On Wed, Feb 21, 2024 at 2:41 PM Suren Baghdasaryan <[email protected]>
Quoted 13 lines · wrote:
Quoted 13 lines
Quoted 13 lines
From: Kent Overstreet <[email protected]>
It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only.
Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 5 lines
Quoted 5 lines
mm/slub.c bool slab_free_hook(struct kmem_cache *s, void
Quoted 5 lines
mm/slub.c +0
-0@@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void*x, bool init)
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
mm/slub.c bool slab_free_hook(struct kmem_cache *s, void *x, bool init)
Quoted 5 lines
mm/slub.c +1
-1return !kasan_slab_free (s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct
kmem_cache*s,Quoted 7 lines
Quoted 3 lines
__fastpath_inlineseems to me more appropriate here. It prioritizes memory vs performance.Hmm. AFAIKT this function is used only in one place and we do not add any additional users, so I don't think changing to
__fastpath_inlinehere would gain us anything.
For consistency __fastpath_inline makes more sense, but I am ok with or without this change.
Quoted 14 lines
It would have been more future-proof and self-documenting. But I don't insist.
Reviewed-by Vlastimil Babka <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
mm/slub.c +0
-1void **head, void **tail, int *cnt) { --2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerRe: [PATCH v4 03/36] mm/slub: Mark slab_free_freelist_hook() __always_inline
Quoted 37 lines · Pasha Tatashin
Quoted 33 lines · Vlastimil Babka
Quoted 31 lines · Suren Baghdasaryan
Quoted 28 lines · Pasha Tatashin
Quoted 18 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only.
Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/slub.c +0
-0mm/slub.c bool slab_free_hook(struct kmem_cache *s, void *x, bool init) · line 2124
Quoted 5 lines
mm/slub.c +1
-1@@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init)return !kasan_slab_free (s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s,
__fastpath_inlineseems to me more appropriate here. It prioritizes memory vs performance.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Hmm. AFAIKT this function is used only in one place and we do not add any additional users, so I don't think changing to
__fastpath_inlinehere would gain us anything.For consistency
__fastpath_inlinemakes more sense, but I am ok with or without this change.
Ok, I'll update in the next revision. Thanks!
Quoted 16 lines
Reviewed-by Pasha Tatashin <[email protected]>Quoted 13 lines
It would have been more future-proof and self-documenting. But I don't insist.
Reviewed-by Vlastimil Babka <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
mm/slub.c +0
-1void **head, void **tail, int *cnt) { --2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 4/36 scripts/kallysms: Always include __start and __stop symbols
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup.
--- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
scripts/kallsyms.c +13 -0
@@ -204,6 +204,11 @@ static int symbol_in_range(const struct sym_entry *s,return 0 ; } +static bool string_starts_with (const char *s,const char *prefix) +{ +return strncmp (s, prefix,strlen (prefix)) ==0 ; +} +static int symbol_valid (const struct sym_entry *s) {const char *name =sym_name (s);
@@ -211,6 +216,14 @@ static int symbol_valid(const struct sym_entry *s)/* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { +/* +* Symbols starting with __start and __stop are used to denote +* section boundaries, and should always be included: +*/ +if (string_starts_with (name,"__start_" ) || +string_starts_with (name,"__stop_" )) +return 1 ; +if (symbol_in_range (s, text_ranges,ARRAY_SIZE (text_ranges)) ==0 )return 0 ;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 04/36] scripts/kallysms: Always include __start and __stop symbols
Quoted 10 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup.
Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Suren Baghdasaryan
maintainerPATCH v4 5/36 fs: Convert alloc_inode_sb() to a macro
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful.
--- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
include/linux/fs.h +1 -5
@@ -3010,11 +3010,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap,* This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb (struct super_block *sb,struct kmem_cache *cache,gfp_t gfp) -{ -return kmem_cache_alloc_lru (cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb (_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp)extern void __insert_inode_hash (struct inode *,unsigned long hashval);static inline void insert_inode_hash (struct inode *inode)
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 05/36] fs: Convert alloc_inode_sb() to a macro
Quoted 11 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
We're introducing alloc tagging, which tracks memory allocations by callsite. Converting
alloc_inode_sb()to a macro means allocations will be tracked by its caller, which is a bit more useful.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Cc Alexander Viro <[email protected]>Reviewed-by Kees Cook <[email protected]>
Re: [PATCH v4 05/36] fs: Convert alloc_inode_sb() to a macro
In reply to Suren BaghdasaryanQuoted 19 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
We're introducing alloc tagging, which tracks memory allocations by callsite. Converting
alloc_inode_sb()to a macro means allocations will be tracked by its caller, which is a bit more useful.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Cc Alexander Viro <[email protected]>Reviewed-by Kees Cook <[email protected]>--- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)include/linux/fs.h +0
-0
include/linux/fs.h int setattr_should_drop_sgid(struct mnt_idmap *idmap, · line 3014
Quoted 5 lines
include/linux/fs.h +0
-2@@ -3010,11 +3010,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap,* This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb (struct super_block *sb,struct kmem_cache *cache,gfp_t gfp)
A __always_inline wouldn't have the same effect? Just wondering.
Quoted 7 lines
include/linux/fs.h +1
-3-{ -return kmem_cache_alloc_lru (cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb (_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp)extern void __insert_inode_hash (struct inode *,unsigned long hashval);static inline void insert_inode_hash (struct inode *inode)
Suren Baghdasaryan
maintainerRe: [PATCH v4 05/36] fs: Convert alloc_inode_sb() to a macro
Quoted 16 lines · Vlastimil Babka
Quoted 14 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
We're introducing alloc tagging, which tracks memory allocations by callsite. Converting
alloc_inode_sb()to a macro means allocations will be tracked by its caller, which is a bit more useful.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Cc Alexander Viro <[email protected]>Reviewed-by Kees Cook <[email protected]>--- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
Quoted 12 lines
include/linux/fs.h int setattr_should_drop_sgid(struct mnt_idmap *idmap, · line 3014
Quoted 10 lines
include/linux/fs.h +0
-2@@ -3010,11 +3010,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap,* This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb (struct super_block *sb,struct kmem_cache *cache,gfp_t gfp)A
__always_inlinewouldn't have the same effect? Just wondering.
I think inlining it would still keep __LINE__ and __FILE__ pointing to this location in the header instead of the location where the call happens. If we change alloc_inode_sb() to inline we will have to wrap it with alloc_hook() and call kmem_cache_alloc_lru_noprof() inside it. Doable but this change seems much simpler.
Quoted 12 lines
include/linux/fs.h int setattr_should_drop_sgid(struct mnt_idmap *idmap, · line 3016
Quoted 8 lines
include/linux/fs.h +1
-3-{ -return kmem_cache_alloc_lru (cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb (_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp)extern void __insert_inode_hash (struct inode *,unsigned long hashval);static inline void insert_inode_hash (struct inode *inode)--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Kent Overstreet
maintainerRe: [PATCH v4 05/36] fs: Convert alloc_inode_sb() to a macro
In reply to Vlastimil BabkaQuoted 15 lines · Vlastimil Babka
Quoted 14 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
We're introducing alloc tagging, which tracks memory allocations by callsite. Converting
alloc_inode_sb()to a macro means allocations will be tracked by its caller, which is a bit more useful.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Cc Alexander Viro <[email protected]>Reviewed-by Kees Cook <[email protected]>--- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
Quoted 12 lines
include/linux/fs.h int setattr_should_drop_sgid(struct mnt_idmap *idmap, · line 3014
Quoted 10 lines
include/linux/fs.h +0
-2@@ -3010,11 +3010,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap,* This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb (struct super_block *sb,struct kmem_cache *cache,gfp_t gfp)A
__always_inlinewouldn't have the same effect? Just wondering.
nope, macro expansion within an inline happens once, and will show __func__ and __line__ of the helper, we want it expanded in the caller
Suren Baghdasaryan
maintainerPATCH v4 6/36 mm: enumerate all gfp flags
In reply to Suren BaghdasaryanIntroduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies __GFP_BITS_SHIFT calculation.
--- include/linux/gfp_types.h | 90 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 28 deletions(-)
include/linux/gfp_types.h +62 -28
@@ -21,44 +21,78 @@ typedef unsigned int __bitwise gfp_t;* include/trace/events/mmflags.h and tools/perf/builtin-kmem.c */ +enum { + ___GFP_DMA_BIT, + ___GFP_HIGHMEM_BIT, + ___GFP_DMA32_BIT, + ___GFP_MOVABLE_BIT, + ___GFP_RECLAIMABLE_BIT, + ___GFP_HIGH_BIT, + ___GFP_IO_BIT, + ___GFP_FS_BIT, + ___GFP_ZERO_BIT, + ___GFP_UNUSED_BIT,/* 0x200u unused */ + ___GFP_DIRECT_RECLAIM_BIT, + ___GFP_KSWAPD_RECLAIM_BIT, + ___GFP_WRITE_BIT, + ___GFP_NOWARN_BIT, + ___GFP_RETRY_MAYFAIL_BIT, + ___GFP_NOFAIL_BIT, + ___GFP_NORETRY_BIT, + ___GFP_MEMALLOC_BIT, + ___GFP_COMP_BIT, + ___GFP_NOMEMALLOC_BIT, + ___GFP_HARDWALL_BIT, + ___GFP_THISNODE_BIT, + ___GFP_ACCOUNT_BIT, + ___GFP_ZEROTAGS_BIT, +#ifdef CONFIG_KASAN_HW_TAGS + ___GFP_SKIP_ZERO_BIT, + ___GFP_SKIP_KASAN_BIT, +#endif +#ifdef CONFIG_LOCKDEP + ___GFP_NOLOCKDEP_BIT, +#endif + ___GFP_LAST_BIT +}; +/* Plain integer GFP bitmasks. Do not use this directly. */ -#define ___GFP_DMA 0x01u -#define ___GFP_HIGHMEM 0x02u -#define ___GFP_DMA32 0x04u -#define ___GFP_MOVABLE 0x08u -#define ___GFP_RECLAIMABLE 0x10u -#define ___GFP_HIGH 0x20u -#define ___GFP_IO 0x40u -#define ___GFP_FS 0x80u -#define ___GFP_ZERO 0x100u +#define ___GFP_DMA BIT(___GFP_DMA_BIT) +#define ___GFP_HIGHMEM BIT(___GFP_HIGHMEM_BIT) +#define ___GFP_DMA32 BIT(___GFP_DMA32_BIT) +#define ___GFP_MOVABLE BIT(___GFP_MOVABLE_BIT) +#define ___GFP_RECLAIMABLE BIT(___GFP_RECLAIMABLE_BIT) +#define ___GFP_HIGH BIT(___GFP_HIGH_BIT) +#define ___GFP_IO BIT(___GFP_IO_BIT) +#define ___GFP_FS BIT(___GFP_FS_BIT) +#define ___GFP_ZERO BIT(___GFP_ZERO_BIT)/* 0x200u unused */ -#define ___GFP_DIRECT_RECLAIM 0x400u -#define ___GFP_KSWAPD_RECLAIM 0x800u -#define ___GFP_WRITE 0x1000u -#define ___GFP_NOWARN 0x2000u -#define ___GFP_RETRY_MAYFAIL 0x4000u -#define ___GFP_NOFAIL 0x8000u -#define ___GFP_NORETRY 0x10000u -#define ___GFP_MEMALLOC 0x20000u -#define ___GFP_COMP 0x40000u -#define ___GFP_NOMEMALLOC 0x80000u -#define ___GFP_HARDWALL 0x100000u -#define ___GFP_THISNODE 0x200000u -#define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT) +#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT) +#define ___GFP_WRITE BIT(___GFP_WRITE_BIT) +#define ___GFP_NOWARN BIT(___GFP_NOWARN_BIT) +#define ___GFP_RETRY_MAYFAIL BIT(___GFP_RETRY_MAYFAIL_BIT) +#define ___GFP_NOFAIL BIT(___GFP_NOFAIL_BIT) +#define ___GFP_NORETRY BIT(___GFP_NORETRY_BIT) +#define ___GFP_MEMALLOC BIT(___GFP_MEMALLOC_BIT) +#define ___GFP_COMP BIT(___GFP_COMP_BIT) +#define ___GFP_NOMEMALLOC BIT(___GFP_NOMEMALLOC_BIT) +#define ___GFP_HARDWALL BIT(___GFP_HARDWALL_BIT) +#define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) +#define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) +#define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT)#ifdef CONFIG_KASAN_HW_TAGS -#define ___GFP_SKIP_ZERO 0x1000000u -#define ___GFP_SKIP_KASAN 0x2000000u +#define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) +#define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT)#else #define ___GFP_SKIP_ZERO 0#define ___GFP_SKIP_KASAN 0#endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x4000000u +#define ___GFP_NOLOCKDEP BIT(___GFP_NOLOCKDEP_BIT)#else #define ___GFP_NOLOCKDEP 0#endif -/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -249,7 +283,7 @@ typedef unsigned int __bitwise gfp_t;#define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)/* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT ___GFP_LAST_BIT#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))/**
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 06/36] mm: enumerate all gfp flags
Quoted 8 lines · Suren Baghdasaryan
Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies
__GFP_BITS_SHIFTcalculation.Suggested-by Petr Tesařík <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Re: [PATCH v4 06/36] mm: enumerate all gfp flags
In reply to Suren BaghdasaryanQuoted 7 lines · Suren Baghdasaryan
Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies
__GFP_BITS_SHIFTcalculation.Suggested-by Petr Tesařík <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
I thought I have responded to this patch but obviously not the case. I like this change. Makes sense even without the rest of the series.
It seems that KASAN flags already __GFP_BITS_SHIFT which just proves how fragile this existing scheme is.
Quoted 125 lines
--- include/linux/gfp_types.h | 90 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 28 deletions(-)include/linux/gfp_types.h +62
-28@@ -21,44 +21,78 @@ typedef unsigned int __bitwise gfp_t;* include/trace/events/mmflags.h and tools/perf/builtin-kmem.c */ +enum { + ___GFP_DMA_BIT, + ___GFP_HIGHMEM_BIT, + ___GFP_DMA32_BIT, + ___GFP_MOVABLE_BIT, + ___GFP_RECLAIMABLE_BIT, + ___GFP_HIGH_BIT, + ___GFP_IO_BIT, + ___GFP_FS_BIT, + ___GFP_ZERO_BIT, + ___GFP_UNUSED_BIT,/* 0x200u unused */ + ___GFP_DIRECT_RECLAIM_BIT, + ___GFP_KSWAPD_RECLAIM_BIT, + ___GFP_WRITE_BIT, + ___GFP_NOWARN_BIT, + ___GFP_RETRY_MAYFAIL_BIT, + ___GFP_NOFAIL_BIT, + ___GFP_NORETRY_BIT, + ___GFP_MEMALLOC_BIT, + ___GFP_COMP_BIT, + ___GFP_NOMEMALLOC_BIT, + ___GFP_HARDWALL_BIT, + ___GFP_THISNODE_BIT, + ___GFP_ACCOUNT_BIT, + ___GFP_ZEROTAGS_BIT, +#ifdef CONFIG_KASAN_HW_TAGS + ___GFP_SKIP_ZERO_BIT, + ___GFP_SKIP_KASAN_BIT, +#endif +#ifdef CONFIG_LOCKDEP + ___GFP_NOLOCKDEP_BIT, +#endif + ___GFP_LAST_BIT +}; +/* Plain integer GFP bitmasks. Do not use this directly. */ -#define ___GFP_DMA 0x01u -#define ___GFP_HIGHMEM 0x02u -#define ___GFP_DMA32 0x04u -#define ___GFP_MOVABLE 0x08u -#define ___GFP_RECLAIMABLE 0x10u -#define ___GFP_HIGH 0x20u -#define ___GFP_IO 0x40u -#define ___GFP_FS 0x80u -#define ___GFP_ZERO 0x100u +#define ___GFP_DMA BIT(___GFP_DMA_BIT) +#define ___GFP_HIGHMEM BIT(___GFP_HIGHMEM_BIT) +#define ___GFP_DMA32 BIT(___GFP_DMA32_BIT) +#define ___GFP_MOVABLE BIT(___GFP_MOVABLE_BIT) +#define ___GFP_RECLAIMABLE BIT(___GFP_RECLAIMABLE_BIT) +#define ___GFP_HIGH BIT(___GFP_HIGH_BIT) +#define ___GFP_IO BIT(___GFP_IO_BIT) +#define ___GFP_FS BIT(___GFP_FS_BIT) +#define ___GFP_ZERO BIT(___GFP_ZERO_BIT)/* 0x200u unused */ -#define ___GFP_DIRECT_RECLAIM 0x400u -#define ___GFP_KSWAPD_RECLAIM 0x800u -#define ___GFP_WRITE 0x1000u -#define ___GFP_NOWARN 0x2000u -#define ___GFP_RETRY_MAYFAIL 0x4000u -#define ___GFP_NOFAIL 0x8000u -#define ___GFP_NORETRY 0x10000u -#define ___GFP_MEMALLOC 0x20000u -#define ___GFP_COMP 0x40000u -#define ___GFP_NOMEMALLOC 0x80000u -#define ___GFP_HARDWALL 0x100000u -#define ___GFP_THISNODE 0x200000u -#define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT) +#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT) +#define ___GFP_WRITE BIT(___GFP_WRITE_BIT) +#define ___GFP_NOWARN BIT(___GFP_NOWARN_BIT) +#define ___GFP_RETRY_MAYFAIL BIT(___GFP_RETRY_MAYFAIL_BIT) +#define ___GFP_NOFAIL BIT(___GFP_NOFAIL_BIT) +#define ___GFP_NORETRY BIT(___GFP_NORETRY_BIT) +#define ___GFP_MEMALLOC BIT(___GFP_MEMALLOC_BIT) +#define ___GFP_COMP BIT(___GFP_COMP_BIT) +#define ___GFP_NOMEMALLOC BIT(___GFP_NOMEMALLOC_BIT) +#define ___GFP_HARDWALL BIT(___GFP_HARDWALL_BIT) +#define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) +#define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) +#define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT)#ifdef CONFIG_KASAN_HW_TAGS -#define ___GFP_SKIP_ZERO 0x1000000u -#define ___GFP_SKIP_KASAN 0x2000000u +#define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) +#define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT)#else #define ___GFP_SKIP_ZERO 0#define ___GFP_SKIP_KASAN 0#endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x4000000u +#define ___GFP_NOLOCKDEP BIT(___GFP_NOLOCKDEP_BIT)#else #define ___GFP_NOLOCKDEP 0#endif -/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -249,7 +283,7 @@ typedef unsigned int __bitwise gfp_t;#define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)/* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT ___GFP_LAST_BIT#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))/** Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v4 06/36] mm: enumerate all gfp flags
Quoted 5 lines · On Thu, 22 Feb 2024 13:12:29 +0100 Michal Hocko <[email protected]> wrote:
Quoted 4 lines · Suren Baghdasaryan
Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies
__GFP_BITS_SHIFTcalculation.
Quoted 7 lines
Quoted 3 lines
Suggested-by Petr Tesařík <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>I thought I have responded to this patch but obviously not the case. I like this change. Makes sense even without the rest of the series.
Acked-by Michal Hocko <[email protected]>
Thank you, Michal. I also hope it can be merged without waiting for the rest of the series.
Petr T
Suren Baghdasaryan
maintainerRe: [PATCH v4 06/36] mm: enumerate all gfp flags
Quoted 13 lines · Petr Tesařík' via kernel-team
Quoted 9 lines · On Thu, 22 Feb 2024 13:12:29 +0100 Michal Hocko <[email protected]> wrote:
Quoted 7 lines · Suren Baghdasaryan
Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies
__GFP_BITS_SHIFTcalculation.Suggested-by Petr Tesařík <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Quoted 6 lines
Quoted 3 lines
I thought I have responded to this patch but obviously not the case. I like this change. Makes sense even without the rest of the series.
Acked-by Michal Hocko <[email protected]>Thank you, Michal. I also hope it can be merged without waiting for the rest of the series.
Thanks Michal! I can post it separately. With the Ack I don't think it will delay the rest of the series.
Thanks,
Suren.
Quoted 6 lines
Petr T
--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerRe: [PATCH v4 06/36] mm: enumerate all gfp flags
Quoted 20 lines · Suren Baghdasaryan
Quoted 17 lines · Petr Tesařík' via kernel-team
Quoted 12 lines · On Thu, 22 Feb 2024 13:12:29 +0100 Michal Hocko <[email protected]> wrote:
Quoted 7 lines · Suren Baghdasaryan
Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies
__GFP_BITS_SHIFTcalculation.Suggested-by Petr Tesařík <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>I thought I have responded to this patch but obviously not the case. I like this change. Makes sense even without the rest of the series.
Acked-by Michal Hocko <[email protected]>
Quoted 5 lines
Quoted 2 lines
Thank you, Michal. I also hope it can be merged without waiting for the rest of the series.
Thanks Michal! I can post it separately. With the Ack I don't think it will delay the rest of the series.
Stand-alone version is posted as v5 here:
https://lore.kernel.org/all/[email protected]/
Quoted 9 lines
Thanks,
Suren.Quoted 6 lines
Petr T
--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerPATCH v4 7/36 mm: introduce slabobj_ext to support slab object extensions
In reply to Suren BaghdasaryanCurrently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future.
--- include/linux/memcontrol.h | 20 +++++-- include/linux/mm_types.h | 4 +- init/Kconfig | 4 ++ mm/kfence/core.c | 14 ++--- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 +++---------------- mm/page_owner.c | 2 +- mm/slab.h | 62 +++++++++++++-------- mm/slub.c | 109 +++++++++++++++++++++++++++---------- 9 files changed, 156 insertions(+), 119 deletions(-)
include/linux/memcontrol.h +14 -6
@@ -348,8 +348,8 @@ struct mem_cgroup {extern struct mem_cgroup *root_mem_cgroup;enum page_memcg_data_flags { -/* page->memcg_data is a pointer to an objcgs vector */ -MEMCG_DATA_OBJCGS = (1UL <<0 ), +/* page->memcg_data is a pointer to an slabobj_ext vector */ +MEMCG_DATA_OBJEXTS = (1UL <<0 ),/* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL <<1 ),/* the next bit after the last actual flag */
@@ -387,7 +387,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio)unsigned long memcg_data = folio->memcg_data;VM_BUG_ON_FOLIO (folio_test_slab (folio), folio); -VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJCGS , folio); +VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJEXTS , folio);VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_KMEM , folio);return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK );
@@ -408,7 +408,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)unsigned long memcg_data = folio->memcg_data;VM_BUG_ON_FOLIO (folio_test_slab (folio), folio); -VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJCGS , folio); +VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJEXTS , folio);VM_BUG_ON_FOLIO (!(memcg_data &MEMCG_DATA_KMEM ), folio);return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK );
@@ -505,7 +505,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio)*/ unsigned long memcg_data =READ_ONCE (folio->memcg_data); -if (memcg_data &MEMCG_DATA_OBJCGS ) +if (memcg_data &MEMCG_DATA_OBJEXTS )return NULL ;if (memcg_data &MEMCG_DATA_KMEM ) {
@@ -551,7 +551,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *obstatic inline bool folio_memcg_kmem (struct folio *folio) {VM_BUG_ON_PGFLAGS (PageTail (&folio->page), &folio->page); -VM_BUG_ON_FOLIO (folio->memcg_data &MEMCG_DATA_OBJCGS , folio); +VM_BUG_ON_FOLIO (folio->memcg_data &MEMCG_DATA_OBJEXTS , folio);return folio->memcg_data &MEMCG_DATA_KMEM ; }
@@ -1633,6 +1633,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, }#endif /* CONFIG_MEMCG */ +/* +* Extended information for slab objects stored as an array in page->memcg_data +* if MEMCG_DATA_OBJEXTS is set. +*/ +struct slabobj_ext { +struct obj_cgroup *objcg; +}__aligned(8) ; +static inline void __inc_lruvec_kmem_state (void *p,enum node_stat_item idx) {__mod_lruvec_kmem_state (p, idx,1 );
include/linux/mm_types.h +2 -2
@@ -169,7 +169,7 @@ struct page {/* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data;#endif
@@ -306,7 +306,7 @@ struct folio { };atomic_t _mapcount;atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data;#endif #if defined(WANT_PAGE_VIRTUAL )
init/Kconfig +4 -0
@@ -929,6 +929,9 @@ config NUMA_BALANCING_DEFAULT_ENABLED If set, automatic NUMA balancing will be enabled if running on a NUMA machine. +config SLAB_OBJ_EXT + bool + menuconfig CGROUPS bool "Control Group support" select KERNFS
@@ -962,6 +965,7 @@ config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup.
mm/kfence/core.c +7 -7
@@ -595,9 +595,9 @@ static unsigned long kfence_init_pool(void)continue ;__folio_set_slab (slab_folio (slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long )&kfence_metadata_init[i /2 -1 ].objcg | -MEMCG_DATA_OBJCGS ; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long )&kfence_metadata_init[i /2 -1 ].obj_exts | +MEMCG_DATA_OBJEXTS ;#endif }
@@ -645,8 +645,8 @@ static unsigned long kfence_init_pool(void)if (!i || (i %2 ))continue ; -#ifdef CONFIG_MEMCG - slab->memcg_data =0 ; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts =0 ;#endif __folio_clear_slab (slab_folio (slab)); }
@@ -1139,8 +1139,8 @@ void __kfence_free(void *addr) {struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr); -#ifdef CONFIG_MEMCG -KFENCE_WARN_ON (meta->objcg); +#ifdef CONFIG_MEMCG_KMEM +KFENCE_WARN_ON (meta->obj_exts.objcg);#endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
mm/kfence/kfence.h +2 -2
@@ -97,8 +97,8 @@ struct kfence_metadata {struct kfence_track free_track;/* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG -struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM +struct slabobj_ext obj_exts;#endif };
mm/memcontrol.c +7 -49
89 lines not on this page · raw message
mm/page_owner.c +1 -1
@@ -377,7 +377,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,if (!memcg_data) goto out_unlock; -if (memcg_data &MEMCG_DATA_OBJCGS ) +if (memcg_data &MEMCG_DATA_OBJEXTS ) ret +=scnprintf (kbuf + ret, count - ret,"Slab cache page\n" );
mm/slab.h +40 -22
98 lines not on this page · raw message
mm/slub.c +79 -30
194 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 07/36] mm: introduce slabobj_ext to support slab object extensions
Quoted 8 lines · Suren Baghdasaryan
Currently slab pages can store only vectors of
obj_cgrouppointers inpage->memcg_data. Introduceslabobj_extstructure to allow more data to be stored for each slab object. Wrapobj_cgroupintoslabobj_extto support current functionality while allowing to extendslabobj_extin the future.Signed-off-by Suren Baghdasaryan <[email protected]>
Re: [PATCH v4 07/36] mm: introduce slabobj_ext to support slab object extensions
In reply to Suren BaghdasaryanQuoted 7 lines · Suren Baghdasaryan
Currently slab pages can store only vectors of
obj_cgrouppointers inpage->memcg_data. Introduceslabobj_extstructure to allow more data to be stored for each slab object. Wrapobj_cgroupintoslabobj_extto support current functionality while allowing to extendslabobj_extin the future.Signed-off-by Suren Baghdasaryan <[email protected]>
Hi, mostly good from slab perspective, just some fixups:
mm/slab.h static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla · line 570
Quoted 11 lines
mm/slab.h +2
-6-int memcg_alloc_slab_cgroups (struct slab *slab,struct kmem_cache *s, -gfp_t gfp,bool new_slab); -void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, -enum node_stat_item idx,int nr); -#else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs (struct slab *slab) +int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s, +gfp_t gfp,bool new_slab);
We could remove this declaration and make the function static in mm/slub.c.
Quoted 6 lines
mm/slab.h +3
-0+#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts (struct slab *slab) {return NULL ; }
mm/slab.h static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla · line 584
Quoted 10 lines
mm/slab.h +3
-3-static inline int memcg_alloc_slab_cgroups (struct slab *slab, -struct kmem_cache *s,gfp_t gfp, -bool new_slab) +static inline int alloc_slab_obj_exts (struct slab *slab, +struct kmem_cache *s,gfp_t gfp, +bool new_slab) {return 0 ; }
Ditto
mm/slab.h static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla · line 590
Quoted 7 lines
mm/slab.h +6
-1-#endif /* CONFIG_MEMCG_KMEM */ + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook (struct kmem_cache *s,gfp_t flags,void *p) +{ +return NULL ; +}
Same here (and the definition and usage even happens in later patch).
Quoted 29 lines
mm/slab.h +6
-0+#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, +enum node_stat_item idx,int nr); +#endif size_t __ksize (const void *objp);mm/slub.c +2
-2@@ -683,10 +683,10 @@ static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slaif (s->flags & __CMPXCHG_DOUBLE) { ret =__update_freelist_fast (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); }else { ret =__update_freelist_slow (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); }if (likely (ret))return true;@@ -710,13 +710,13 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab,if (s->flags & __CMPXCHG_DOUBLE) {
mm/slub.c static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, · line 719
Quoted 10 lines
mm/slub.c +2
-2ret =__update_freelist_fast (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); }else {unsigned long flags;local_irq_save (flags); ret =__update_freelist_slow (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new);
Please no drive-by fixups of whitespace in code you're not actually changing. I thought you agreed in v3?
mm/slub.c static __always_inline void account_slab(struct slab *slab, int order, · line 2367
Quoted 9 lines
mm/slub.c +1
-1static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp,@@ -2314,7 +2364,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) {if (memcg_kmem_online () && (s->flags &SLAB_ACCOUNT )) -memcg_alloc_slab_cgroups (slab, s, gfp, true); +alloc_slab_obj_exts (slab, s, gfp, true);
This is still guarded by the memcg_kmem_online() static key, which is good.
mm/slub.c static __always_inline void account_slab(struct slab *slab, int order, · line 2376
Quoted 10 lines
mm/slub.c +1
-2mod_node_page_state (slab_pgdat (slab),cache_vmstat_idx (s),PAGE_SIZE << order);@@ -2323,8 +2373,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order,struct kmem_cache *s) { -if (memcg_kmem_online ()) -memcg_free_slab_cgroups (slab); +free_slab_obj_exts (slab);
But this no longer is, yet it still could be?
Quoted 3 lines
mm/slub.c +0
-0mod_node_page_state (slab_pgdat (slab),cache_vmstat_idx (s), -(PAGE_SIZE << order));
Suren Baghdasaryan
maintainerRe: [PATCH v4 07/36] mm: introduce slabobj_ext to support slab object extensions
Quoted 12 lines · Vlastimil Babka
Quoted 7 lines · Suren Baghdasaryan
Currently slab pages can store only vectors of
obj_cgrouppointers inpage->memcg_data. Introduceslabobj_extstructure to allow more data to be stored for each slab object. Wrapobj_cgroupintoslabobj_extto support current functionality while allowing to extendslabobj_extin the future.Signed-off-by Suren Baghdasaryan <[email protected]>Hi, mostly good from slab perspective, just some fixups:
Quoted 13 lines
mm/slab.h static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla · line 570
Quoted 11 lines
mm/slab.h +2
-6-int memcg_alloc_slab_cgroups (struct slab *slab,struct kmem_cache *s, -gfp_t gfp,bool new_slab); -void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, -enum node_stat_item idx,int nr); -#else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs (struct slab *slab) +int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s, +gfp_t gfp,bool new_slab);We could remove this declaration and make the function static in
mm/slub.c.
Ack.
Quoted 8 lines
Quoted 8 lines
mm/slab.h +3
-0+#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts (struct slab *slab) {return NULL ; }
Quoted 11 lines
mm/slab.h static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla · line 584
Quoted 9 lines
mm/slab.h +3
-3-static inline int memcg_alloc_slab_cgroups (struct slab *slab, -struct kmem_cache *s,gfp_t gfp, -bool new_slab) +static inline int alloc_slab_obj_exts (struct slab *slab, +struct kmem_cache *s,gfp_t gfp, +bool new_slab) {return 0 ; }Ditto
Ack.
Quoted 10 lines
mm/slab.h static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla · line 590
Quoted 8 lines
mm/slab.h +6
-1-#endif /* CONFIG_MEMCG_KMEM */ + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook (struct kmem_cache *s,gfp_t flags,void *p) +{ +return NULL ; +}Same here (and the definition and usage even happens in later patch).
Ack.
Quoted 36 lines
Quoted 36 lines
mm/slab.h +6
-0+#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, +enum node_stat_item idx,int nr); +#endif size_t __ksize (const void *objp);mm/slub.c +3
-3@@ -683,10 +683,10 @@ static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slaif (s->flags & __CMPXCHG_DOUBLE) { ret =__update_freelist_fast (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); }else { ret =__update_freelist_slow (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); }if (likely (ret))return true;@@ -710,13 +710,13 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab,if (s->flags & __CMPXCHG_DOUBLE) { ret =__update_freelist_fast (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); }else {unsigned long flags;
Quoted 7 lines
mm/slub.c static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, · line 719
Quoted 4 lines
mm/slub.c +1
-1local_irq_save (flags); ret =__update_freelist_slow (slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new);Please no drive-by fixups of whitespace in code you're not actually changing. I thought you agreed in v3?
Sorry, I must have misunderstood your previous comment. I thought you were saying that the alignment I changed to was incorrect. I'll keep them untouched.
Quoted 14 lines
mm/slub.c static __always_inline void account_slab(struct slab *slab, int order, · line 2367
Quoted 10 lines
mm/slub.c +1
-1static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp,@@ -2314,7 +2364,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) {if (memcg_kmem_online () && (s->flags &SLAB_ACCOUNT )) -memcg_alloc_slab_cgroups (slab, s, gfp, true); +alloc_slab_obj_exts (slab, s, gfp, true);This is still guarded by the
memcg_kmem_online()static key, which is good.Quoted 1 line
Quoted 11 lines
mm/slub.c static __always_inline void account_slab(struct slab *slab, int order, · line 2376
Quoted 9 lines
mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order);mm/slub.c +1
-2@@ -2323,8 +2373,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order,struct kmem_cache *s) { -if (memcg_kmem_online ()) -memcg_free_slab_cgroups (slab); +free_slab_obj_exts (slab);But this no longer is, yet it still could be?
Yes, I missed that, it seems. free_slab_obj_exts() would bail out but still checking the static key is more efficient. I'll revive this check.
Thanks for the review!
Suren.
Quoted 5 lines
Quoted 4 lines
mm/slub.c +0
-0mod_node_page_state (slab_pgdat (slab),cache_vmstat_idx (s), -(PAGE_SIZE << order));
Suren Baghdasaryan
maintainerPATCH v4 8/36 mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation
In reply to Suren BaghdasaryanIntroduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab.
--- include/linux/gfp_types.h | 11 +++++++++++ mm/slub.c | 2 ++ 2 files changed, 13 insertions(+)
include/linux/gfp_types.h +11 -0
@@ -52,6 +52,9 @@ enum {#endif #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, +#endif +#ifdef CONFIG_SLAB_OBJ_EXT + ___GFP_NO_OBJ_EXT_BIT, #endif ___GFP_LAST_BIT };
@@ -93,6 +96,11 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -133,12 +141,15 @@ enum {* node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. +* +* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers
mm/slub.c +2 -0
@@ -1899,6 +1899,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,void *vec; gfp &= ~OBJCGS_CLEAR_MASK ; +/* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec =kcalloc_node (objects,sizeof (struct slabobj_ext ), gfp,slab_nid (slab));if (!vec)
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 08/36] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation
Quoted 6 lines · Suren Baghdasaryan
Introduce
__GFP_NO_OBJ_EXTflag in order to prevent recursive allocations when allocatingslabobj_exton a slab.Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Re: [PATCH v4 08/36] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation
In reply to Suren BaghdasaryanQuoted 5 lines · Suren Baghdasaryan
Introduce
__GFP_NO_OBJ_EXTflag in order to prevent recursive allocations when allocatingslabobj_exton a slab.Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Suren Baghdasaryan
maintainerPATCH v4 9/36 mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation
In reply to Suren BaghdasaryanSlab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs.
--- include/linux/slab.h | 6 ++++++ mm/slub.c | 5 +++-- 2 files changed, 9 insertions(+), 2 deletions(-)
include/linux/slab.h +6 -0
@@ -28,6 +28,12 @@*/ /* DEBUG: Perform (expensive) checks on alloc/free */ #define SLAB_CONSISTENCY_CHECKS ((slab_flags_t __force)0x00000100U) +/* Slab created using create_boot_cache */ +#ifdef CONFIG_SLAB_OBJ_EXT +#define SLAB_NO_OBJ_EXT ((slab_flags_t __force)0x00000200U) +#else +#define SLAB_NO_OBJ_EXT 0 +#endif /* DEBUG: Red zone objs in a cache */ #define SLAB_RED_ZONE ((slab_flags_t __force)0x00000400U)/* DEBUG: Poison objects */
mm/slub.c +3 -2
@@ -5697,7 +5697,8 @@ void __init kmem_cache_init(void)node_set (node, slab_nodes);create_boot_cache (kmem_cache_node,"kmem_cache_node" , -sizeof (struct kmem_cache_node ),SLAB_HWCACHE_ALIGN ,0 ,0 ); +sizeof (struct kmem_cache_node ), +SLAB_HWCACHE_ALIGN |SLAB_NO_OBJ_EXT ,0 ,0 );hotplug_memory_notifier (slab_memory_callback,SLAB_CALLBACK_PRI );
@@ -5707,7 +5708,7 @@ void __init kmem_cache_init(void)create_boot_cache (kmem_cache,"kmem_cache" , offsetof(struct kmem_cache , node) + nr_node_ids *sizeof (struct kmem_cache_node *), -SLAB_HWCACHE_ALIGN ,0 ,0 ); +SLAB_HWCACHE_ALIGN |SLAB_NO_OBJ_EXT ,0 ,0 ); kmem_cache =bootstrap (&boot_kmem_cache); kmem_cache_node =bootstrap (&boot_kmem_cache_node);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 09/36] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation
Quoted 10 lines · Suren Baghdasaryan
Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like
kmem_cacheandkmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. IntroduceSLAB_NO_OBJ_EXTslab flag to mark these caches and avoid creating extensions for objects allocated from these slabs.Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Re: [PATCH v4 09/36] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation
In reply to Suren BaghdasaryanQuoted 9 lines · Suren Baghdasaryan
Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like
kmem_cacheandkmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. IntroduceSLAB_NO_OBJ_EXTslab flag to mark these caches and avoid creating extensions for objects allocated from these slabs.Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Suren Baghdasaryan
maintainerPATCH v4 10/36 slab: objext: introduce objext_flags as extension to page_memcg_data_flags
In reply to Suren BaghdasaryanIntroduce objext_flags to store additional objext flags unrelated to memcg.
--- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 4 +--- 2 files changed, 23 insertions(+), 10 deletions(-)
include/linux/memcontrol.h +22 -7
@@ -356,7 +356,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { +/* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem (struct folio *folio);
@@ -390,7 +405,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio)VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJEXTS , folio);VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_KMEM , folio); -return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); }/*
@@ -411,7 +426,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJEXTS , folio);VM_BUG_ON_FOLIO (!(memcg_data &MEMCG_DATA_KMEM ), folio); -return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); }/*
@@ -468,11 +483,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)if (memcg_data &MEMCG_DATA_KMEM ) {struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK );return obj_cgroup_memcg (objcg); } -return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); }/*
@@ -511,11 +526,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio)if (memcg_data &MEMCG_DATA_KMEM ) {struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK );return obj_cgroup_memcg (objcg); } -return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); }static inline struct mem_cgroup *page_memcg_check (struct page *page)
mm/slab.h +1 -3
@@ -560,10 +560,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) slab_page(slab));VM_BUG_ON_PAGE (obj_exts &MEMCG_DATA_KMEM ,slab_page (slab)); -return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK ); -#else -return (struct slabobj_ext *)obj_exts;#endif +return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK ); } int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 10/36] slab: objext: introduce objext_flags as extension to page_memcg_data_flags
Quoted 5 lines · Suren Baghdasaryan
Introduce
objext_flagsto store additional objext flags unrelated to memcg.Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Re: [PATCH v4 10/36] slab: objext: introduce objext_flags as extension to page_memcg_data_flags
In reply to Suren BaghdasaryanQuoted 4 lines · Suren Baghdasaryan
Introduce
objext_flagsto store additional objext flags unrelated to memcg.Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Kees Cook <[email protected]>
Suren Baghdasaryan
maintainerPATCH v4 11/36 lib: code tagging framework
In reply to Suren BaghdasaryanAdd basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags.
--- include/linux/codetag.h | 68 +++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 219 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c
include/linux/codetag.h new file +68 -0
@@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* code tagging framework +*/ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include <linux/types.h> + +struct codetag_iterator ; +struct codetag_type ; +struct codetag_module ; +struct seq_buf ; +struct module ; + +/* +* An instance of this structure is created in a special ELF section at every +* code location being tagged. At runtime, the special section is treated as +* an array of these. +*/ +struct codetag { +unsigned int flags;/* used in later patches */ +unsigned int lineno; +const char *modname; +const char *function; +const char *filename; +}__aligned(8) ; + +union codetag_ref { +struct codetag *ct; +}; + +struct codetag_type_desc { +const char *section; +size_t tag_size; +}; + +struct codetag_iterator { +struct codetag_type *cttype; +struct codetag_module *cmod; +unsigned long mod_id; +struct codetag *ct; +}; + +#ifdef MODULE +#define CT_MODULE_NAME KBUILD_MODNAME +#else +#define CT_MODULE_NAME NULL +#endif + +#define CODE_TAG_INIT { \ + .modname = CT_MODULE_NAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list (struct codetag_type *cttype,bool lock); +struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype); +struct codetag *codetag_next_ct (struct codetag_iterator *iter); + +void codetag_to_text (struct seq_buf *out,struct codetag *ct); + +struct codetag_type * +codetag_register_type (const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */
lib/Kconfig.debug +4 -0
@@ -968,6 +968,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan"
lib/Makefile +1 -0
@@ -235,6 +235,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o
lib/codetag.c new file +219 -0
219 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 12/36 lib: code tagging module support
In reply to Suren BaghdasaryanAdd support for code tagging from dynamically loaded modules.
--- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-)
include/linux/codetag.h +12 -0
@@ -33,6 +33,10 @@ union codetag_ref {struct codetag_type_desc {const char *section;size_t tag_size; +void (*module_load)(struct codetag_type *cttype, +struct codetag_module *cmod); +void (*module_unload)(struct codetag_type *cttype, +struct codetag_module *cmod); };struct codetag_iterator {
@@ -65,4 +69,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct);struct codetag_type *codetag_register_type (const struct codetag_type_desc *desc); +#if defined(CONFIG_CODE_TAGGING ) && defined(CONFIG_MODULES ) +void codetag_load_module (struct module *mod); +void codetag_unload_module (struct module *mod); +#else +static inline void codetag_load_module (struct module *mod) {} +static inline void codetag_unload_module (struct module *mod) {} +#endif +#endif /* _LINUX_CODETAG_H */
kernel/module/main.c +4 -0
@@ -56,6 +56,7 @@#include <linux/dynamic_debug.h> #include <linux/audit.h> #include <linux/cfi.h> +#include <linux/codetag.h> #include <linux/debugfs.h> #include <uapi/linux/module.h> #include "internal.h"
@@ -1242,6 +1243,7 @@ static void free_module(struct module *mod) {trace_module_free (mod); +codetag_unload_module (mod);mod_sysfs_teardown (mod);/*
@@ -2978,6 +2980,8 @@ static int load_module(struct load_info *info, const char __user *uargs,/* Get rid of temporary copy. */ free_copy (info, flags); +codetag_load_module (mod); +/* Done! */ trace_module_load (mod);
lib/codetag.c +56 -2
82 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 13/36 lib: prevent module unloading if memory is not freed
In reply to Suren BaghdasaryanSkip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF.
--- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 23 +++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 26 insertions(+), 14 deletions(-)
include/linux/codetag.h +3 -3
@@ -35,7 +35,7 @@ struct codetag_type_desc {size_t tag_size;void (*module_load)(struct codetag_type *cttype,struct codetag_module *cmod); -void (*module_unload)(struct codetag_type *cttype, +bool (*module_unload)(struct codetag_type *cttype,struct codetag_module *cmod); };
@@ -71,10 +71,10 @@ codetag_register_type(const struct codetag_type_desc *desc);#if defined(CONFIG_CODE_TAGGING ) && defined(CONFIG_MODULES )void codetag_load_module (struct module *mod); -void codetag_unload_module (struct module *mod); +bool codetag_unload_module (struct module *mod);#else static inline void codetag_load_module (struct module *mod) {} -static inline void codetag_unload_module (struct module *mod) {} +static inline bool codetag_unload_module (struct module *mod) {return true; }#endif #endif /* _LINUX_CODETAG_H */
kernel/module/main.c +15 -8
@@ -1211,15 +1211,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type)return module_alloc (size); } -static void module_memory_free (void *ptr,enum mod_mem_type type) +static void module_memory_free (void *ptr,enum mod_mem_type type, +bool unload_codetags) { +if (!unload_codetags &&mod_mem_type_is_core_data (type)) +return ; +if (mod_mem_use_vmalloc (type))vfree (ptr);else module_memfree (ptr); } -static void free_mod_mem (struct module *mod) +static void free_mod_mem (struct module *mod,bool unload_codetags) {for_each_mod_mem_type (type) {struct module_memory *mod_mem = &mod->mem[type];
@@ -1230,20 +1234,23 @@ static void free_mod_mem(struct module *mod)/* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range (mod_mem->base, mod_mem->size);if (mod_mem->size) -module_memory_free (mod_mem->base, type); +module_memory_free (mod_mem->base, type, + unload_codetags); }/* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range (mod->mem[MOD_DATA ].base, mod->mem[MOD_DATA ].size); -module_memory_free (mod->mem[MOD_DATA ].base,MOD_DATA ); +module_memory_free (mod->mem[MOD_DATA ].base,MOD_DATA , unload_codetags); }/* Free a module, remove from lists, etc. */ static void free_module (struct module *mod) { +bool unload_codetags; +trace_module_free (mod); -codetag_unload_module (mod); + unload_codetags =codetag_unload_module (mod);mod_sysfs_teardown (mod);/*
@@ -1285,7 +1292,7 @@ static void free_module(struct module *mod)kfree (mod->args);percpu_modfree (mod); -free_mod_mem (mod); +free_mod_mem (mod, unload_codetags); }void *__symbol_get (const char *symbol)
@@ -2298,7 +2305,7 @@ static int move_module(struct module *mod, struct load_info *info)return 0 ; out_enomem:for (t--; t >=0 ; t--) -module_memory_free (mod->mem[t].base, t); +module_memory_free (mod->mem[t].base, t, true);return ret; }
@@ -2428,7 +2435,7 @@ static void module_deallocate(struct module *mod, struct load_info *info)percpu_modfree (mod);module_arch_freeing_init (mod); -free_mod_mem (mod); +free_mod_mem (mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr,
lib/codetag.c +8 -3
@@ -5,6 +5,7 @@#include <linux/module.h> #include <linux/seq_buf.h> #include <linux/slab.h> +#include <linux/vmalloc.h> struct codetag_type {struct list_head link;
@@ -239,12 +240,13 @@ void codetag_load_module(struct module *mod)mutex_unlock (&codetag_lock); } -void codetag_unload_module (struct module *mod) +bool codetag_unload_module (struct module *mod) {struct codetag_type *cttype; +bool unload_ok = true;if (!mod) -return ; +return true;mutex_lock (&codetag_lock);list_for_each_entry (cttype, &codetag_types, link) {
@@ -261,7 +263,8 @@ void codetag_unload_module(struct module *mod) }if (found) {if (cttype->desc.module_unload) - cttype->desc.module_unload (cttype, cmod); +if (!cttype->desc.module_unload (cttype, cmod)) + unload_ok = false; cttype->count -=range_size (cttype, &cmod->range);idr_remove (&cttype->mod_idr, mod_id);
@@ -270,4 +273,6 @@ void codetag_unload_module(struct module *mod)up_write (&cttype->mod_lock); }mutex_unlock (&codetag_lock); + +return unload_ok; }
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 13/36] lib: prevent module unloading if memory is not freed
Quoted 5 lines · Suren Baghdasaryan
Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF.
Signed-off-by Suren Baghdasaryan <[email protected]>
I know that module unloading was never considered really supported etc. But should we printk something so the admin knows why it didn't unload, and can go check those outstanding allocations?
Suren Baghdasaryan
maintainerRe: [PATCH v4 13/36] lib: prevent module unloading if memory is not freed
Quoted 11 lines · Vlastimil Babka
Quoted 5 lines · Suren Baghdasaryan
Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF.
Signed-off-by Suren Baghdasaryan <[email protected]>I know that module unloading was never considered really supported etc. But should we printk something so the admin knows why it didn't unload, and can go check those outstanding allocations?
Yes, that sounds reasonable. I'll add a pr_warn() in the next version. Thanks!
Quoted 4 lines
--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Luis Chamberlain
maintainerRe: [PATCH v4 13/36] lib: prevent module unloading if memory is not freed
In reply to Vlastimil BabkaQuoted 8 lines · Vlastimil Babka
Quoted 5 lines · Suren Baghdasaryan
Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF.
Signed-off-by Suren Baghdasaryan <[email protected]>I know that module unloading was never considered really supported etc.
If its not supported then we should not have it on modules. Module loading and unloading should just work, otherwise then this should not work with modules and leave them in a zombie state.
Luis
Suren Baghdasaryan
maintainerRe: [PATCH v4 13/36] lib: prevent module unloading if memory is not freed
Quoted 7 lines · Luis Chamberlain
Quoted 5 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Suren Baghdasaryan <[email protected]>I know that module unloading was never considered really supported etc.
If its not supported then we should not have it on modules. Module loading and unloading should just work, otherwise then this should not work with modules and leave them in a zombie state.
I replied on the v5 thread here:
https://lore.kernel.org/all/[email protected]/
. Let's continue the discussion in that thread. Thanks!
Quoted 2 lines
Luis
Kent Overstreet
maintainerRe: [PATCH v4 13/36] lib: prevent module unloading if memory is not freed
In reply to Luis ChamberlainQuoted 6 lines · Luis Chamberlain
Quoted 5 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Suren Baghdasaryan <[email protected]>I know that module unloading was never considered really supported etc.
If its not supported then we should not have it on modules. Module loading and unloading should just work, otherwise then this should not work with modules and leave them in a zombie state.
Not have memory allocation profiling on modules?
Suren Baghdasaryan
maintainerPATCH v4 14/36 lib: add allocation tagging support for memory allocation profiling
In reply to Suren BaghdasaryanIntroduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It registers an "alloc_tags" codetag type with /proc/allocinfo interface to output allocation tag information when the feature is enabled.CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation.
Memory allocation profiling can be enabled or disabled at runtime using /proc/sys/vm/mem_profiling sysctl when CONFIG_MEM_ALLOC_PROFILING_DEBUG=n. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT enables memory allocation profiling by default.
--- Documentation/admin-guide/sysctl/vm.rst | 16 +++ Documentation/filesystems/proc.rst | 29 +++++ include/asm-generic/codetag.lds.h | 14 +++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 144 +++++++++++++++++++++++ include/linux/sched.h | 24 ++++ lib/Kconfig.debug | 25 ++++ lib/Makefile | 2 + lib/alloc_tag.c | 149 ++++++++++++++++++++++++ scripts/module.lds.S | 7 ++ 10 files changed, 413 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c
Documentation/admin-guide/sysctl/vm.rst +16 -0
@@ -43,6 +43,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- mem_profiling (only if CONFIG_MEM_ALLOC_PROFILING=y) - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes
@@ -425,6 +426,21 @@ e.g., up to one or two maps per allocation. The default value is 65530. +mem_profiling +============== + +Enable memory profiling (when CONFIG_MEM_ALLOC_PROFILING=y) + +1: Enable memory profiling. + +0: Disable memory profiling. + +Enabling memory profiling introduces a small performance overhead for all +memory allocations. + +The default value depends on CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. + + memory_failure_early_kill: ==========================
Documentation/filesystems/proc.rst +29 -0
@@ -688,6 +688,7 @@ files are there, and which are missing. ============ =============================================================== File Content ============ =============================================================== + allocinfo Memory allocations profiling information apm Advanced power management info bootconfig Kernel command line obtained from boot config, and, if there were kernel parameters from the
@@ -953,6 +954,34 @@ also be allocatable although a lot of filesystem metadata may have to be reclaimed to achieve this. +allocinfo +~~~~~~~ + +Provides information about memory allocations at all locations in the code +base. Each allocation in the code is identified by its source file, line +number, module (if originates from a loadable module) and the function calling +the allocation. The number of bytes allocated and number of calls at each +location are reported. + +Example output. + +:: + + > sort -rn /proc/allocinfo + 127664128 31168 mm/page_ext.c:270 func:alloc_page_ext + 56373248 4737 mm/slub.c:2259 func:alloc_slab_page + 14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded + 14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio + 9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node + 4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 3940352 962 mm/memory.c:4214 func:alloc_anon_folio + 2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node + ... + + meminfo ~~~~~~~
include/asm-generic/codetag.lds.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES (_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS () \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */
include/asm-generic/vmlinux.lds.h +3 -0
@@ -50,6 +50,8 @@* [__nosave_begin, __nosave_end] for the nosave data */ +#include <asm-generic/codetag.lds.h> +#ifndef LOAD_OFFSET #define LOAD_OFFSET 0#endif
@@ -366,6 +368,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \TRACE_PRINTKS () \
include/linux/alloc_tag.h new file +144 -0
144 lines not on this page · raw message
include/linux/sched.h +24 -0
@@ -770,6 +770,10 @@ struct task_struct {unsigned int flags;unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING +struct alloc_tag *alloc_tag; +#endif +#ifdef CONFIG_SMP int on_cpu;struct __call_single_node wake_entry;
@@ -810,6 +814,7 @@ struct task_struct {struct task_group *sched_task_group;#endif +#ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity.
@@ -2183,4 +2188,23 @@ static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); }extern void sched_set_stop_task (int cpu,struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save (struct alloc_tag *tag) +{ +swap (current->alloc_tag, tag); +return tag; +} + +static inline void alloc_tag_restore (struct alloc_tag *tag,struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +WARN (current->alloc_tag != tag,"current->alloc_tag was changed:\n" ); +#endif + current->alloc_tag = old; +} +#else +#define alloc_tag_save (_tag) NULL +#define alloc_tag_restore (_tag, _old) do {} while (0) +#endif +#endif
lib/Kconfig.debug +25 -0
@@ -972,6 +972,31 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on PROC_FS + depends on !DEBUG_FORCE_WEAK_PER_CPU + select CODE_TAGGING + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance and memory impact. + +config MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + bool "Enable memory allocation profiling by default" + default y + depends on MEM_ALLOC_PROFILING + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + select MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan"
lib/Makefile +2 -0
@@ -236,6 +236,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o
lib/alloc_tag.c new file +149 -0
149 lines not on this page · raw message
scripts/module.lds.S +7 -0
@@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include <asm-generic/codetag.lds.h> + SECTIONS { /DISCARD/ : { *(.discard)
@@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif }
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 8 lines · Suren Baghdasaryan
[...]
include/linux/alloc_tag.h new file +6
-0+struct alloc_tag { +struct codetag ct; +struct alloc_tag_counters __percpu *counters; +}__aligned(8) ; +#define DEFINE_ALLOC_TAG (_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \
include/linux/alloc_tag.h line 60
Quoted 10 lines
include/linux/alloc_tag.h new file +9
-0+ static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; +static inline struct alloc_tag *alloc_tag_save (struct alloc_tag *tag) +{ +swap (current->alloc_tag, tag); +return tag; +}
Future security hardening improvement idea based on this infrastructure:
it should be possible to implement per-allocation-site kmem caches. For example, we could create:
struct alloc_details {
u32 flags;
union {
u32 size; /* not valid after __init completes */
struct kmem_cache *cache;
};
};- add struct alloc_details to struct alloc_tag
- move the tags section into .ro_after_init
- extend alloc_hooks() to populate flags and size:
.flags = __builtin_constant_p(size) ? KMALLOC_ALLOCATE_FIXED : KMALLOC_ALLOCATE_BUCKETS; .size = __builtin_constant_p(size) ? size : SIZE_MAX;
- during kernel start or module init, walk the alloc_tag list and create either a fixed-size kmem_cache or to allocate a full set of kmalloc-buckets, and update the "cache" member.
- adjust kmalloc core routines to use current->alloc_tag->cache instead of using the global buckets.
This would get us fully separated allocations, producing better than type-based levels of granularity, exceeding what we have currently with CONFIG_RANDOM_KMALLOC_CACHES.
Does this look possible, or am I misunderstanding something in the infrastructure being created here?
Signature · 1 line
Kees Cook
Kent Overstreet
maintainerRe: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 44 lines · Kees Cook
Quoted 8 lines · Suren Baghdasaryan
[...]
include/linux/alloc_tag.h new file +6
-0+struct alloc_tag { +struct codetag ct; +struct alloc_tag_counters __percpu *counters; +}__aligned(8) ; +#define DEFINE_ALLOC_TAG (_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \include/linux/alloc_tag.h line 60
Quoted 10 lines
include/linux/alloc_tag.h new file +9
-0+ static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; +static inline struct alloc_tag *alloc_tag_save (struct alloc_tag *tag) +{ +swap (current->alloc_tag, tag); +return tag; +}Future security hardening improvement idea based on this infrastructure:
it should be possible to implement per-allocation-site kmem caches. For example, we could create:struct alloc_details { u32 flags; union { u32 size; /* not valid after __init completes */ struct kmem_cache *cache; }; };- add struct
alloc_detailsto structalloc_tag
- move the tags section into .ro_after_init
- extendalloc_hooks()to populate flags and size:.flags = __builtin_constant_p(size) ? KMALLOC_ALLOCATE_FIXED : KMALLOC_ALLOCATE_BUCKETS; .size = __builtin_constant_p(size) ? size : SIZE_MAX;- during kernel start or module init, walk the
alloc_taglist and create either a fixed-sizekmem_cacheor to allocate a full set of kmalloc-buckets, and update the "cache" member.
- adjust kmalloc core routines to usecurrent->alloc_tag->cacheinstead of using the global buckets.
Quoted 6 lines
This would get us fully separated allocations, producing better than type-based levels of granularity, exceeding what we have currently with
CONFIG_RANDOM_KMALLOC_CACHES.Does this look possible, or am I misunderstanding something in the infrastructure being created here?
Definitely possible, but... would we want this? That would produce a _lot_ of kmem caches, and don't we already try to collapse those where possible to reduce internal fragmentation?
Re: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 45 lines · Kent Overstreet
Quoted 44 lines · Kees Cook
Quoted 8 lines · Suren Baghdasaryan
[...]
include/linux/alloc_tag.h new file +6
-0+struct alloc_tag { +struct codetag ct; +struct alloc_tag_counters __percpu *counters; +}__aligned(8) ; +#define DEFINE_ALLOC_TAG (_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \include/linux/alloc_tag.h line 60
Quoted 10 lines
include/linux/alloc_tag.h new file +9
-0+ static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; +static inline struct alloc_tag *alloc_tag_save (struct alloc_tag *tag) +{ +swap (current->alloc_tag, tag); +return tag; +}Future security hardening improvement idea based on this infrastructure:
it should be possible to implement per-allocation-site kmem caches. For example, we could create:struct alloc_details { u32 flags; union { u32 size; /* not valid after __init completes */ struct kmem_cache *cache; }; };- add struct
alloc_detailsto structalloc_tag
- move the tags section into .ro_after_init
- extendalloc_hooks()to populate flags and size:.flags = __builtin_constant_p(size) ? KMALLOC_ALLOCATE_FIXED : KMALLOC_ALLOCATE_BUCKETS; .size = __builtin_constant_p(size) ? size : SIZE_MAX;- during kernel start or module init, walk the
alloc_taglist and create either a fixed-sizekmem_cacheor to allocate a full set of kmalloc-buckets, and update the "cache" member.
- adjust kmalloc core routines to usecurrent->alloc_tag->cacheinstead of using the global buckets.
Quoted 8 lines
Quoted 6 lines
This would get us fully separated allocations, producing better than type-based levels of granularity, exceeding what we have currently with
CONFIG_RANDOM_KMALLOC_CACHES.Does this look possible, or am I misunderstanding something in the infrastructure being created here?
Definitely possible, but... would we want this?
Yes, very very much. One of the worst and mostly unaddressed weaknesses with the kernel right now is use-after-free based type confusion[0], which depends on merged caches (or cache reuse).
This doesn't solve cross-allocator (kmalloc/page_alloc) type confusion (as terrifyingly demonstrated[1] by Jann Horn), but it does help with what has been a very common case of "use msg_msg to impersonate your target object"[2] exploitation.
Quoted 1 line
That would produce a _lot_ of kmem caches
Fewer than you'd expect, but yes, there is some overhead. However, out-of-tree forks of Linux have successfully experimented with this already and seen good results[3].
Quoted 2 lines
and don't we already try to collapse those where possible to reduce internal fragmentation?
In the past, yes, but the desire for security has tended to have more people building with SLAB_MERGE_DEFAULT=n and/or CONFIG_RANDOM_KMALLOC_CACHES=y (or booting with "slab_nomerge").
Just doing the type safety isn't sufficient without the cross-allocator safety, but we've also had solutions for that proposed[4].
-Kees
[0] https://github.com/KSPP/linux/issues/189
[1] https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html
[2] https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html
https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html#exploring-struct-msg_msg
[3] https://grsecurity.net/how_autoslab_changes_the_memory_unsafety_game
[4] https://lore.kernel.org/linux-hardening/[email protected]/
Signature · 1 line
Kees Cook
Kent Overstreet
maintainerRe: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 55 lines · Kees Cook
Quoted 45 lines · Kent Overstreet
Quoted 44 lines · Kees Cook
Quoted 8 lines · Suren Baghdasaryan
[...]
include/linux/alloc_tag.h new file +6
-0+struct alloc_tag { +struct codetag ct; +struct alloc_tag_counters __percpu *counters; +}__aligned(8) ; +#define DEFINE_ALLOC_TAG (_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \include/linux/alloc_tag.h line 60
Quoted 10 lines
include/linux/alloc_tag.h new file +9
-0+ static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; +static inline struct alloc_tag *alloc_tag_save (struct alloc_tag *tag) +{ +swap (current->alloc_tag, tag); +return tag; +}Future security hardening improvement idea based on this infrastructure:
it should be possible to implement per-allocation-site kmem caches. For example, we could create:struct alloc_details { u32 flags; union { u32 size; /* not valid after __init completes */ struct kmem_cache *cache; }; };- add struct
alloc_detailsto structalloc_tag
- move the tags section into .ro_after_init
- extendalloc_hooks()to populate flags and size:.flags = __builtin_constant_p(size) ? KMALLOC_ALLOCATE_FIXED : KMALLOC_ALLOCATE_BUCKETS; .size = __builtin_constant_p(size) ? size : SIZE_MAX;- during kernel start or module init, walk the
alloc_taglist and create either a fixed-sizekmem_cacheor to allocate a full set of kmalloc-buckets, and update the "cache" member.
- adjust kmalloc core routines to usecurrent->alloc_tag->cacheinstead of using the global buckets.Quoted 8 lines
Quoted 6 lines
This would get us fully separated allocations, producing better than type-based levels of granularity, exceeding what we have currently with
CONFIG_RANDOM_KMALLOC_CACHES.Does this look possible, or am I misunderstanding something in the infrastructure being created here?
Definitely possible, but... would we want this?
Quoted 8 lines
Yes, very very much. One of the worst and mostly unaddressed weaknesses with the kernel right now is use-after-free based type confusion[0], which depends on merged caches (or cache reuse).
This doesn't solve cross-allocator (kmalloc/page_alloc) type confusion (as terrifyingly demonstrated[1] by Jann Horn), but it does help with what has been a very common case of "use
msg_msgto impersonate your target object"[2] exploitation.
We have a ton of code that references PAGE_SIZE and uses the page allocator completely unnecessarily - that's something worth harping about at conferences; if we could motivate people to clean that stuff up it'd have a lot of positive effects.
Quoted 5 lines
Quoted 1 line
That would produce a _lot_ of kmem caches
Fewer than you'd expect, but yes, there is some overhead. However, out-of-tree forks of Linux have successfully experimented with this already and seen good results[3].
So in that case - I don't think there's any need for a separate alloc_details; we'd just add a kmem_cache * to alloc_tag and then hook into the codetag init/unload path to create and destroy the kmem caches.
No need to adjust the slab code either; alloc_hooks() itself could dispatch to kmem_cache_alloc() instead of kmalloc() if this is in use.
Re: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 72 lines · Kent Overstreet
Quoted 55 lines · Kees Cook
Quoted 45 lines · Kent Overstreet
Quoted 44 lines · Kees Cook
Quoted 8 lines · Suren Baghdasaryan
[...]
include/linux/alloc_tag.h new file +6
-0+struct alloc_tag { +struct codetag ct; +struct alloc_tag_counters __percpu *counters; +}__aligned(8) ; +#define DEFINE_ALLOC_TAG (_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \include/linux/alloc_tag.h line 60
Quoted 10 lines
include/linux/alloc_tag.h new file +9
-0+ static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; +static inline struct alloc_tag *alloc_tag_save (struct alloc_tag *tag) +{ +swap (current->alloc_tag, tag); +return tag; +}Future security hardening improvement idea based on this infrastructure:
it should be possible to implement per-allocation-site kmem caches. For example, we could create:struct alloc_details { u32 flags; union { u32 size; /* not valid after __init completes */ struct kmem_cache *cache; }; };- add struct
alloc_detailsto structalloc_tag
- move the tags section into .ro_after_init
- extendalloc_hooks()to populate flags and size:.flags = __builtin_constant_p(size) ? KMALLOC_ALLOCATE_FIXED : KMALLOC_ALLOCATE_BUCKETS; .size = __builtin_constant_p(size) ? size : SIZE_MAX;- during kernel start or module init, walk the
alloc_taglist and create either a fixed-sizekmem_cacheor to allocate a full set of kmalloc-buckets, and update the "cache" member.
- adjust kmalloc core routines to usecurrent->alloc_tag->cacheinstead of using the global buckets.Quoted 8 lines
Quoted 6 lines
This would get us fully separated allocations, producing better than type-based levels of granularity, exceeding what we have currently with
CONFIG_RANDOM_KMALLOC_CACHES.Does this look possible, or am I misunderstanding something in the infrastructure being created here?
Definitely possible, but... would we want this?
Quoted 8 lines
Yes, very very much. One of the worst and mostly unaddressed weaknesses with the kernel right now is use-after-free based type confusion[0], which depends on merged caches (or cache reuse).
This doesn't solve cross-allocator (kmalloc/page_alloc) type confusion (as terrifyingly demonstrated[1] by Jann Horn), but it does help with what has been a very common case of "use
msg_msgto impersonate your target object"[2] exploitation.We have a ton of code that references
PAGE_SIZEand uses the page allocator completely unnecessarily - that's something worth harping about at conferences; if we could motivate people to clean that stuff up it'd have a lot of positive effects.Quoted 2 lines
Quoted 1 line
That would produce a _lot_ of kmem caches
Quoted 7 lines
Quoted 3 lines
Fewer than you'd expect, but yes, there is some overhead. However, out-of-tree forks of Linux have successfully experimented with this already and seen good results[3].
So in that case - I don't think there's any need for a separate
alloc_details; we'd just add akmem_cache* toalloc_tagand then hook into the codetag init/unload path to create and destroy the kmem caches.
Okay, sounds good. There needs to be a place to track "is this a fixed size or a run-time size" choice.
Quoted 2 lines
No need to adjust the slab code either;
alloc_hooks()itself could dispatch tokmem_cache_alloc()instead ofkmalloc()if this is in use.
Right, it'd go to either kmem_cache_alloc() directly, or to a modified kmalloc() that used the passed-in cache is the base for an array of sized buckets, rather than the global (or 16-way global) buckets.
Yay for the future!
Signature · 1 line
Kees Cook
Re: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
In reply to Suren Baghdasaryaninclude/linux/alloc_tag.h line 104
Quoted 11 lines · Suren Baghdasaryan
include/linux/alloc_tag.h new file +10
-0+static inline void alloc_tag_sub (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} + +static inline void alloc_tag_sub_noalloc (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} +
Nit: just notice these are now the same and maybe you could just drop both wrappers and rename __alloc_tag_sub to alloc_tag_sub?
Suren Baghdasaryan
maintainerRe: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 16 lines · Vlastimil Babka
include/linux/alloc_tag.h line 104
Quoted 11 lines · Suren Baghdasaryan
include/linux/alloc_tag.h new file +10
-0+static inline void alloc_tag_sub (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} + +static inline void alloc_tag_sub_noalloc (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} +Nit: just notice these are now the same and maybe you could just drop both wrappers and rename
__alloc_tag_subtoalloc_tag_sub?
Ack.
Quoted 4 lines
--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Re: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
In reply to Suren BaghdasaryanAnother thing I noticed, dunno how critical
include/linux/alloc_tag.h line 85
Quoted 9 lines · Suren Baghdasaryan
include/linux/alloc_tag.h new file +9
-0+static inline void __alloc_tag_sub (union codetag_ref *ref,size_t bytes) +{ +struct alloc_tag *tag; + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +WARN_ONCE (ref && !ref->ct,"alloc_tag was not set\n" ); +#endif +if (!ref || !ref->ct) +return ;
This is quite careful.
Quoted 20 lines
include/linux/alloc_tag.h new file +20
-0+ + tag =ct_to_alloc_tag (ref->ct); + +this_cpu_sub (tag->counters->bytes, bytes); +this_cpu_dec (tag->counters->calls); + + ref->ct =NULL ; +} + +static inline void alloc_tag_sub (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} + +static inline void alloc_tag_sub_noalloc (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} + +static inline void alloc_tag_ref_set (union codetag_ref *ref,struct alloc_tag *tag)
include/linux/alloc_tag.h line 115
Quoted 10 lines
include/linux/alloc_tag.h new file +10
-0+{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +WARN_ONCE (ref && ref->ct, +"alloc_tag was not cleared (got tag for %s:%u)\n" ,\ + ref->ct->filename, ref->ct->lineno); + +WARN_ONCE (!tag,"current->alloc_tag not set" ); +#endif +if (!ref || !tag) +return ;
This too.
Quoted 4 lines
include/linux/alloc_tag.h new file +4
-0+ + ref->ct = &tag->ct; +/* +* We need in increment the call counter every time we have a new
include/linux/alloc_tag.h line 129
Quoted 10 lines
include/linux/alloc_tag.h new file +10
-0+* allocation or when we split a large allocation into smaller ones. +* Each new reference for every sub-allocation needs to increment call +* counter because when we free each part the counter will be decremented. +*/ +this_cpu_inc (tag->counters->calls); +} + +static inline void alloc_tag_add (union codetag_ref *ref,struct alloc_tag *tag,size_t bytes) +{ +alloc_tag_ref_set (ref, tag);
We might have returned from alloc_tag_ref_set() due to !tag
include/linux/alloc_tag.h line 130
Quoted 1 line
include/linux/alloc_tag.h new file +1
-0+this_cpu_add (tag->counters->bytes, bytes);
But here we still assume it's valid.
Quoted 2 lines
(quoted diff) +2
-0+} +
Suren Baghdasaryan
maintainerRe: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling
Quoted 66 lines · Vlastimil Babka
Another thing I noticed, dunno how critical
include/linux/alloc_tag.h line 85
Quoted 9 lines · Suren Baghdasaryan
include/linux/alloc_tag.h new file +9
-0+static inline void __alloc_tag_sub (union codetag_ref *ref,size_t bytes) +{ +struct alloc_tag *tag; + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +WARN_ONCE (ref && !ref->ct,"alloc_tag was not set\n" ); +#endif +if (!ref || !ref->ct) +return ;This is quite careful.
Quoted 20 lines
include/linux/alloc_tag.h new file +20
-0+ + tag =ct_to_alloc_tag (ref->ct); + +this_cpu_sub (tag->counters->bytes, bytes); +this_cpu_dec (tag->counters->calls); + + ref->ct =NULL ; +} + +static inline void alloc_tag_sub (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} + +static inline void alloc_tag_sub_noalloc (union codetag_ref *ref,size_t bytes) +{ +__alloc_tag_sub (ref, bytes); +} + +static inline void alloc_tag_ref_set (union codetag_ref *ref,struct alloc_tag *tag)include/linux/alloc_tag.h line 115
Quoted 10 lines
include/linux/alloc_tag.h new file +10
-0+{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +WARN_ONCE (ref && ref->ct, +"alloc_tag was not cleared (got tag for %s:%u)\n" ,\ + ref->ct->filename, ref->ct->lineno); + +WARN_ONCE (!tag,"current->alloc_tag not set" ); +#endif +if (!ref || !tag) +return ;This too.
Quoted 4 lines
include/linux/alloc_tag.h new file +4
-0+ + ref->ct = &tag->ct; +/* +* We need in increment the call counter every time we have a new include/linux/alloc_tag.h line 129
Quoted 10 lines
include/linux/alloc_tag.h new file +10
-0+* allocation or when we split a large allocation into smaller ones. +* Each new reference for every sub-allocation needs to increment call +* counter because when we free each part the counter will be decremented. +*/ +this_cpu_inc (tag->counters->calls); +} + +static inline void alloc_tag_add (union codetag_ref *ref,struct alloc_tag *tag,size_t bytes) +{ +alloc_tag_ref_set (ref, tag);We might have returned from
alloc_tag_ref_set()due to !tag
Quoted 3 lines
Quoted 1 line
+
this_cpu_add(tag->counters->bytes, bytes);But here we still assume it's valid.
Yes, this is a blunder on my side after splitting alloc_tag_ref_set() into a separate function. I'll fix this in the next version. Thanks!
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+} +--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerPATCH v4 15/36 lib: introduce support for page allocation tagging
In reply to Suren BaghdasaryanIntroduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field.
--- include/linux/page_ext.h | 1 - include/linux/pgalloc_tag.h | 78 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 ++++++++ mm/mm_init.c | 1 + mm/page_alloc.c | 4 ++ mm/page_ext.c | 4 ++ 7 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 include/linux/pgalloc_tag.h
include/linux/page_ext.h +0 -1
@@ -4,7 +4,6 @@#include <linux/types.h> #include <linux/stacktrace.h> -#include <linux/stackdepot.h> struct pglist_data ;
include/linux/pgalloc_tag.h new file +78 -0
78 lines not on this page · raw message
lib/Kconfig.debug +1 -0
@@ -978,6 +978,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track
lib/alloc_tag.c +17 -0
@@ -3,6 +3,7 @@#include <linux/fs.h> #include <linux/gfp.h> #include <linux/module.h> +#include <linux/page_ext.h> #include <linux/proc_fs.h> #include <linux/seq_buf.h> #include <linux/seq_file.h>
@@ -115,6 +116,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype,return module_unused; } +static __init bool need_page_alloc_tagging (void ) +{ +return true; +} + +static __init void init_page_alloc_tagging (void ) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size =sizeof (union codetag_ref ), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL (page_alloc_tagging_ops); +static struct ctl_table memory_allocation_profiling_sysctls[] = { { .procname ="mem_profiling" ,
mm/mm_init.c +1 -0
@@ -24,6 +24,7 @@#include <linux/page_ext.h> #include <linux/pti.h> #include <linux/pgtable.h> +#include <linux/stackdepot.h> #include <linux/swap.h> #include <linux/cma.h> #include <linux/crash_dump.h>
mm/page_alloc.c +4 -0
@@ -53,6 +53,7 @@#include <linux/khugepaged.h> #include <linux/delayacct.h> #include <linux/cacheinfo.h> +#include <linux/pgalloc_tag.h> #include <asm/div64.h> #include "internal.h" #include "shuffle.h"
@@ -1100,6 +1101,7 @@ static __always_inline bool free_pages_prepare(struct page *page,/* Do not let hwpoison pages hit pcplists/buddy */ reset_page_owner (page, order);page_table_check_free (page, order); +pgalloc_tag_sub (page, order);return false; }
@@ -1139,6 +1141,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;reset_page_owner (page, order);page_table_check_free (page, order); +pgalloc_tag_sub (page, order);if (!PageHighMem (page)) { debug_check_no_locks_freed(page_address(page),
@@ -1532,6 +1535,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,set_page_owner (page, order, gfp_flags);page_table_check_alloc (page, order); +pgalloc_tag_add (page, current, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
mm/page_ext.c +4 -0
@@ -10,6 +10,7 @@#include <linux/page_idle.h> #include <linux/page_table_check.h> #include <linux/rcupdate.h> +#include <linux/pgalloc_tag.h> /* * struct page extension
@@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#if defined(CONFIG_PAGE_IDLE_FLAG ) && !defined(CONFIG_64BIT ) &page_idle_ops,#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 15/36] lib: introduce support for page allocation tagging
Quoted 7 lines · Suren Baghdasaryan
Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a
page_extfield.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>
The static key usage seems fine now. Even if the page_ext overhead is still always paid when compiled in, you mention in the cover letter there's a plan for boot-time toggle later, so
Suren Baghdasaryan
maintainerRe: [PATCH v4 15/36] lib: introduce support for page allocation tagging
Quoted 6 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a
page_extfield.
Quoted 7 lines
Quoted 3 lines
Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>The static key usage seems fine now. Even if the
page_extoverhead is still always paid when compiled in, you mention in the cover letter there's a plan for boot-time toggle later, so
Yes, I already have a simple patch for that to be included in the next
revision: https://github.com/torvalds/linux/commit/7ca367e80232345f471b77b3ea71cf82faf50954
Quoted 2 lines
Reviewed-by Vlastimil Babka <[email protected]>
Thanks!
Quoted 2 lines
Re: [PATCH v4 15/36] lib: introduce support for page allocation tagging
Quoted 11 lines · Suren Baghdasaryan
Quoted 10 lines · Vlastimil Babka
Quoted 7 lines · Suren Baghdasaryan
Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a
page_extfield.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>
Quoted 6 lines
Quoted 3 lines
The static key usage seems fine now. Even if the
page_extoverhead is still always paid when compiled in, you mention in the cover letter there's a plan for boot-time toggle later, soYes, I already have a simple patch for that to be included in the next
revision: https://github.com/torvalds/linux/commit/7ca367e80232345f471b77b3ea71cf82faf50954
This opt-out logic would require a distro kernel with allocation profiling compiled-in to ship together with something that modifies kernel command line to disable it by default, so it's not very practical. Could the CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT be turned into having 3 possible choices, where one of them would initialize mem_profiling_enabled to false?
Or, taking a step back, is it going to be a common usecase to pay the memory overhead unconditionally, but only enable the profiling later during runtime? Also what happens if someone would enable and disable it multiple times during one boot? Would the statistics get all skewed because some frees would be not accounted while it's disabled?
Quoted 7 lines
Quoted 2 lines
Reviewed-by Vlastimil Babka <[email protected]>Thanks!
Quoted 2 lines
Kent Overstreet
maintainerRe: [PATCH v4 15/36] lib: introduce support for page allocation tagging
Quoted 28 lines · Vlastimil Babka
Quoted 11 lines · Suren Baghdasaryan
Quoted 10 lines · Vlastimil Babka
Quoted 7 lines · Suren Baghdasaryan
Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a
page_extfield.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>Quoted 6 lines
Quoted 3 lines
The static key usage seems fine now. Even if the
page_extoverhead is still always paid when compiled in, you mention in the cover letter there's a plan for boot-time toggle later, soYes, I already have a simple patch for that to be included in the next
revision: https://github.com/torvalds/linux/commit/7ca367e80232345f471b77b3ea71cf82faf50954This opt-out logic would require a distro kernel with allocation profiling compiled-in to ship together with something that modifies kernel command line to disable it by default, so it's not very practical. Could the
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULTbe turned into having 3 possible choices, where one of them would initializemem_profiling_enabledto false?
Quoted 5 lines
Or, taking a step back, is it going to be a common usecase to pay the memory overhead unconditionally, but only enable the profiling later during runtime? Also what happens if someone would enable and disable it multiple times during one boot? Would the statistics get all skewed because some frees would be not accounted while it's disabled?
I already wrote the code for fast lookup from codetag index -> codetag - i.e. pointer compression - so this is all going away shortly.
It just won't be in the initial pull request because of other dependencies (it requires my eytzinger code, which I was already lifting from fs/bcachefs/ for 6.9), but it can still probably make 6.9 in a second smaller pull.
Suren Baghdasaryan
maintainerRe: [PATCH v4 15/36] lib: introduce support for page allocation tagging
In reply to Vlastimil BabkaQuoted 22 lines · Vlastimil Babka
Quoted 11 lines · Suren Baghdasaryan
Quoted 10 lines · Vlastimil Babka
Quoted 7 lines · Suren Baghdasaryan
Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a
page_extfield.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>Quoted 6 lines
Quoted 3 lines
The static key usage seems fine now. Even if the
page_extoverhead is still always paid when compiled in, you mention in the cover letter there's a plan for boot-time toggle later, soYes, I already have a simple patch for that to be included in the next
revision: https://github.com/torvalds/linux/commit/7ca367e80232345f471b77b3ea71cf82faf50954
Quoted 6 lines
This opt-out logic would require a distro kernel with allocation profiling compiled-in to ship together with something that modifies kernel command line to disable it by default, so it's not very practical. Could the
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULTbe turned into having 3 possible choices, where one of them would initializemem_profiling_enabledto false?
I was thinking about a similar approach of having the early boot parameter to be a tri-state with "0 | 1 | Never". The default option would be "Never" if CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and "1" if CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y. Would that solve the problem for distributions?
Quoted 4 lines
Or, taking a step back, is it going to be a common usecase to pay the memory overhead unconditionally, but only enable the profiling later during runtime?
I think that would be the option one would use in the early deployments, to be able to enable the feature on specific devices without a reboot. Pasha brought up also an option when we disable the feature initially (via early boot option) but can enable it and reboot the system that will come up with enabled option.
As Kent mentioned, he has been working on a pointer compression mechanism to cut the overhead of each codtag reference from one pointer (8 bytes) to 2 bytes index. I'm yet to check the performance but if that works and we can fit this index into page flags, that would completely eliminate dependency on page_ext and this memory overhead will be gone. This mechanism is not mature enough and I don't want to include these optimizations into the initial patchset, that's why it's not included in this patchset.
Quoted 3 lines
Also what happens if someone would enable and disable it multiple times during one boot? Would the statistics get all skewed because some frees would be not accounted while it's disabled?
Yes and this was discussed during last LSFMM when the runtime control was brought up for the first time. That loss of accounting while the feature is disabled seems to be expected and acceptable. One could snapshot the state before re-enabling the feature and then compare later results with the initial snapshot to figure out the allocation growth.
Quoted 12 lines
Quoted 8 lines
Quoted 3 lines
Reviewed-by Vlastimil Babka <[email protected]>Thanks!
Quoted 2 lines
--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerPATCH v4 16/36 mm: percpu: increase PERCPU_MODULE_RESERVE to accommodate allocation tags
In reply to Suren BaghdasaryanAs each allocation tag generates a per-cpu variable, more space is required to store them. Increase PERCPU_MODULE_RESERVE to provide enough area. A better long-term solution would be to allocate this memory dynamically.
--- include/linux/percpu.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/percpu.h +4 -0
@@ -14,7 +14,11 @@/* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES +#ifdef CONFIG_MEM_ALLOC_PROFILING +#define PERCPU_MODULE_RESERVE (8 << 12) +#else #define PERCPU_MODULE_RESERVE (8 << 10) +#endif #else #define PERCPU_MODULE_RESERVE 0#endif
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 17/36 change alloc_pages name in dma_map_ops to avoid name conflicts
In reply to Suren BaghdasaryanAfter redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended.
--- arch/alpha/kernel/pci_iommu.c | 2 +- arch/mips/jazz/jazzdma.c | 2 +- arch/powerpc/kernel/dma-iommu.c | 2 +- arch/powerpc/platforms/ps3/system-bus.c | 4 ++-- arch/powerpc/platforms/pseries/vio.c | 2 +- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/parisc/ccio-dma.c | 2 +- drivers/parisc/sba_iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 13 files changed, 15 insertions(+), 15 deletions(-)
arch/alpha/kernel/pci_iommu.c +1 -1
@@ -929,7 +929,7 @@ const struct dma_map_ops alpha_pci_ops = { .dma_supported = alpha_pci_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };EXPORT_SYMBOL (alpha_pci_ops);
arch/mips/jazz/jazzdma.c +1 -1
@@ -617,7 +617,7 @@ const struct dma_map_ops jazz_dma_ops = { .sync_sg_for_device = jazz_dma_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };EXPORT_SYMBOL (jazz_dma_ops);
arch/powerpc/kernel/dma-iommu.c +1 -1
@@ -216,6 +216,6 @@ const struct dma_map_ops dma_iommu_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };
arch/powerpc/platforms/ps3/system-bus.c +2 -2
@@ -695,7 +695,7 @@ static const struct dma_map_ops ps3_sb_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };
@@ -709,7 +709,7 @@ static const struct dma_map_ops ps3_ioc0_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };
arch/powerpc/platforms/pseries/vio.c +1 -1
@@ -611,7 +611,7 @@ static const struct dma_map_ops vio_dma_mapping_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };
arch/x86/kernel/amd_gart_64.c +1 -1
@@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, };
drivers/iommu/dma-iommu.c +1 -1
@@ -1710,7 +1710,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous,
drivers/parisc/ccio-dma.c +1 -1
@@ -1022,7 +1022,7 @@ static const struct dma_map_ops ccio_ops = { .map_sg = ccio_map_sg, .unmap_sg = ccio_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };
drivers/parisc/sba_iommu.c +1 -1
@@ -1090,7 +1090,7 @@ static const struct dma_map_ops sba_ops = { .map_sg = sba_map_sg, .unmap_sg = sba_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, };
drivers/xen/grant-dma-ops.c +1 -1
@@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask)static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable,
drivers/xen/swiotlb-xen.c +1 -1
@@ -403,7 +403,7 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, };
include/linux/dma-map-ops.h +1 -1
@@ -29,7 +29,7 @@ struct dma_map_ops { unsigned long attrs);void (*free)(struct device *dev,size_t size,void *vaddr,dma_addr_t dma_handle,unsigned long attrs); -struct page *(*alloc_pages)(struct device *dev,size_t size, +struct page *(*alloc_pages_op)(struct device *dev,size_t size,dma_addr_t *dma_handle,enum dma_data_direction dir,gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr,
kernel/dma/mapping.c +2 -2
@@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size =PAGE_ALIGN (size);if (dma_alloc_direct (dev, ops))return dma_direct_alloc_pages (dev, size, dma_handle, dir, gfp); -if (!ops->alloc_pages) +if (!ops->alloc_pages_op)return NULL ; -return ops->alloc_pages (dev, size, dma_handle, dir, gfp); +return ops->alloc_pages_op (dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size,
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 18/36 mm: enable page allocation tagging
In reply to Suren BaghdasaryanRedefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag.
--- include/linux/alloc_tag.h | 14 +++++ include/linux/gfp.h | 126 ++++++++++++++++++++++++-------------- include/linux/pagemap.h | 9 ++- mm/compaction.c | 7 ++- mm/filemap.c | 6 +- mm/mempolicy.c | 52 ++++++++-------- mm/page_alloc.c | 60 +++++++++--------- 7 files changed, 164 insertions(+), 110 deletions(-)
include/linux/alloc_tag.h +14 -0
@@ -141,4 +141,18 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag,#endif /* CONFIG_MEM_ALLOC_PROFILING */ +#define alloc_hooks_tag (_tag, _do_alloc) \ +({ \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(_tag); \ + typeof(_do_alloc) _res = _do_alloc; \ + alloc_tag_restore(_tag, _old); \ + _res; \ +}) + +#define alloc_hooks (_do_alloc) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc); \ +}) +#endif /* _LINUX_ALLOC_TAG_H */
include/linux/gfp.h +79 -47
219 lines not on this page · raw message
include/linux/pagemap.h +6 -3
@@ -542,14 +542,17 @@ static inline void *detach_page_private(struct page *page)#endif #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio (gfp_t gfp,unsigned int order); +struct folio *filemap_alloc_folio_noprof (gfp_t gfp,unsigned int order);#else -static inline struct folio *filemap_alloc_folio (gfp_t gfp,unsigned int order) +static inline struct folio *filemap_alloc_folio_noprof (gfp_t gfp,unsigned int order) { -return folio_alloc (gfp, order); +return folio_alloc_noprof (gfp, order); }#endif +#define filemap_alloc_folio (...) \ + alloc_hooks(filemap_alloc_folio_noprof(__VA_ARGS__)) +static inline struct page *__page_cache_alloc (gfp_t gfp) {return &filemap_alloc_folio (gfp,0 )->page;
mm/compaction.c +6 -1
@@ -1781,7 +1781,7 @@ static void isolate_freepages(struct compact_control *cc)* This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct folio *compaction_alloc (struct folio *src,unsigned long data) +static struct folio *compaction_alloc_noprof (struct folio *src,unsigned long data) {struct compact_control *cc = (struct compact_control *)data;struct folio *dst;
@@ -1800,6 +1800,11 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)return dst; } +static struct folio *compaction_alloc (struct folio *src,unsigned long data) +{ +return alloc_hooks (compaction_alloc_noprof (src, data)); +} +/* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no
mm/filemap.c +3 -3
@@ -957,7 +957,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio,EXPORT_SYMBOL_GPL (filemap_add_folio);#ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio (gfp_t gfp,unsigned int order) +struct folio *filemap_alloc_folio_noprof (gfp_t gfp,unsigned int order) {int n;struct folio *folio;
@@ -972,9 +972,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } -return folio_alloc (gfp, order); +return folio_alloc_noprof (gfp, order); } -EXPORT_SYMBOL (filemap_alloc_folio); +EXPORT_SYMBOL (filemap_alloc_folio_noprof);#endif /*
mm/mempolicy.c +26 -26
153 lines not on this page · raw message
mm/page_alloc.c +30 -30
180 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 19/36 mm: create new codetag references during page splitting
In reply to Suren BaghdasaryanWhen a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page.
--- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+)
include/linux/pgalloc_tag.h +30 -0
@@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split (struct page *page,unsigned int nr) +{ +int i; +struct page_ext *page_ext; +union codetag_ref *ref; +struct alloc_tag *tag; + +if (!mem_alloc_profiling_enabled ()) +return ; + + page_ext =page_ext_get (page); +if (unlikely (!page_ext)) +return ; + + ref =codetag_ref_from_page_ext (page_ext); +if (!ref->ct) + goto out; + + tag =ct_to_alloc_tag (ref->ct); + page_ext =page_ext_next (page_ext); +for (i =1 ; i < nr; i++) { +/* Set new reference to point to the original tag */ +alloc_tag_ref_set (codetag_ref_from_page_ext (page_ext), tag); + page_ext =page_ext_next (page_ext); + } +out: +page_ext_put (page_ext); +} +#else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add (struct page *page,struct task_struct *task,unsigned int order) {}static inline void pgalloc_tag_sub (struct page *page,unsigned int order) {} +static inline void pgalloc_tag_split (struct page *page,unsigned int nr) {}#endif /* CONFIG_MEM_ALLOC_PROFILING */
mm/huge_memory.c +2 -0
@@ -38,6 +38,7 @@#include <linux/sched/sysctl.h> #include <linux/memory-tiers.h> #include <linux/compat.h> +#include <linux/pgalloc_tag.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
@@ -2899,6 +2900,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr); +pgalloc_tag_split (head, nr);/* See comment in __split_huge_page_tail() */ if (PageAnon (head)) {
mm/page_alloc.c +2 -0
@@ -2621,6 +2621,7 @@ void split_page(struct page *page, unsigned int order)for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i);split_page_owner (page,1 << order); +pgalloc_tag_split (page,1 << order);split_page_memcg (page,1 << order); }EXPORT_SYMBOL_GPL (split_page);
@@ -4806,6 +4807,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order,struct page *last = page + nr;split_page_owner (page,1 << order); +pgalloc_tag_split (page,1 << order);split_page_memcg (page,1 << order);while (page < --last)set_page_refcounted (last);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 19/36] mm: create new codetag references during page splitting
Quoted 4 lines · Suren Baghdasaryan
When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page.
This was v3 but then you refactored (for the better) so the commit log could reflect it?
Quoted 1 line
Signed-off-by Suren Baghdasaryan <[email protected]>
I was going to R-b, but now I recalled the trickiness of __free_pages() for non-compound pages if it loses the race to a speculative reference. Will the codetag handling work fine there?
Quoted 92 lines
--- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+)include/linux/pgalloc_tag.h +30
-0@@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split (struct page *page,unsigned int nr) +{ +int i; +struct page_ext *page_ext; +union codetag_ref *ref; +struct alloc_tag *tag; + +if (!mem_alloc_profiling_enabled ()) +return ; + + page_ext =page_ext_get (page); +if (unlikely (!page_ext)) +return ; + + ref =codetag_ref_from_page_ext (page_ext); +if (!ref->ct) + goto out; + + tag =ct_to_alloc_tag (ref->ct); + page_ext =page_ext_next (page_ext); +for (i =1 ; i < nr; i++) { +/* Set new reference to point to the original tag */ +alloc_tag_ref_set (codetag_ref_from_page_ext (page_ext), tag); + page_ext =page_ext_next (page_ext); + } +out: +page_ext_put (page_ext); +} +#else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add (struct page *page,struct task_struct *task,unsigned int order) {}static inline void pgalloc_tag_sub (struct page *page,unsigned int order) {} +static inline void pgalloc_tag_split (struct page *page,unsigned int nr) {}#endif /* CONFIG_MEM_ALLOC_PROFILING */ mm/huge_memory.c +2
-0@@ -38,6 +38,7 @@#include <linux/sched/sysctl.h> #include <linux/memory-tiers.h> #include <linux/compat.h> +#include <linux/pgalloc_tag.h> #include <asm/tlb.h> #include <asm/pgalloc.h> @@ -2899,6 +2900,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr); +pgalloc_tag_split (head, nr);/* See comment in __split_huge_page_tail() */ if (PageAnon (head)) {mm/page_alloc.c +2
-0@@ -2621,6 +2621,7 @@ void split_page(struct page *page, unsigned int order)for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i);split_page_owner (page,1 << order); +pgalloc_tag_split (page,1 << order);split_page_memcg (page,1 << order); }EXPORT_SYMBOL_GPL (split_page);@@ -4806,6 +4807,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order,struct page *last = page + nr;split_page_owner (page,1 << order); +pgalloc_tag_split (page,1 << order);split_page_memcg (page,1 << order);while (page < --last)set_page_refcounted (last);
Suren Baghdasaryan
maintainerRe: [PATCH v4 19/36] mm: create new codetag references during page splitting
Quoted 9 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page.
This was v3 but then you refactored (for the better) so the commit log could reflect it?
Yes, technically mechnism didn't change but I should word it better. Smth like this:
When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Suren Baghdasaryan <[email protected]>I was going to R-b, but now I recalled the trickiness of
__free_pages()for non-compound pages if it loses the race to a speculative reference. Will the codetag handling work fine there?
I think so. Each non-compoud page has its individual reference to its codetag and will decrement it whenever the page is freed. IIUC the logic in __free_pages(), when it loses race to a speculative reference it will free all pages except for the first one and the first one will be freed when the last put_page() happens. If prior to this all these pages were split from one page then all of them will have their own reference which points to the same codetag. Every time one of these pages are freed that codetag's "bytes" and "calls" counters will be decremented. I think accounting will work correctly irrespective of where these pages are freed, in __free_pages() or by put_page().
Quoted 93 lines
Quoted 93 lines
--- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+)include/linux/pgalloc_tag.h +30
-0@@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split (struct page *page,unsigned int nr) +{ +int i; +struct page_ext *page_ext; +union codetag_ref *ref; +struct alloc_tag *tag; + +if (!mem_alloc_profiling_enabled ()) +return ; + + page_ext =page_ext_get (page); +if (unlikely (!page_ext)) +return ; + + ref =codetag_ref_from_page_ext (page_ext); +if (!ref->ct) + goto out; + + tag =ct_to_alloc_tag (ref->ct); + page_ext =page_ext_next (page_ext); +for (i =1 ; i < nr; i++) { +/* Set new reference to point to the original tag */ +alloc_tag_ref_set (codetag_ref_from_page_ext (page_ext), tag); + page_ext =page_ext_next (page_ext); + } +out: +page_ext_put (page_ext); +} +#else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add (struct page *page,struct task_struct *task,unsigned int order) {}static inline void pgalloc_tag_sub (struct page *page,unsigned int order) {} +static inline void pgalloc_tag_split (struct page *page,unsigned int nr) {}#endif /* CONFIG_MEM_ALLOC_PROFILING */ mm/huge_memory.c +2
-0@@ -38,6 +38,7 @@#include <linux/sched/sysctl.h> #include <linux/memory-tiers.h> #include <linux/compat.h> +#include <linux/pgalloc_tag.h> #include <asm/tlb.h> #include <asm/pgalloc.h> @@ -2899,6 +2900,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr); +pgalloc_tag_split (head, nr);/* See comment in __split_huge_page_tail() */ if (PageAnon (head)) {mm/page_alloc.c +2
-0@@ -2621,6 +2621,7 @@ void split_page(struct page *page, unsigned int order)for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i);split_page_owner (page,1 << order); +pgalloc_tag_split (page,1 << order);split_page_memcg (page,1 << order); }EXPORT_SYMBOL_GPL (split_page);@@ -4806,6 +4807,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order,struct page *last = page + nr;split_page_owner (page,1 << order); +pgalloc_tag_split (page,1 << order);split_page_memcg (page,1 << order);while (page < --last)set_page_refcounted (last);
Vlastimil Babka
Reviewed-byRe: [PATCH v4 19/36] mm: create new codetag references during page splitting
Quoted 14 lines · Suren Baghdasaryan
Quoted 9 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page.
This was v3 but then you refactored (for the better) so the commit log could reflect it?
Yes, technically mechnism didn't change but I should word it better. Smth like this:
Quoted 6 lines
When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed.
Great, thanks.
The concern with __free_pages() is not really related to splitting, so for this patch:
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 8 lines
Quoted 3 lines
I was going to R-b, but now I recalled the trickiness of
__free_pages()for non-compound pages if it loses the race to a speculative reference. Will the codetag handling work fine there?I think so. Each non-compoud page has its individual reference to its codetag and will decrement it whenever the page is freed. IIUC the logic in
__free_pages(), when it loses race to a speculative reference it will free all pages except for the first one and the
The "tail" pages of this non-compound high-order page will AFAICS not have code tags assigned, so alloc_tag_sub() will be a no-op (or a warning with _DEBUG).
Quoted 3 lines
first one will be freed when the last
put_page()happens. If prior to this all these pages were split from one page then all of them will have their own reference which points to the same codetag.
Yeah I'm assuming there's no split before the freeing. This patch about splitting just reminded me of that tricky freeing scenario.
So IIUC the "else if (!head)" path of __free_pages() will do nothing about the "tail" pages wrt code tags as there are no code tags.
Then whoever took the speculative "head" page reference will put_page() and free it, which will end up in alloc_tag_sub(). This will decrement calls properly, but bytes will become imbalanced, because that put_page() will pass order-0 worth of bytes - the original order is lost.
Now this might be rare enough that it's not worth fixing if that would be too complicated, just FYI.
Quoted 6 lines
Every time
one of these pages are freed that codetag's "bytes" and "calls" counters will be decremented. I think accounting will work correctly irrespective of where these pages are freed, in__free_pages()or byput_page().
Suren Baghdasaryan
maintainerRe: [PATCH v4 19/36] mm: create new codetag references during page splitting
Quoted 37 lines · Vlastimil Babka
Quoted 14 lines · Suren Baghdasaryan
Quoted 9 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page.
This was v3 but then you refactored (for the better) so the commit log could reflect it?
Yes, technically mechnism didn't change but I should word it better. Smth like this:
Quoted 6 lines
When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed.
Great, thanks.
The concern with__free_pages()is not really related to splitting, so for this patch:Reviewed-by Vlastimil Babka <[email protected]>Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Signed-off-by Suren Baghdasaryan <[email protected]>I was going to R-b, but now I recalled the trickiness of
__free_pages()for non-compound pages if it loses the race to a speculative reference. Will the codetag handling work fine there?
Quoted 8 lines
Quoted 4 lines
I think so. Each non-compoud page has its individual reference to its codetag and will decrement it whenever the page is freed. IIUC the logic in
__free_pages(), when it loses race to a speculative reference it will free all pages except for the first one and theThe "tail" pages of this non-compound high-order page will AFAICS not have code tags assigned, so
alloc_tag_sub()will be a no-op (or a warning with_DEBUG).
Yes, that is correct.
Quoted 7 lines
Quoted 4 lines
first one will be freed when the last
put_page()happens. If prior to this all these pages were split from one page then all of them will have their own reference which points to the same codetag.Yeah I'm assuming there's no split before the freeing. This patch about splitting just reminded me of that tricky freeing scenario.
Ah, I see. I thought you were talking about a page that was previously split.
Quoted 7 lines
So IIUC the "else if (!head)" path of
__free_pages()will do nothing about the "tail" pages wrt code tags as there are no code tags.
Then whoever took the speculative "head" page reference willput_page()and free it, which will end up inalloc_tag_sub(). This will decrement calls properly, but bytes will become imbalanced, because thatput_page()will pass order-0 worth of bytes - the original order is lost.
Yeah, that's true. put_page() will end up calling free_unref_page(&folio->page, 0) even if the original order was more than 0.
Quoted 3 lines
Now this might be rare enough that it's not worth fixing if that would be too complicated, just FYI.
Yeah. We can fix this by subtracting the "bytes" counter of the "head" page for all free_the_page(page + (1 << order), order) calls we do inside __free_pages(). But we can't simply use pgalloc_tag_sub() because the "calls" counter will get over-decremented (we allocated all of these pages with one call). I'll need to introduce a new pgalloc_tag_sub_bytes() API and use it here. I feel it's too targeted of a solution but OTOH this is a special situation, so maybe it's acceptable. WDYT?
Quoted 9 lines
Quoted 8 lines
Every time
one of these pages are freed that codetag's "bytes" and "calls" counters will be decremented. I think accounting will work correctly irrespective of where these pages are freed, in__free_pages()or byput_page().
Re: [PATCH v4 19/36] mm: create new codetag references during page splitting
Quoted 6 lines · Suren Baghdasaryan
Quoted 3 lines · Vlastimil Babka
Now this might be rare enough that it's not worth fixing if that would be too complicated, just FYI.
Quoted 8 lines
Yeah. We can fix this by subtracting the "bytes" counter of the "head" page for all
free_the_page(page + (1 << order), order)calls we do inside__free_pages(). But we can't simply usepgalloc_tag_sub()because the "calls" counter will get over-decremented (we allocated all of these pages with one call). I'll need to introduce a newpgalloc_tag_sub_bytes()API and use it here. I feel it's too targeted of a solution but OTOH this is a special situation, so maybe it's acceptable. WDYT?
Hmm I think there's a problem that once you fail put_page_testzero() and detect you need to do this, the page might be already gone or reallocated so you can't get to the tag for decrementing bytes. You'd have to get it upfront (I guess for "head && order > 0" cases) just in case it happens. Maybe it's not worth the trouble for such a rare case.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Every time
one of these pages are freed that codetag's "bytes" and "calls" counters will be decremented. I think accounting will work correctly irrespective of where these pages are freed, in__free_pages()or byput_page().
Suren Baghdasaryan
maintainerRe: [PATCH v4 19/36] mm: create new codetag references during page splitting
Quoted 17 lines · Vlastimil Babka
Quoted 6 lines · Suren Baghdasaryan
Quoted 3 lines · Vlastimil Babka
Now this might be rare enough that it's not worth fixing if that would be too complicated, just FYI.
Quoted 8 lines
Yeah. We can fix this by subtracting the "bytes" counter of the "head" page for all
free_the_page(page + (1 << order), order)calls we do inside__free_pages(). But we can't simply usepgalloc_tag_sub()because the "calls" counter will get over-decremented (we allocated all of these pages with one call). I'll need to introduce a newpgalloc_tag_sub_bytes()API and use it here. I feel it's too targeted of a solution but OTOH this is a special situation, so maybe it's acceptable. WDYT?
Quoted 5 lines
Hmm I think there's a problem that once you fail
put_page_testzero()and detect you need to do this, the page might be already gone or reallocated so you can't get to the tag for decrementing bytes. You'd have to get it upfront (I guess for "head && order > 0" cases) just in case it happens. Maybe it's not worth the trouble for such a rare case.
Yes, that hit me when I tried to implement it but there is a simple solution around that. I can obtain alloc_tag before doing put_page_testzero() and then decrement bytes counter directly as needed.
Not sure if it is a rare enough case that we can ignore it but if the fix is simple enough then might as well do it?
Quoted 14 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Every time
one of these pages are freed that codetag's "bytes" and "calls" counters will be decremented. I think accounting will work correctly irrespective of where these pages are freed, in__free_pages()or byput_page().--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerPATCH v4 20/36 mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
In reply to Suren BaghdasaryanFor all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING.
--- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+)
mm/page_ext.c +9 -0
@@ -95,7 +95,16 @@ unsigned long page_ext_size;static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* +* To ensure correct allocation tagging for pages, page_ext should be available +* before the first page allocation. Otherwise early task stacks will be +* allocated before page_ext initialization and missing tags will be flagged. +*/ +bool early_page_ext__meminitdata = true; +#else bool early_page_ext__meminitdata ; +#endif static int __init setup_early_page_ext (char *str) { early_page_ext = true;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 20/36] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
Quoted 12 lines · Suren Baghdasaryan
For all page allocations to be tagged,
page_exthas to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator beforealloc_node_page_ext()initializespage_extarea, unlessearly_page_extis enabled. Therefore these allocations will generate a warning whenCONFIG_MEM_ALLOC_PROFILING_DEBUGis enabled. Enableearly_page_extwheneverCONFIG_MEM_ALLOC_PROFILING_DEBUG=yto ensurepage_extinitialization prior to any page allocation. This will have all the negative effects associated withearly_page_ext, such as possible longer boot time, therefore we enable it only when debugging withCONFIG_MEM_ALLOC_PROFILING_DEBUGenabled and not universally forCONFIG_MEM_ALLOC_PROFILING.
Quoted 1 line
Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 25 lines
--- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+)mm/page_ext.c +9
-0@@ -95,7 +95,16 @@ unsigned long page_ext_size;static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* +* To ensure correct allocation tagging for pages, page_ext should be available +* before the first page allocation. Otherwise early task stacks will be +* allocated before page_ext initialization and missing tags will be flagged. +*/ +bool early_page_ext__meminitdata = true; +#else bool early_page_ext__meminitdata ; +#endif static int __init setup_early_page_ext (char *str) { early_page_ext = true;
Suren Baghdasaryan
maintainerPATCH v4 21/36 lib: add codetag reference into slabobj_ext
In reply to Suren BaghdasaryanTo store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y.
--- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + 2 files changed, 6 insertions(+)
include/linux/memcontrol.h +5 -0
@@ -1653,7 +1653,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,* if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +union codetag_ref ref; +#endif }__aligned(8) ;static inline void __inc_lruvec_kmem_state (void *p,enum node_stat_item idx)
lib/Kconfig.debug +1 -0
@@ -979,6 +979,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 21/36] lib: add codetag reference into slabobj_ext
Quoted 6 lines · Suren Baghdasaryan
To store code tag for every slab object, a codetag reference is embedded into
slabobj_extwhenCONFIG_MEM_ALLOC_PROFILING=y.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>
Quoted 34 lines
--- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + 2 files changed, 6 insertions(+)include/linux/memcontrol.h +5
-0@@ -1653,7 +1653,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,* if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +union codetag_ref ref; +#endif }__aligned(8) ;static inline void __inc_lruvec_kmem_state (void *p,enum node_stat_item idx)lib/Kconfig.debug +1
-0@@ -979,6 +979,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track
Suren Baghdasaryan
maintainerPATCH v4 22/36 mm/slab: add allocation accounting into slab allocation and free paths
In reply to Suren BaghdasaryanAccount slab allocations using codetag reference embedded into slabobj_ext.
--- mm/slab.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 9 ++++++++ 2 files changed, 75 insertions(+)
mm/slab.h +66 -0
@@ -567,6 +567,46 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab)int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s,gfp_t gfp,bool new_slab); +static inline bool need_slab_obj_ext (void ) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING +if (mem_alloc_profiling_enabled ()) +return true; +#endif +/* +* CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally +* inside memcg_slab_post_alloc_hook. No other users for now. +*/ +return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook (struct kmem_cache *s,gfp_t flags,void *p) +{ +struct slab *slab; + +if (!p) +return NULL ; + +if (!need_slab_obj_ext ()) +return NULL ; + +if (s->flags &SLAB_NO_OBJ_EXT ) +return NULL ; + +if (flags & __GFP_NO_OBJ_EXT) +return NULL ; + + slab =virt_to_slab (p); +if (!slab_obj_exts (slab) && +WARN (alloc_slab_obj_exts (slab, s, flags, false), +"%s, %s: Failed to create slab extension vector!\n" , + __func__, s->name)) +return NULL ; + +return slab_obj_exts (slab) +obj_to_index (s, slab, p); +} +#else /* CONFIG_SLAB_OBJ_EXT */ static inline struct slabobj_ext *slab_obj_exts (struct slab *slab)
@@ -589,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p)#endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook (struct kmem_cache *s,struct slab *slab, +void **p,int objects) +{ +struct slabobj_ext *obj_exts; +int i; + + obj_exts =slab_obj_exts (slab); +if (!obj_exts) +return ; + +for (i =0 ; i < objects; i++) { +unsigned int off =obj_to_index (s, slab, p[i]); + +alloc_tag_sub (&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook (struct kmem_cache *s,struct slab *slab, +void **p,int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ +#ifdef CONFIG_MEMCG_KMEM void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat,enum node_stat_item idx,int nr);
mm/slub.c +9 -0
@@ -3826,6 +3826,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) {unsigned int zero_size = s->object_size; +struct slabobj_ext *obj_exts;bool kasan_init = init;size_t i;gfp_t init_flags = flags & gfp_allowed_mask;
@@ -3868,6 +3869,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg,kmemleak_alloc_recursive (p[i], s->object_size,1 , s->flags, init_flags);kmsan_slab_alloc (s, p[i], init_flags); + obj_exts =prepare_slab_obj_exts_hook (s, flags, p[i]); +#ifdef CONFIG_MEM_ALLOC_PROFILING +/* obj_exts can be allocated for other reasons */ +if (likely (obj_exts) &&mem_alloc_profiling_enabled ()) +alloc_tag_add (&obj_exts->ref, current->alloc_tag, s->size); +#endif }memcg_slab_post_alloc_hook (s, objcg, flags, size, p);
@@ -4346,6 +4353,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) {memcg_slab_free_hook (s, slab, &object,1 ); +alloc_tagging_slab_free_hook (s, slab, &object,1 );if (likely (slab_free_hook (s, object,slab_want_init_on_free (s))))do_slab_free (s, slab, object, object,1 , addr);
@@ -4356,6 +4364,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) {memcg_slab_free_hook (s ,slab ,p ,cnt ); +alloc_tagging_slab_free_hook (s, slab, p, cnt);/* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed.
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 22/36] mm/slab: add allocation accounting into slab allocation and free paths
Quoted 63 lines · Suren Baghdasaryan
Account slab allocations using codetag reference embedded into
slabobj_ext.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slab.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 9 ++++++++ 2 files changed, 75 insertions(+)mm/slab.h +40
-0@@ -567,6 +567,46 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab)int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s,gfp_t gfp,bool new_slab); +static inline bool need_slab_obj_ext (void ) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING +if (mem_alloc_profiling_enabled ()) +return true; +#endif +/* +* CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally +* inside memcg_slab_post_alloc_hook. No other users for now. +*/ +return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook (struct kmem_cache *s,gfp_t flags,void *p) +{ +struct slab *slab; + +if (!p) +return NULL ; + +if (!need_slab_obj_ext ()) +return NULL ; + +if (s->flags &SLAB_NO_OBJ_EXT ) +return NULL ; + +if (flags & __GFP_NO_OBJ_EXT) +return NULL ; + + slab =virt_to_slab (p); +if (!slab_obj_exts (slab) && +WARN (alloc_slab_obj_exts (slab, s, flags, false), +"%s, %s: Failed to create slab extension vector!\n" , + __func__, s->name)) +return NULL ; + +return slab_obj_exts (slab) +obj_to_index (s, slab, p); +} +#else /* CONFIG_SLAB_OBJ_EXT */ static inline struct slabobj_ext *slab_obj_exts (struct slab *slab)
mm/slab.h prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) · line 635
Quoted 7 lines
mm/slab.h +4
-0@@ -589,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p)#endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook (struct kmem_cache *s,struct slab *slab, +void **p,int objects)
Only used from mm/slub.c so could move?
Quoted 38 lines
mm/slab.h +22
-0+{ +struct slabobj_ext *obj_exts; +int i; + + obj_exts =slab_obj_exts (slab); +if (!obj_exts) +return ; + +for (i =0 ; i < objects; i++) { +unsigned int off =obj_to_index (s, slab, p[i]); + +alloc_tag_sub (&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook (struct kmem_cache *s,struct slab *slab, +void **p,int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ +#ifdef CONFIG_MEMCG_KMEM void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat,enum node_stat_item idx,int nr);mm/slub.c +1
-0@@ -3826,6 +3826,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) {unsigned int zero_size = s->object_size; +struct slabobj_ext *obj_exts;bool kasan_init = init;size_t i;gfp_t init_flags = flags & gfp_allowed_mask;
mm/slub.c void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, · line 3877
Quoted 9 lines
mm/slub.c +6
-0@@ -3868,6 +3869,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg,kmemleak_alloc_recursive (p[i], s->object_size,1 , s->flags, init_flags);kmsan_slab_alloc (s, p[i], init_flags); + obj_exts =prepare_slab_obj_exts_hook (s, flags, p[i]); +#ifdef CONFIG_MEM_ALLOC_PROFILING +/* obj_exts can be allocated for other reasons */ +if (likely (obj_exts) &&mem_alloc_profiling_enabled ()) +alloc_tag_add (&obj_exts->ref, current->alloc_tag, s->size); +#endif
I think that like in the page allocator, this could be better guarded by mem_alloc_profiling_enabled() as the outermost thing.
mm/slub.c void slab_free(struct kmem_cache *s, struct slab *slab, void *object, · line 4356
Quoted 8 lines
mm/slub.c +1
-0}memcg_slab_post_alloc_hook (s, objcg, flags, size, p);@@ -4346,6 +4353,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) {memcg_slab_free_hook (s, slab, &object,1 ); +alloc_tagging_slab_free_hook (s, slab, &object,1 );
Same here, the static key is not even inside of this?
mm/slub.c void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, · line 4367
Quoted 8 lines
mm/slub.c +1
-0if (likely (slab_free_hook (s, object,slab_want_init_on_free (s))))do_slab_free (s, slab, object, object,1 , addr);@@ -4356,6 +4364,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) {memcg_slab_free_hook (s ,slab ,p ,cnt ); +alloc_tagging_slab_free_hook (s, slab, p, cnt);
Ditto.
Quoted 3 lines
mm/slub.c +0
-0/* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed.
Suren Baghdasaryan
maintainerRe: [PATCH v4 22/36] mm/slab: add allocation accounting into slab allocation and free paths
Quoted 68 lines · Vlastimil Babka
Quoted 64 lines · Suren Baghdasaryan
Account slab allocations using codetag reference embedded into
slabobj_ext.Signed-off-by Suren Baghdasaryan <[email protected]>Co-developed-by Kent Overstreet <[email protected]>Signed-off-by Kent Overstreet <[email protected]>Reviewed-by Kees Cook <[email protected]>--- mm/slab.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 9 ++++++++ 2 files changed, 75 insertions(+)mm/slab.h +40
-0@@ -567,6 +567,46 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab)int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s,gfp_t gfp,bool new_slab); +static inline bool need_slab_obj_ext (void ) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING +if (mem_alloc_profiling_enabled ()) +return true; +#endif +/* +* CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally +* inside memcg_slab_post_alloc_hook. No other users for now. +*/ +return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook (struct kmem_cache *s,gfp_t flags,void *p) +{ +struct slab *slab; + +if (!p) +return NULL ; + +if (!need_slab_obj_ext ()) +return NULL ; + +if (s->flags &SLAB_NO_OBJ_EXT ) +return NULL ; + +if (flags & __GFP_NO_OBJ_EXT) +return NULL ; + + slab =virt_to_slab (p); +if (!slab_obj_exts (slab) && +WARN (alloc_slab_obj_exts (slab, s, flags, false), +"%s, %s: Failed to create slab extension vector!\n" , + __func__, s->name)) +return NULL ; + +return slab_obj_exts (slab) +obj_to_index (s, slab, p); +} +#else /* CONFIG_SLAB_OBJ_EXT */ static inline struct slabobj_ext *slab_obj_exts (struct slab *slab)@@ -589,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p)
Quoted 8 lines
mm/slab.h prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) · line 635
Quoted 6 lines
mm/slab.h +4
-0#endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook (struct kmem_cache *s,struct slab *slab, +void **p,int objects)Only used from
mm/slub.cso could move?
Ack.
Quoted 43 lines
Quoted 43 lines
mm/slab.h +22
-0+{ +struct slabobj_ext *obj_exts; +int i; + + obj_exts =slab_obj_exts (slab); +if (!obj_exts) +return ; + +for (i =0 ; i < objects; i++) { +unsigned int off =obj_to_index (s, slab, p[i]); + +alloc_tag_sub (&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook (struct kmem_cache *s,struct slab *slab, +void **p,int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ +#ifdef CONFIG_MEMCG_KMEM void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat,enum node_stat_item idx,int nr);mm/slub.c +2
-0@@ -3826,6 +3826,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) {unsigned int zero_size = s->object_size; +struct slabobj_ext *obj_exts;bool kasan_init = init;size_t i;gfp_t init_flags = flags & gfp_allowed_mask;@@ -3868,6 +3869,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg,kmemleak_alloc_recursive (p[i], s->object_size,1 , s->flags, init_flags);kmsan_slab_alloc (s, p[i], init_flags); + obj_exts =prepare_slab_obj_exts_hook (s, flags, p[i]);
Quoted 8 lines
mm/slub.c void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, · line 3877
Quoted 5 lines
mm/slub.c +5
-0+#ifdef CONFIG_MEM_ALLOC_PROFILING +/* obj_exts can be allocated for other reasons */ +if (likely (obj_exts) &&mem_alloc_profiling_enabled ()) +alloc_tag_add (&obj_exts->ref, current->alloc_tag, s->size); +#endif I think that like in the page allocator, this could be better guarded by
mem_alloc_profiling_enabled()as the outermost thing.
Oops, missed it. Will fix.
Quoted 11 lines
mm/slub.c void slab_free(struct kmem_cache *s, struct slab *slab, void *object, · line 4356
Quoted 9 lines
mm/slub.c +1
-0}memcg_slab_post_alloc_hook (s, objcg, flags, size, p);@@ -4346,6 +4353,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) {memcg_slab_free_hook (s, slab, &object,1 ); +alloc_tagging_slab_free_hook (s, slab, &object,1 );Same here, the static key is not even inside of this?
Ack.
Quoted 11 lines
mm/slub.c void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, · line 4367
Quoted 9 lines
mm/slub.c +1
-0if (likely (slab_free_hook (s, object,slab_want_init_on_free (s))))do_slab_free (s, slab, object, object,1 , addr);@@ -4356,6 +4364,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) {memcg_slab_free_hook (s ,slab ,p ,cnt ); +alloc_tagging_slab_free_hook (s, slab, p, cnt);Ditto.
Ack.
Quoted 8 lines
mm/slub.c void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, · line 4370
Quoted 4 lines
mm/slub.c +0
-0/* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. --
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerPATCH v4 23/36 mm/slab: enable slab allocation tagging for kmalloc and friends
In reply to Suren BaghdasaryanRedefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions.
--- include/linux/fortify-string.h | 5 +- include/linux/slab.h | 169 +++++++++++++++++---------------- include/linux/string.h | 4 +- mm/slab_common.c | 6 +- mm/slub.c | 52 +++++----- mm/util.c | 20 ++-- 6 files changed, 130 insertions(+), 126 deletions(-)
include/linux/fortify-string.h +3 -2
@@ -697,9 +697,9 @@ __FORTIFY_INLINE void *memchr_inv(const void * const POS0 p, int c, size_t size)return __real_memchr_inv (p, c, size); } -extern void *__real_kmemdup (const void *src,size_t len,gfp_t gfp)__RENAME (kmemdup) +extern void *__real_kmemdup (const void *src,size_t len,gfp_t gfp)__RENAME (kmemdup_noprof)__realloc_size (2 ); -__FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp) +__FORTIFY_INLINE void *kmemdup_noprof(const void * const POS0 p, size_t size, gfp_t gfp) {const size_t p_size =__struct_size (p);
@@ -709,6 +709,7 @@ __FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfpfortify_panic (__func__);return __real_kmemdup (p, size, gfp); } +#define kmemdup (...) alloc_hooks(kmemdup_noprof(__VA_ARGS__))/** * strcpy - Copy a string into another string buffer
include/linux/slab.h +85 -84
310 lines not on this page · raw message
include/linux/string.h +3 -1
@@ -214,7 +214,9 @@ extern void kfree_const(const void *x);extern char *kstrdup (const char *s,gfp_t gfp) __malloc;extern const char *kstrdup_const (const char *s,gfp_t gfp);extern char *kstrndup (const char *s,size_t len,gfp_t gfp); -extern void *kmemdup (const void *src,size_t len,gfp_t gfp)__realloc_size (2 ); +extern void *kmemdup_noprof (const void *src,size_t len,gfp_t gfp)__realloc_size (2 ); +#define kmemdup (...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) +extern void *kvmemdup (const void *src,size_t len,gfp_t gfp)__realloc_size (2 );extern char *kmemdup_nul (const char *s,size_t len,gfp_t gfp);
mm/slab_common.c +3 -3
@@ -1184,7 +1184,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)return (void *)p; } - ret =kmalloc_track_caller (new_size, flags); + ret =kmalloc_node_track_caller_noprof (new_size, flags,NUMA_NO_NODE , _RET_IP_);if (ret && p) {/* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current ();
@@ -1208,7 +1208,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)* * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc (const void *p,size_t new_size,gfp_t flags) +void *krealloc_noprof (const void *p,size_t new_size,gfp_t flags) {void *ret;
@@ -1223,7 +1223,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags)return ret; } -EXPORT_SYMBOL (krealloc); +EXPORT_SYMBOL (krealloc_noprof);/** * kfree_sensitive - Clear sensitive information in memory before freeing
mm/slub.c +26 -26
153 lines not on this page · raw message
mm/util.c +10 -10
@@ -115,7 +115,7 @@ char *kstrndup(const char *s, size_t max, gfp_t gfp)EXPORT_SYMBOL (kstrndup);/** -* kmemdup - duplicate region of memory +* kmemdup_noprof - duplicate region of memory * * @src: memory region to duplicate * @len: memory region length
@@ -124,16 +124,16 @@ EXPORT_SYMBOL(kstrndup);* Return: newly allocated copy of @src or %NULL in case of error, * result is physically contiguous. Use kfree() to free. */ -void *kmemdup (const void *src,size_t len,gfp_t gfp) +void *kmemdup_noprof (const void *src,size_t len,gfp_t gfp) {void *p; - p =kmalloc_track_caller (len, gfp); + p =kmalloc_node_track_caller_noprof (len, gfp,NUMA_NO_NODE , _RET_IP_);if (p)memcpy (p, src, len);return p; } -EXPORT_SYMBOL (kmemdup); +EXPORT_SYMBOL (kmemdup_noprof);/** * kvmemdup - duplicate region of memory
@@ -577,7 +577,7 @@ unsigned long vm_mmap(struct file *file, unsigned long addr,EXPORT_SYMBOL (vm_mmap);/** -* kvmalloc_node - attempt to allocate physically contiguous memory, but upon +* kvmalloc_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
@@ -592,7 +592,7 @@ EXPORT_SYMBOL(vm_mmap);* * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node (size_t size,gfp_t flags,int node) +void *kvmalloc_node_noprof (size_t size,gfp_t flags,int node) {gfp_t kmalloc_flags = flags;void *ret;
@@ -614,7 +614,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret =kmalloc_node (size, kmalloc_flags, node); + ret =kmalloc_node_noprof (size, kmalloc_flags, node);/* * It doesn't really make sense to fallback to vmalloc for sub page
@@ -643,7 +643,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags,PAGE_KERNEL ,VM_ALLOW_HUGE_VMAP , node, __builtin_return_address(0)); } -EXPORT_SYMBOL (kvmalloc_node); +EXPORT_SYMBOL (kvmalloc_node_noprof);/** * kvfree() - Free memory.
@@ -682,7 +682,7 @@ void kvfree_sensitive(const void *addr, size_t len) }EXPORT_SYMBOL (kvfree_sensitive); -void *kvrealloc (const void *p,size_t oldsize,size_t newsize,gfp_t flags) +void *kvrealloc_noprof (const void *p,size_t oldsize,size_t newsize,gfp_t flags) {void *newp;
@@ -695,7 +695,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags)kvfree (p);return newp; } -EXPORT_SYMBOL (kvrealloc); +EXPORT_SYMBOL (kvrealloc_noprof);/** * __vmalloc_array - allocate memory for a virtually contiguous array.
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 24/36 rust: Add a rust helper for krealloc()
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
Memory allocation profiling is turning krealloc() into a nontrivial macro - so for now, we need a helper for it.
Until we have proper support on the rust side for memory allocation profiling this does mean that all Rust allocations will be accounted to the helper.
--- rust/helpers.c | 8 ++++++++ 1 file changed, 8 insertions(+)
rust/helpers.c +8 -0
@@ -28,6 +28,7 @@#include <linux/mutex.h> #include <linux/refcount.h> #include <linux/sched/signal.h> +#include <linux/slab.h> #include <linux/spinlock.h> #include <linux/wait.h> #include <linux/workqueue.h>
@@ -157,6 +158,13 @@ void rust_helper_init_work_with_key(struct work_struct *work, work_func_t func, }EXPORT_SYMBOL_GPL (rust_helper_init_work_with_key); +void *__must_check rust_helper_krealloc (const void *objp,size_t new_size, +gfp_t flags)__realloc_size (2 ) +{ +return krealloc (objp, new_size, flags); +} +EXPORT_SYMBOL_GPL (rust_helper_krealloc); +/* * `bindgen` binds the C `size_t` type as the Rust `usize` type, so we can * use it in contexts where Rust expects a `usize` like slice (array) indices.
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 24/36] rust: Add a rust helper for krealloc()
Quoted 10 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
Memory allocation profiling is turning
krealloc()into a nontrivial macro - so for now, we need a helper for it.Until we have proper support on the rust side for memory allocation profiling this does mean that all Rust allocations will be accounted to the helper.
Quoted 12 lines
Signed-off-by Kent Overstreet <[email protected]>Cc Miguel Ojeda <[email protected]>Cc Alex Gaynor <[email protected]>Cc Wedson Almeida Filho <[email protected]>Cc Boqun Feng <[email protected]>Cc Gary Guo <[email protected]>Cc "Björn Roy Baron" <[email protected]>Cc Benno Lossin <[email protected]>Cc Andreas Hindborg <[email protected]>Cc Alice Ryhl <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Currently, the Rust build doesn't work throughout the entire series since there are some commits where krealloc is missing before you introduce the helper. If you introduce the helper first before krealloc stops being an exported function, then the Rust build should work throughout the entire series. (Having both the helper and the exported function at the same time is not a problem.)
With the patch reordered:
Alice
Suren Baghdasaryan
maintainerRe: [PATCH v4 24/36] rust: Add a rust helper for krealloc()
Quoted 25 lines · Alice Ryhl
Quoted 10 lines · Suren Baghdasaryan
From: Kent Overstreet <[email protected]>
Memory allocation profiling is turning
krealloc()into a nontrivial macro - so for now, we need a helper for it.Until we have proper support on the rust side for memory allocation profiling this does mean that all Rust allocations will be accounted to the helper.
Quoted 12 lines
Signed-off-by Kent Overstreet <[email protected]>Cc Miguel Ojeda <[email protected]>Cc Alex Gaynor <[email protected]>Cc Wedson Almeida Filho <[email protected]>Cc Boqun Feng <[email protected]>Cc Gary Guo <[email protected]>Cc "Björn Roy Baron" <[email protected]>Cc Benno Lossin <[email protected]>Cc Andreas Hindborg <[email protected]>Cc Alice Ryhl <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 6 lines
Currently, the Rust build doesn't work throughout the entire series since there are some commits where krealloc is missing before you introduce the helper. If you introduce the helper first before krealloc stops being an exported function, then the Rust build should work throughout the entire series. (Having both the helper and the exported function at the same time is not a problem.)
Ack. I'll move it up in the series.
Quoted 4 lines
With the patch reordered:
Reviewed-by Alice Ryhl <[email protected]>
Thanks Alice!
Quoted 2 lines
Alice
Suren Baghdasaryan
maintainerPATCH v4 25/36 mempool: Hook up to memory allocation profiling
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations.
Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places.
--- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 36 ++++++++------------ 2 files changed, 49 insertions(+), 60 deletions(-)
include/linux/mempool.h +35 -38
@@ -5,6 +5,8 @@#ifndef _LINUX_MEMPOOL_H#define _LINUX_MEMPOOL_H +#include <linux/sched.h> +#include <linux/alloc_tag.h> #include <linux/wait.h> #include <linux/compiler.h>
@@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool);int mempool_init_node (mempool_t *pool,int min_nr,mempool_alloc_t *alloc_fn,mempool_free_t *free_fn,void *pool_data,gfp_t gfp_mask,int node_id); -int mempool_init (mempool_t *pool,int min_nr,mempool_alloc_t *alloc_fn, + +int mempool_init_noprof (mempool_t *pool,int min_nr,mempool_alloc_t *alloc_fn,mempool_free_t *free_fn,void *pool_data); +#define mempool_init (...) \ + alloc_hooks(mempool_init_noprof(__VA_ARGS__))extern mempool_t *mempool_create (int min_nr,mempool_alloc_t *alloc_fn,mempool_free_t *free_fn,void *pool_data); -extern mempool_t *mempool_create_node (int min_nr,mempool_alloc_t *alloc_fn, + +extern mempool_t *mempool_create_node_noprof (int min_nr,mempool_alloc_t *alloc_fn,mempool_free_t *free_fn,void *pool_data,gfp_t gfp_mask,int nid); +#define mempool_create_node (...) \ + alloc_hooks(mempool_create_node_noprof(__VA_ARGS__)) + +#define mempool_create (_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE)extern int mempool_resize (mempool_t *pool,int new_min_nr);extern void mempool_destroy (mempool_t *pool); -extern void *mempool_alloc (mempool_t *pool,gfp_t gfp_mask) __malloc; + +extern void *mempool_alloc_noprof (mempool_t *pool,gfp_t gfp_mask) __malloc; +#define mempool_alloc (...) \ + alloc_hooks(mempool_alloc_noprof(__VA_ARGS__)) +extern void *mempool_alloc_preallocated (mempool_t *pool) __malloc;extern void mempool_free (void *element,mempool_t *pool);
@@ -62,19 +78,10 @@ extern void mempool_free(void *element, mempool_t *pool);void *mempool_alloc_slab (gfp_t gfp_mask,void *pool_data);void mempool_free_slab (void *element,void *pool_data); -static inline int -mempool_init_slab_pool (mempool_t *pool,int min_nr,struct kmem_cache *kc) -{ -return mempool_init (pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool (int min_nr,struct kmem_cache *kc) -{ -return mempool_create (min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool (_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool (_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc))/* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the
@@ -83,17 +90,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc)void *mempool_kmalloc (gfp_t gfp_mask,void *pool_data);void mempool_kfree (void *element,void *pool_data); -static inline int mempool_init_kmalloc_pool (mempool_t *pool,int min_nr,size_t size) -{ -return mempool_init (pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool (int min_nr,size_t size) -{ -return mempool_create (min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool (_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool (_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size))/* * A mempool_alloc_t and mempool_free_t for a simple page allocator that
@@ -102,16 +104,11 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size)void *mempool_alloc_pages (gfp_t gfp_mask,void *pool_data);void mempool_free_pages (void *element,void *pool_data); -static inline int mempool_init_page_pool (mempool_t *pool,int min_nr,int order) -{ -return mempool_init (pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long )order); -} - -static inline mempool_t *mempool_create_page_pool (int min_nr,int order) -{ -return mempool_create (min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long )order); -} +#define mempool_init_page_pool (_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool (_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order))#endif /* _LINUX_MEMPOOL_H */
mm/mempool.c +14 -22
@@ -240,17 +240,17 @@ EXPORT_SYMBOL(mempool_init_node);* * Return: %0 on success, negative error code otherwise. */ -int mempool_init (mempool_t *pool,int min_nr,mempool_alloc_t *alloc_fn, -mempool_free_t *free_fn,void *pool_data) +int mempool_init_noprof (mempool_t *pool,int min_nr,mempool_alloc_t *alloc_fn, +mempool_free_t *free_fn,void *pool_data) {return mempool_init_node (pool, min_nr, alloc_fn, free_fn, pool_data,GFP_KERNEL ,NUMA_NO_NODE ); } -EXPORT_SYMBOL (mempool_init); +EXPORT_SYMBOL (mempool_init_noprof);/** -* mempool_create - create a memory pool +* mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function.
@@ -265,17 +265,9 @@ EXPORT_SYMBOL(mempool_init);* * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create (int min_nr,mempool_alloc_t *alloc_fn, -mempool_free_t *free_fn,void *pool_data) -{ -return mempool_create_node (min_nr, alloc_fn, free_fn, pool_data, -GFP_KERNEL ,NUMA_NO_NODE ); -} -EXPORT_SYMBOL (mempool_create); - -mempool_t *mempool_create_node (int min_nr,mempool_alloc_t *alloc_fn, -mempool_free_t *free_fn,void *pool_data, -gfp_t gfp_mask,int node_id) +mempool_t *mempool_create_node_noprof (int min_nr,mempool_alloc_t *alloc_fn, +mempool_free_t *free_fn,void *pool_data, +gfp_t gfp_mask,int node_id) {mempool_t *pool;
@@ -291,7 +283,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn,return pool; } -EXPORT_SYMBOL (mempool_create_node); +EXPORT_SYMBOL (mempool_create_node_noprof);/** * mempool_resize - resize an existing memory pool
@@ -374,7 +366,7 @@ int mempool_resize(mempool_t *pool, int new_min_nr)EXPORT_SYMBOL (mempool_resize);/** -* mempool_alloc - allocate an element from a specific memory pool +* mempool_alloc_noprof - allocate an element from a specific memory pool * @pool: pointer to the memory pool which was allocated via * mempool_create(). * @gfp_mask: the usual allocation bitmask.
@@ -387,7 +379,7 @@ EXPORT_SYMBOL(mempool_resize);* * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc (mempool_t *pool,gfp_t gfp_mask) +void *mempool_alloc_noprof (mempool_t *pool,gfp_t gfp_mask) {void *element;unsigned long flags;
@@ -454,7 +446,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask)finish_wait (&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL (mempool_alloc); +EXPORT_SYMBOL (mempool_alloc_noprof);/** * mempool_alloc_preallocated - allocate an element from preallocated elements
@@ -562,7 +554,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) {struct kmem_cache *mem = pool_data;VM_BUG_ON (mem->ctor); -return kmem_cache_alloc (mem, gfp_mask); +return kmem_cache_alloc_noprof (mem, gfp_mask); }EXPORT_SYMBOL (mempool_alloc_slab);
@@ -580,7 +572,7 @@ EXPORT_SYMBOL(mempool_free_slab);void *mempool_kmalloc (gfp_t gfp_mask,void *pool_data) {size_t size = (size_t )pool_data; -return kmalloc (size, gfp_mask); +return kmalloc_noprof (size, gfp_mask); }EXPORT_SYMBOL (mempool_kmalloc);
@@ -597,7 +589,7 @@ EXPORT_SYMBOL(mempool_kfree);void *mempool_alloc_pages (gfp_t gfp_mask,void *pool_data) {int order = (int )(long )pool_data; -return alloc_pages (gfp_mask, order); +return alloc_pages_noprof (gfp_mask, order); }EXPORT_SYMBOL (mempool_alloc_pages);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 26/36 mm: percpu: Introduce pcpuobj_ext
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
Upcoming alloc tagging patches require a place to stash per-allocation metadata.
We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch.
--- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-)
mm/percpu-internal.h +17 -2
@@ -32,6 +32,16 @@ struct pcpu_block_md {int nr_bits;/* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM +struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif +struct pcpu_chunk {#ifdef CONFIG_PERCPU_STATS int nr_alloc;/* # of allocations */
@@ -64,8 +74,8 @@ struct pcpu_chunk {int end_offset;/* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM -struct obj_cgroup **obj_cgroups;/* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT +struct pcpuobj_ext *obj_exts;/* vector of object cgroups */ #endif int nr_pages;/* # of pages served by this chunk */
@@ -74,6 +84,11 @@ struct pcpu_chunk {unsigned long populated[];/* populated bitmap */ }; +static inline bool need_pcpuobj_ext (void ) +{ +return !mem_cgroup_kmem_disabled (); +} +extern spinlock_t pcpu_lock;extern struct list_head *pcpu_chunk_lists;
mm/percpu.c +15 -15
@@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr,panic ("%s: Failed to allocate %zu bytes\n" , __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups =NULL ; + chunk->obj_exts =NULL ;#endif pcpu_init_md_blocks (chunk);
@@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp)if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM -if (!mem_cgroup_kmem_disabled ()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT +if (need_pcpuobj_ext ()) { + chunk->obj_exts =pcpu_mem_zalloc (pcpu_chunk_map_bits (chunk) * -sizeof (struct obj_cgroup *), gfp); -if (!chunk->obj_cgroups) +sizeof (struct pcpuobj_ext ), gfp); +if (!chunk->obj_exts) goto objcg_fail; }#endif
@@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp)return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail:pcpu_mem_free (chunk->md_blocks);#endif
@@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) {if (!chunk)return ; -#ifdef CONFIG_MEMCG_KMEM -pcpu_mem_free (chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT +pcpu_mem_free (chunk->obj_exts);#endif pcpu_mem_free (chunk->md_blocks);pcpu_mem_free (chunk->bound_map);
@@ -1646,9 +1646,9 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg,if (!objcg)return ; -if (likely (chunk && chunk->obj_cgroups)) { +if (likely (chunk && chunk->obj_exts)) {obj_cgroup_get (objcg); - chunk->obj_cgroups[off >>PCPU_MIN_ALLOC_SHIFT ] = objcg; + chunk->obj_exts[off >>PCPU_MIN_ALLOC_SHIFT ].cgroup = objcg;rcu_read_lock (); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B,
@@ -1663,13 +1663,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) {struct obj_cgroup *objcg; -if (unlikely (!chunk->obj_cgroups)) +if (unlikely (!chunk->obj_exts))return ; - objcg = chunk->obj_cgroups[off >>PCPU_MIN_ALLOC_SHIFT ]; + objcg = chunk->obj_exts[off >>PCPU_MIN_ALLOC_SHIFT ].cgroup;if (!objcg)return ; - chunk->obj_cgroups[off >>PCPU_MIN_ALLOC_SHIFT ] =NULL ; + chunk->obj_exts[off >>PCPU_MIN_ALLOC_SHIFT ].cgroup =NULL ;obj_cgroup_uncharge (objcg,pcpu_obj_full_size (size));
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 27/36 mm: percpu: Add codetag reference into pcpuobj_ext
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added.
--- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-)
mm/percpu-internal.h +9 -2
@@ -36,9 +36,12 @@ struct pcpuobj_ext {#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup;#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM ) || defined(CONFIG_MEM_ALLOC_PROFILING )#define NEED_PCPUOBJ_EXT #endif
@@ -86,7 +89,11 @@ struct pcpu_chunk {static inline bool need_pcpuobj_ext (void ) { -return !mem_cgroup_kmem_disabled (); +if (IS_ENABLED (CONFIG_MEM_ALLOC_PROFILING )) +return true; +if (!mem_cgroup_kmem_disabled ()) +return true; +return false; }extern spinlock_t pcpu_lock;
mm/percpu.c +26 -0
@@ -1699,6 +1699,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) }#endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook (struct pcpu_chunk *chunk,int off, +size_t size) +{ +if (mem_alloc_profiling_enabled () &&likely (chunk->obj_exts)) { +alloc_tag_add (&chunk->obj_exts[off >>PCPU_MIN_ALLOC_SHIFT ].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook (struct pcpu_chunk *chunk,int off,size_t size) +{ +if (mem_alloc_profiling_enabled () &&likely (chunk->obj_exts)) +alloc_tag_sub_noalloc (&chunk->obj_exts[off >>PCPU_MIN_ALLOC_SHIFT ].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook (struct pcpu_chunk *chunk,int off, +size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook (struct pcpu_chunk *chunk,int off,size_t size) +{ +} +#endif +/** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 28/36 mm: percpu: enable per-cpu allocation tagging
In reply to Suren BaghdasaryanRedefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions.
--- include/linux/percpu.h | 23 ++++++++++----- mm/percpu.c | 64 +++++------------------------------------- 2 files changed, 23 insertions(+), 64 deletions(-)
include/linux/percpu.h +16 -7
@@ -2,6 +2,7 @@#ifndef __LINUX_PERCPU_H#define __LINUX_PERCPU_H +#include <linux/alloc_tag.h> #include <linux/mmdebug.h> #include <linux/preempt.h> #include <linux/smp.h>
@@ -9,6 +10,7 @@#include <linux/pfn.h> #include <linux/init.h> #include <linux/cleanup.h> +#include <linux/sched.h> #include <asm/percpu.h>
@@ -125,7 +127,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn);#endif -extern void __percpu *__alloc_reserved_percpu (size_t size,size_t align)__alloc_size(1) ;extern bool __is_kernel_percpu_address (unsigned long addr,unsigned long *can_addr);extern bool is_kernel_percpu_address (unsigned long addr);
@@ -133,14 +134,16 @@ extern bool is_kernel_percpu_address(unsigned long addr);extern void __init setup_per_cpu_areas (void );#endif -extern void __percpu *__alloc_percpu_gfp (size_t size,size_t align,gfp_t gfp)__alloc_size(1) ; -extern void __percpu *__alloc_percpu (size_t size,size_t align)__alloc_size(1) ; -extern void free_percpu (void __percpu *__pdata); +extern void __percpu *pcpu_alloc_noprof (size_t size,size_t align,bool reserved, +gfp_t gfp)__alloc_size(1) ;extern size_t pcpu_alloc_size (void __percpu *__pdata); -DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) - -extern phys_addr_t per_cpu_ptr_to_phys (void *addr); +#define __alloc_percpu_gfp (_size, _align, _gfp) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, _gfp)) +#define __alloc_percpu (_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, GFP_KERNEL)) +#define __alloc_reserved_percpu (_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, true, GFP_KERNEL))#define alloc_percpu_gfp (type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \
@@ -149,6 +152,12 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu (void __percpu *__pdata); + +DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) + +extern phys_addr_t per_cpu_ptr_to_phys (void *addr); +extern unsigned long pcpu_nr_pages (void );#endif /* __LINUX_PERCPU_H */
mm/percpu.c +7 -57
94 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 29/36 mm: vmalloc: Enable memory allocation profiling
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
This wrapps all external vmalloc allocation functions with the alloc_hooks() wrapper, and switches internal allocations to _noprof variants where appropriate, for the new memory allocation profiling feature.
--- drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- include/linux/vmalloc.h | 60 ++++++++++---- kernel/kallsyms_selftest.c | 2 +- mm/nommu.c | 64 +++++++-------- mm/util.c | 24 +++--- mm/vmalloc.c | 88 ++++++++++----------- 6 files changed, 135 insertions(+), 105 deletions(-)
drivers/staging/media/atomisp/pci/hmm/hmm.c +1 -1
@@ -205,7 +205,7 @@ static ia_css_ptr __hmm_alloc(size_t bytes, enum hmm_bo_type type, }dev_dbg (atomisp_dev,"pages: 0x%08x (%zu bytes), type: %d, vmalloc %p\n" , - bo->start, bytes, type, vmalloc); + bo->start, bytes, type, vmalloc_noprof);return bo->start;
include/linux/vmalloc.h +45 -15
77 lines not on this page · raw message
kernel/kallsyms_selftest.c +1 -1
@@ -82,7 +82,7 @@ static struct test_item test_items[] = {ITEM_FUNC (kallsyms_test_func_static),ITEM_FUNC (kallsyms_test_func),ITEM_FUNC (kallsyms_test_func_weak), -ITEM_FUNC (vmalloc), +ITEM_FUNC (vmalloc_noprof),ITEM_FUNC (vfree), #ifdef CONFIG_KALLSYMS_ALL ITEM_DATA(kallsyms_test_var_bss_static),
mm/nommu.c +32 -32
154 lines not on this page · raw message
mm/util.c +12 -12
@@ -639,7 +639,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node)* about the resulting pointer, and cannot play * protection games. */ -return __vmalloc_node_range (size,1 ,VMALLOC_START ,VMALLOC_END , +return __vmalloc_node_range_noprof (size,1 ,VMALLOC_START ,VMALLOC_END , flags,PAGE_KERNEL ,VM_ALLOW_HUGE_VMAP , node,__builtin_return_address (0 )); }
@@ -698,12 +698,12 @@ void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flagEXPORT_SYMBOL (kvrealloc_noprof);/** -* __vmalloc_array - allocate memory for a virtually contiguous array. +* __vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vmalloc_array (size_t n,size_t size,gfp_t flags) +void *__vmalloc_array_noprof (size_t n,size_t size,gfp_t flags) {size_t bytes;
@@ -711,18 +711,18 @@ void *__vmalloc_array(size_t n, size_t size, gfp_t flags)return NULL ;return __vmalloc (bytes, flags); } -EXPORT_SYMBOL (__vmalloc_array); +EXPORT_SYMBOL (__vmalloc_array_noprof);/** -* vmalloc_array - allocate memory for a virtually contiguous array. +* vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vmalloc_array (size_t n,size_t size) +void *vmalloc_array_noprof (size_t n,size_t size) {return __vmalloc_array (n, size,GFP_KERNEL ); } -EXPORT_SYMBOL (vmalloc_array); +EXPORT_SYMBOL (vmalloc_array_noprof);/** * __vcalloc - allocate and zero memory for a virtually contiguous array.
@@ -730,22 +730,22 @@ EXPORT_SYMBOL(vmalloc_array);* @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vcalloc (size_t n,size_t size,gfp_t flags) +void *__vcalloc_noprof (size_t n,size_t size,gfp_t flags) {return __vmalloc_array (n, size, flags | __GFP_ZERO); } -EXPORT_SYMBOL (__vcalloc); +EXPORT_SYMBOL (__vcalloc_noprof);/** -* vcalloc - allocate and zero memory for a virtually contiguous array. +* vcalloc_noprof - allocate and zero memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vcalloc (size_t n,size_t size) +void *vcalloc_noprof (size_t n,size_t size) {return __vmalloc_array (n, size,GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL (vcalloc); +EXPORT_SYMBOL (vcalloc_noprof);struct anon_vma *folio_anon_vma (struct folio *folio) {
mm/vmalloc.c +44 -44
241 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 30/36 rhashtable: Plumb through alloc tag
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
This gives better memory allocation profiling results; rhashtable allocations will be accounted to the code that initialized the rhashtable.
--- include/linux/alloc_tag.h | 3 +++ include/linux/rhashtable-types.h | 11 +++++++++-- lib/rhashtable.c | 28 +++++++++++++++++----------- 3 files changed, 29 insertions(+), 13 deletions(-)
include/linux/alloc_tag.h +3 -0
@@ -130,6 +130,8 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, this_cpu_add(tag->counters->bytes, bytes); } +#define alloc_tag_record (p) ((p) = current->alloc_tag) +#else /* CONFIG_MEM_ALLOC_PROFILING */ #define DEFINE_ALLOC_TAG (_alloc_tag)
@@ -138,6 +140,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {}static inline void alloc_tag_sub_noalloc (union codetag_ref *ref,size_t bytes) {}static inline void alloc_tag_add (union codetag_ref *ref,struct alloc_tag *tag,size_t bytes) {} +#define alloc_tag_record (p) do {} while (0)#endif /* CONFIG_MEM_ALLOC_PROFILING */
include/linux/rhashtable-types.h +9 -2
@@ -9,6 +9,7 @@#ifndef _LINUX_RHASHTABLE_TYPES_H#define _LINUX_RHASHTABLE_TYPES_H +#include <linux/alloc_tag.h> #include <linux/atomic.h> #include <linux/compiler.h> #include <linux/mutex.h>
@@ -88,6 +89,9 @@ struct rhashtable {struct mutex mutex;spinlock_t lock;atomic_t nelems; +#ifdef CONFIG_MEM_ALLOC_PROFILING +struct alloc_tag *alloc_tag; +#endif };/**
@@ -127,9 +131,12 @@ struct rhashtable_iter { bool end_of_table; }; -int rhashtable_init (struct rhashtable *ht, +int rhashtable_init_noprof (struct rhashtable *ht,const struct rhashtable_params *params); -int rhltable_init (struct rhltable *hlt, +#define rhashtable_init (...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__)) + +int rhltable_init_noprof (struct rhltable *hlt,const struct rhashtable_params *params); +#define rhltable_init (...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__))#endif /* _LINUX_RHASHTABLE_TYPES_H */
lib/rhashtable.c +17 -11
@@ -130,7 +130,8 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht,if (ntbl)return ntbl; - ntbl =kzalloc (PAGE_SIZE ,GFP_ATOMIC ); + ntbl =alloc_hooks_tag (ht->alloc_tag, +kmalloc_noprof (PAGE_SIZE ,GFP_ATOMIC |__GFP_ZERO));if (ntbl && leaf) {for (i =0 ; i <PAGE_SIZE /sizeof (ntbl[0 ]); i++)
@@ -157,7 +158,8 @@ static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, size =sizeof (*tbl) +sizeof (tbl->buckets[0 ]); - tbl =kzalloc (size, gfp); + tbl =alloc_hooks_tag (ht->alloc_tag, +kmalloc_noprof (size, gfp|__GFP_ZERO));if (!tbl)return NULL ;
@@ -181,7 +183,9 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht,int i;static struct lock_class_key __key; - tbl =kvzalloc (struct_size (tbl, buckets, nbuckets), gfp); + tbl =alloc_hooks_tag (ht->alloc_tag, +kvmalloc_node_noprof (struct_size (tbl, buckets, nbuckets), + gfp|__GFP_ZERO,NUMA_NO_NODE )); size = nbuckets;
@@ -975,7 +979,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) }/** -* rhashtable_init - initialize a new hash table +* rhashtable_init_noprof - initialize a new hash table * @ht: hash table to be initialized * @params: configuration parameters *
@@ -1016,7 +1020,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed)* .obj_hashfn = my_hash_fn, * }; */ -int rhashtable_init (struct rhashtable *ht, +int rhashtable_init_noprof (struct rhashtable *ht,const struct rhashtable_params *params) {struct bucket_table *tbl;
@@ -1031,6 +1035,8 @@ int rhashtable_init(struct rhashtable *ht,spin_lock_init (&ht->lock);memcpy (&ht->p, params,sizeof (*params)); +alloc_tag_record (ht->alloc_tag); +if (params->min_size) ht->p.min_size =roundup_pow_of_two (params->min_size);
@@ -1076,26 +1082,26 @@ int rhashtable_init(struct rhashtable *ht,return 0 ; } -EXPORT_SYMBOL_GPL (rhashtable_init); +EXPORT_SYMBOL_GPL (rhashtable_init_noprof);/** -* rhltable_init - initialize a new hash list table +* rhltable_init_noprof - initialize a new hash list table * @hlt: hash list table to be initialized * @params: configuration parameters * * Initializes a new hash list table. * -* See documentation for rhashtable_init. +* See documentation for rhashtable_init_noprof. */ -int rhltable_init (struct rhltable *hlt,const struct rhashtable_params *params) +int rhltable_init_noprof (struct rhltable *hlt,const struct rhashtable_params *params) {int err; - err =rhashtable_init (&hlt->ht, params); + err =rhashtable_init_noprof (&hlt->ht, params); hlt->ht.rhlist = true;return err; } -EXPORT_SYMBOL_GPL (rhltable_init); +EXPORT_SYMBOL_GPL (rhltable_init_noprof); static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj, void (*free_fn)(void *ptr, void *arg),
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 31/36 lib: add memory allocations report in show_mem()
In reply to Suren BaghdasaryanInclude allocations in show_mem reports.
--- include/linux/alloc_tag.h | 7 +++++++ include/linux/codetag.h | 1 + lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ lib/codetag.c | 5 +++++ mm/show_mem.c | 26 ++++++++++++++++++++++++++ 5 files changed, 77 insertions(+)
include/linux/alloc_tag.h +7 -0
@@ -30,6 +30,13 @@ struct alloc_tag {#ifdef CONFIG_MEM_ALLOC_PROFILING +struct codetag_bytes { +struct codetag *ct; +s64 bytes; +}; + +size_t alloc_tag_top_users (struct codetag_bytes *tags,size_t count,bool can_sleep); +static inline struct alloc_tag *ct_to_alloc_tag (struct codetag *ct) { return container_of(ct, struct alloc_tag, ct);
include/linux/codetag.h +1 -0
@@ -61,6 +61,7 @@ struct codetag_iterator { }void codetag_lock_module_list (struct codetag_type *cttype,bool lock); +bool codetag_trylock_module_list (struct codetag_type *cttype);struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype);struct codetag *codetag_next_ct (struct codetag_iterator *iter);
lib/alloc_tag.c +38 -0
@@ -86,6 +86,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +size_t alloc_tag_top_users (struct codetag_bytes *tags,size_t count,bool can_sleep) +{ +struct codetag_iterator iter; +struct codetag *ct; +struct codetag_bytes n; +unsigned int i, nr =0 ; + +if (can_sleep) +codetag_lock_module_list (alloc_tag_cttype, true); +else if (!codetag_trylock_module_list (alloc_tag_cttype)) +return 0 ; + + iter =codetag_get_ct_iter (alloc_tag_cttype); +while ((ct =codetag_next_ct (&iter))) { +struct alloc_tag_counters counter =alloc_tag_read (ct_to_alloc_tag (ct)); + + n.ct = ct; + n.bytes = counter.bytes; + +for (i =0 ; i < nr; i++) +if (n.bytes > tags[i].bytes) +break ; + +if (i < count) { + nr -= nr == count; +memmove (&tags[i +1 ], + &tags[i], +sizeof (tags[0 ]) * (nr - i)); + nr++; + tags[i] = n; + } + } + +codetag_lock_module_list (alloc_tag_cttype, false); + +return nr; +} +static void __init procfs_init (void ) {proc_create_seq ("allocinfo" ,0444 ,NULL , &allocinfo_seq_op);
lib/codetag.c +5 -0
@@ -36,6 +36,11 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock)up_read (&cttype->mod_lock); } +bool codetag_trylock_module_list (struct codetag_type *cttype) +{ +return down_read_trylock (&cttype->mod_lock) !=0 ; +} +struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype) {struct codetag_iterator iter = {
mm/show_mem.c +26 -0
@@ -423,4 +423,30 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)#ifdef CONFIG_MEMORY_FAILURE printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { +struct codetag_bytes tags[10 ]; +size_t i, nr; + + nr =alloc_tag_top_users (tags,ARRAY_SIZE (tags), false); +if (nr) { +printk (KERN_NOTICE "Memory allocations:\n" ); +for (i =0 ; i < nr; i++) { +struct codetag *ct = tags[i].ct; +struct alloc_tag *tag =ct_to_alloc_tag (ct); +struct alloc_tag_counters counter =alloc_tag_read (tag); + +/* Same as alloc_tag_to_text() but w/o intermediate buffer */ +if (ct->modname) +printk (KERN_NOTICE "%12lli %8llu %s:%u [%s] func:%s\n" , + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->modname, ct->function); +else +printk (KERN_NOTICE "%12lli %8llu %s:%u func:%s\n" , + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->function); + } + } + } +#endif }
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Re: [PATCH v4 31/36] lib: add memory allocations report in show_mem()
Quoted 4 lines · Suren Baghdasaryan
Include allocations in
show_memreports.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Nit: there's pr_notice() that's shorter than printk(KERN_NOTICE
Quoted 138 lines
--- include/linux/alloc_tag.h | 7 +++++++ include/linux/codetag.h | 1 + lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ lib/codetag.c | 5 +++++ mm/show_mem.c | 26 ++++++++++++++++++++++++++ 5 files changed, 77 insertions(+)include/linux/alloc_tag.h +7
-0@@ -30,6 +30,13 @@ struct alloc_tag {#ifdef CONFIG_MEM_ALLOC_PROFILING +struct codetag_bytes { +struct codetag *ct; +s64 bytes; +}; + +size_t alloc_tag_top_users (struct codetag_bytes *tags,size_t count,bool can_sleep); +static inline struct alloc_tag *ct_to_alloc_tag (struct codetag *ct) { return container_of(ct, struct alloc_tag, ct);include/linux/codetag.h +1
-0@@ -61,6 +61,7 @@ struct codetag_iterator { }void codetag_lock_module_list (struct codetag_type *cttype,bool lock); +bool codetag_trylock_module_list (struct codetag_type *cttype);struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype);struct codetag *codetag_next_ct (struct codetag_iterator *iter);lib/alloc_tag.c +38
-0@@ -86,6 +86,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +size_t alloc_tag_top_users (struct codetag_bytes *tags,size_t count,bool can_sleep) +{ +struct codetag_iterator iter; +struct codetag *ct; +struct codetag_bytes n; +unsigned int i, nr =0 ; + +if (can_sleep) +codetag_lock_module_list (alloc_tag_cttype, true); +else if (!codetag_trylock_module_list (alloc_tag_cttype)) +return 0 ; + + iter =codetag_get_ct_iter (alloc_tag_cttype); +while ((ct =codetag_next_ct (&iter))) { +struct alloc_tag_counters counter =alloc_tag_read (ct_to_alloc_tag (ct)); + + n.ct = ct; + n.bytes = counter.bytes; + +for (i =0 ; i < nr; i++) +if (n.bytes > tags[i].bytes) +break ; + +if (i < count) { + nr -= nr == count; +memmove (&tags[i +1 ], + &tags[i], +sizeof (tags[0 ]) * (nr - i)); + nr++; + tags[i] = n; + } + } + +codetag_lock_module_list (alloc_tag_cttype, false); + +return nr; +} +static void __init procfs_init (void ) {proc_create_seq ("allocinfo" ,0444 ,NULL , &allocinfo_seq_op);lib/codetag.c +5
-0@@ -36,6 +36,11 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock)up_read (&cttype->mod_lock); } +bool codetag_trylock_module_list (struct codetag_type *cttype) +{ +return down_read_trylock (&cttype->mod_lock) !=0 ; +} +struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype) {struct codetag_iterator iter = {mm/show_mem.c +26
-0@@ -423,4 +423,30 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)#ifdef CONFIG_MEMORY_FAILURE printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { +struct codetag_bytes tags[10 ]; +size_t i, nr; + + nr =alloc_tag_top_users (tags,ARRAY_SIZE (tags), false); +if (nr) { +printk (KERN_NOTICE "Memory allocations:\n" ); +for (i =0 ; i < nr; i++) { +struct codetag *ct = tags[i].ct; +struct alloc_tag *tag =ct_to_alloc_tag (ct); +struct alloc_tag_counters counter =alloc_tag_read (tag); + +/* Same as alloc_tag_to_text() but w/o intermediate buffer */ +if (ct->modname) +printk (KERN_NOTICE "%12lli %8llu %s:%u [%s] func:%s\n" , + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->modname, ct->function); +else +printk (KERN_NOTICE "%12lli %8llu %s:%u func:%s\n" , + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->function); + } + } + } +#endif }
Suren Baghdasaryan
maintainerRe: [PATCH v4 31/36] lib: add memory allocations report in show_mem()
Quoted 10 lines · Vlastimil Babka
Quoted 4 lines · Suren Baghdasaryan
Include allocations in
show_memreports.Signed-off-by Kent Overstreet <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Nit: there's
pr_notice()that's shorter than printk(KERN_NOTICE
I used printk() since other parts of show_mem() used it but I can change if that's preferable.
Quoted 143 lines
Quoted 129 lines
--- include/linux/alloc_tag.h | 7 +++++++ include/linux/codetag.h | 1 + lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ lib/codetag.c | 5 +++++ mm/show_mem.c | 26 ++++++++++++++++++++++++++ 5 files changed, 77 insertions(+)include/linux/alloc_tag.h +7
-0@@ -30,6 +30,13 @@ struct alloc_tag {#ifdef CONFIG_MEM_ALLOC_PROFILING +struct codetag_bytes { +struct codetag *ct; +s64 bytes; +}; + +size_t alloc_tag_top_users (struct codetag_bytes *tags,size_t count,bool can_sleep); +static inline struct alloc_tag *ct_to_alloc_tag (struct codetag *ct) { return container_of(ct, struct alloc_tag, ct);include/linux/codetag.h +1
-0@@ -61,6 +61,7 @@ struct codetag_iterator { }void codetag_lock_module_list (struct codetag_type *cttype,bool lock); +bool codetag_trylock_module_list (struct codetag_type *cttype);struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype);struct codetag *codetag_next_ct (struct codetag_iterator *iter);lib/alloc_tag.c +38
-0@@ -86,6 +86,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +size_t alloc_tag_top_users (struct codetag_bytes *tags,size_t count,bool can_sleep) +{ +struct codetag_iterator iter; +struct codetag *ct; +struct codetag_bytes n; +unsigned int i, nr =0 ; + +if (can_sleep) +codetag_lock_module_list (alloc_tag_cttype, true); +else if (!codetag_trylock_module_list (alloc_tag_cttype)) +return 0 ; + + iter =codetag_get_ct_iter (alloc_tag_cttype); +while ((ct =codetag_next_ct (&iter))) { +struct alloc_tag_counters counter =alloc_tag_read (ct_to_alloc_tag (ct)); + + n.ct = ct; + n.bytes = counter.bytes; + +for (i =0 ; i < nr; i++) +if (n.bytes > tags[i].bytes) +break ; + +if (i < count) { + nr -= nr == count; +memmove (&tags[i +1 ], + &tags[i], +sizeof (tags[0 ]) * (nr - i)); + nr++; + tags[i] = n; + } + } + +codetag_lock_module_list (alloc_tag_cttype, false); + +return nr; +} +static void __init procfs_init (void ) {proc_create_seq ("allocinfo" ,0444 ,NULL , &allocinfo_seq_op);lib/codetag.c +5
-0@@ -36,6 +36,11 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock)up_read (&cttype->mod_lock); } +bool codetag_trylock_module_list (struct codetag_type *cttype) +{ +return down_read_trylock (&cttype->mod_lock) !=0 ; +} +struct codetag_iterator codetag_get_ct_iter (struct codetag_type *cttype) {struct codetag_iterator iter = {mm/show_mem.c +17
-0@@ -423,4 +423,30 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)#ifdef CONFIG_MEMORY_FAILURE printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { +struct codetag_bytes tags[10 ]; +size_t i, nr; + + nr =alloc_tag_top_users (tags,ARRAY_SIZE (tags), false); +if (nr) { +printk (KERN_NOTICE "Memory allocations:\n" ); +for (i =0 ; i < nr; i++) { +struct codetag *ct = tags[i].ct; +struct alloc_tag *tag =ct_to_alloc_tag (ct); +struct alloc_tag_counters counter =alloc_tag_read (tag); + +/* Same as alloc_tag_to_text() but w/o intermediate buffer */ +if (ct->modname) + printk(KERN_NOTICE "%12lli %8llu %s:%u [%s] func:%s\n", + counter.bytes, counter.calls, ct->filename,mm/show_mem.c void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) · line 452
Quoted 10 lines
mm/show_mem.c +9
-0+ ct->lineno, ct->modname, ct->function); + else + printk(KERN_NOTICE "%12lli %8llu %s:%u func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->function); + } + } + } +#endif }--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerPATCH v4 32/36 codetag: debug: skip objext checking when it's for objext itself
In reply to Suren Baghdasaryanobjext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty.
--- include/linux/alloc_tag.h | 26 ++++++++++++++++++++++++++ mm/slub.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
include/linux/alloc_tag.h +26 -0
@@ -28,6 +28,27 @@ struct alloc_tag {struct alloc_tag_counters __percpu *counters; }__aligned(8) ; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY ((void *)1) + +static inline bool is_codetag_empty (union codetag_ref *ref) +{ +return ref->ct ==CODETAG_EMPTY ; +} + +static inline void set_codetag_empty (union codetag_ref *ref) +{ +if (ref) + ref->ct =CODETAG_EMPTY ; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty (union codetag_ref *ref) {return false; } + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ +#ifdef CONFIG_MEM_ALLOC_PROFILING struct codetag_bytes {
@@ -91,6 +112,11 @@ static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes)if (!ref || !ref->ct)return ; +if (is_codetag_empty (ref)) { + ref->ct =NULL ; +return ; + } + tag =ct_to_alloc_tag (ref->ct);this_cpu_sub (tag->counters->bytes, bytes);
mm/slub.c +33 -0
@@ -1883,6 +1883,30 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s)#ifdef CONFIG_SLAB_OBJ_EXT +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty (struct slabobj_ext *obj_exts) +{ +struct slabobj_ext *slab_exts; +struct slab *obj_exts_slab; + + obj_exts_slab =virt_to_slab (obj_exts); + slab_exts =slab_obj_exts (obj_exts_slab); +if (slab_exts) { +unsigned int offs =obj_to_index (obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); +/* codetag should be NULL */ +WARN_ON (slab_exts[offs].ref.ct); +set_codetag_empty (&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty (struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ +/* * The allocated objcg pointers array is not accounted directly. * Moreover, it should not come from DMA buffer and is not readily
@@ -1923,6 +1947,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,* assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ +mark_objexts_empty (vec);kfree (vec);return 0 ; }
@@ -1939,6 +1964,14 @@ static inline void free_slab_obj_exts(struct slab *slab)if (!obj_exts)return ; +/* +* obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its +* corresponding extension will be NULL. alloc_tag_sub() will throw a +* warning if slab has extensions but the extension of an object is +* NULL, therefore replace NULL with CODETAG_EMPTY to indicate that +* the extension for obj_exts is expected to be NULL. +*/ +mark_objexts_empty (obj_exts);kfree (obj_exts); slab->obj_exts =0 ; }
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 33/36 codetag: debug: mark codetags for reserved pages as empty
In reply to Suren BaghdasaryanTo avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing.
--- include/linux/alloc_tag.h | 1 + include/linux/mm.h | 9 +++++++++ include/linux/pgalloc_tag.h | 2 ++ mm/mm_init.c | 12 +++++++++++- 4 files changed, 23 insertions(+), 1 deletion(-)
include/linux/alloc_tag.h +1 -0
@@ -46,6 +46,7 @@ static inline void set_codetag_empty(union codetag_ref *ref)#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline bool is_codetag_empty (union codetag_ref *ref) {return false; } +static inline void set_codetag_empty (union codetag_ref *ref) {}#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */
include/linux/mm.h +9 -0
@@ -5,6 +5,7 @@#include <linux/errno.h> #include <linux/mmdebug.h> #include <linux/gfp.h> +#include <linux/pgalloc_tag.h> #include <linux/bug.h> #include <linux/list.h> #include <linux/mmzone.h>
@@ -3112,6 +3113,14 @@ extern void reserve_bootmem_region(phys_addr_t start,/* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page (struct page *page) { +if (mem_alloc_profiling_enabled ()) { +union codetag_ref *ref =get_page_tag_ref (page); + +if (ref) { +set_codetag_empty (ref); +put_page_tag_ref (ref); + } + }ClearPageReserved (page);init_page_count (page);__free_page (page);
include/linux/pgalloc_tag.h +2 -0
@@ -98,6 +98,8 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr)#else /* CONFIG_MEM_ALLOC_PROFILING */ +static inline union codetag_ref *get_page_tag_ref (struct page *page) {return NULL ; } +static inline void put_page_tag_ref (union codetag_ref *ref) {}static inline void pgalloc_tag_add (struct page *page,struct task_struct *task,unsigned int order) {}static inline void pgalloc_tag_sub (struct page *page,unsigned int order) {}
mm/mm_init.c +11 -1
@@ -2566,7 +2566,6 @@ void __init set_dma_reserve(unsigned long new_dma_reserve)void __init memblock_free_pages (struct page *page,unsigned long pfn,unsigned int order) { -if (IS_ENABLED (CONFIG_DEFERRED_STRUCT_PAGE_INIT )) {int nid =early_pfn_to_nid (pfn);
@@ -2578,6 +2577,17 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn,/* KMSAN will take care of these pages. */ return ; } + +/* pages were reserved and not allocated */ +if (mem_alloc_profiling_enabled ()) { +union codetag_ref *ref =get_page_tag_ref (page); + +if (ref) { +set_codetag_empty (ref); +put_page_tag_ref (ref); + } + } +__free_pages_core (page, order); }
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 34/36 codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations
In reply to Suren BaghdasaryanIf slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled.
Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks.
--- include/linux/memcontrol.h | 4 +++- mm/slub.c | 46 ++++++++++++++++++++++++++++++++------ 2 files changed, 42 insertions(+), 8 deletions(-)
include/linux/memcontrol.h +3 -1
@@ -365,8 +365,10 @@ enum page_memcg_data_flags {#endif /* CONFIG_MEMCG */ enum objext_flags { +/* slabobj_ext vector failed to allocate */ +OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG,/* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG <<1 ), };#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1)
mm/slub.c +39 -7
77 lines not on this page · raw message
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 35/36 MAINTAINERS: Add entries for code tagging and memory allocation profiling
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
The new code & libraries added are being maintained - mark them as such.
--- MAINTAINERS | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
MAINTAINERS +17 -0
@@ -5210,6 +5210,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan <[email protected]> +M: Kent Overstreet <[email protected]> +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott <[email protected]> M: H Hartley Sweeten <[email protected]>
@@ -14061,6 +14068,16 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan <[email protected]> +M: Kent Overstreet <[email protected]> +L: [email protected] +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski <[email protected]> L: [email protected]
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Suren Baghdasaryan
maintainerPATCH v4 36/36 memprofiling: Documentation
In reply to Suren BaghdasaryanFrom: Kent Overstreet <[email protected]>
Provide documentation for memory allocation profiling.
--- Documentation/mm/allocation-profiling.rst | 86 +++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 Documentation/mm/allocation-profiling.rst
Documentation/mm/allocation-profiling.rst new file +86 -0
@@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========================== +MEMORY ALLOCATION PROFILING +=========================== + +Low overhead (suitable for production) accounting of all memory allocations, +tracked by file and line number. + +Usage: +kconfig options: + - CONFIG_MEM_ALLOC_PROFILING + - CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + - CONFIG_MEM_ALLOC_PROFILING_DEBUG + adds warnings for allocations that weren't accounted because of a + missing annotation + +Boot parameter: + sysctl.vm.mem_profiling=1 + +sysctl: + /proc/sys/vm/mem_profiling + +Runtime info: + /proc/allocinfo + +Example output: + root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec + 2.8M 22648 fs/kernfs/dir.c:615 func:__kernfs_new_node + 3.8M 953 mm/memory.c:4214 func:alloc_anon_folio + 4.0M 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 4.1M 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 6.0M 1532 mm/filemap.c:1919 func:__filemap_get_folio + 8.8M 2785 kernel/fork.c:307 func:alloc_thread_stack_node + 13M 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 14M 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 15M 3656 mm/readahead.c:247 func:page_cache_ra_unbounded + 55M 4887 mm/slub.c:2259 func:alloc_slab_page + 122M 31168 mm/page_ext.c:270 func:alloc_page_ext +=================== +Theory of operation +=================== + +Memory allocation profiling builds off of code tagging, which is a library for +declaring static structs (that typcially describe a file and line number in +some way, hence code tagging) and then finding and operating on them at runtime +- i.e. iterating over them to print them in debugfs/procfs. + +To add accounting for an allocation call, we replace it with a macro +invocation, alloc_hooks(), that + - declares a code tag + - stashes a pointer to it in task_struct + - calls the real allocation function + - and finally, restores the task_struct alloc tag pointer to its previous value. + +This allows for alloc_hooks() calls to be nested, with the most recent one +taking effect. This is important for allocations internal to the mm/ code that +do not properly belong to the outer allocation context and should be counted +separately: for example, slab object extension vectors, or when the slab +allocates pages from the page allocator. + +Thus, proper usage requires determining which function in an allocation call +stack should be tagged. There are many helper functions that essentially wrap +e.g. kmalloc() and do a little more work, then are called in multiple places; +we'll generally want the accounting to happen in the callers of these helpers, +not in the helpers themselves. + +To fix up a given helper, for example foo(), do the following: + - switch its allocation call to the _noprof() version, e.g. kmalloc_noprof() + - rename it to foo_noprof() + - define a macro version of foo() like so: + #define foo(...) alloc_hooks(foo_noprof(__VA_ARGS__)) + +It's also possible to stash a pointer to an alloc tag in your own data structures. + +Do this when you're implementing a generic data structure that does allocations +"on behalf of" some other code - for example, the rhashtable code. This way, +instead of seeing a large line in /proc/allocinfo for rhashtable.c, we can +break it out by rhashtable type. + +To do so: + - Hook your data structure's init function, like any other allocation function + - Within your init function, use the convenience macro alloc_tag_record() to + record alloc tag in your data structure. + - Then, use the following form for your allocations: + alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...))
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Quoted 51 lines · Suren Baghdasaryan
Overview:
Low overhead [1] per-callsite memory allocation profiling. Not just for debug kernels, overhead low enough to be deployed in production.Example output:
root@moria-kvm:~# sort -rn /proc/allocinfo 127664128 31168 mm/page_ext.c:270 func:alloc_page_ext 56373248 4737 mm/slub.c:2259 func:alloc_slab_page 14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded 14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash 13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs 11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio 9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node 4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable 4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start 3940352 962 mm/memory.c:4214 func:alloc_anon_folio 2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node ...Since v3:
- Dropped patch changingstring_get_size()[2] as not needed
- Dropped patch modifying xfs allocators [3] as non needed, per Dave Chinner
- Added Reviewed-by, per Kees Cook
- Movedprepare_slab_obj_exts_hook()andalloc_slab_obj_exts()where they are used, per Vlastimil Babka
- FixedSLAB_NO_OBJ_EXTdefinition to use unused bit, per Vlastimil Babka
- Refactored patch [4] into other patches, per Vlastimil Babka
- Replacedsnprintf()withseq_buf_printf(), per Kees Cook
- Changed output to report bytes, per Andrew Morton and Pasha Tatashin
- Changed output to report [module] only for loadable modules, per Vlastimil Babka
- Movedmem_alloc_profiling_enabled()check earlier, per Vlastimil Babka
- Changed the code to handle page splitting to be more understandable, per Vlastimil Babka
- Movedalloc_tagging_slab_free_hook(),mark_objexts_empty(),mark_failed_objexts_alloc()andhandle_failed_objexts_alloc(), per Vlastimil Babka
- Fixed loss of__alloc_size(1, 2)in kvmalloc functions,
per Vlastimil Babka
- Refactored the code inshow_mem()to avoid memory allocations, per Michal Hocko
- Changed to trylock inshow_mem()to avoid blocking in atomic context, per Tetsuo Handa
- Added mm mailing list intoMAINTAINERS, per Kees Cook
- Added base commit SHA, per Andy Shevchenko
- Added a patch with documentation, per Jani Nikula
- Fixed 0day bugs
- Added benchmark results [5], per Steven Rostedt
- Rebased over Linux 6.8-rc5
Quoted 4 lines
Items not yet addressed:
- Anearly_bootoption to prevent pageext overhead. We are looking into ways for using the same sysctr instead of adding additional early boot parameter.
I have reviewed the parts that integrate the tracking with page and slab allocators, and besides some details to improve it seems ok to me. The early boot option seems coming so that might eventually be suitable for build-time enablement in a distro kernel.
The macros (and their potential spread to upper layers to keep the information useful enough) are of course ugly, but guess it can't be currently helped and I'm unable to decide whether it's worth it or not. That's up to those providing their success stories I guess. If there's at least a path ahead to replace that part with compiler support in the future, great. So I'm not against merging this. BTW, do we know Linus's opinion on the macros approach?
Suren Baghdasaryan
maintainerQuoted 58 lines · Vlastimil Babka
Quoted 51 lines · Suren Baghdasaryan
Overview:
Low overhead [1] per-callsite memory allocation profiling. Not just for debug kernels, overhead low enough to be deployed in production.Example output:
root@moria-kvm:~# sort -rn /proc/allocinfo 127664128 31168 mm/page_ext.c:270 func:alloc_page_ext 56373248 4737 mm/slub.c:2259 func:alloc_slab_page 14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded 14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash 13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs 11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio 9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node 4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable 4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start 3940352 962 mm/memory.c:4214 func:alloc_anon_folio 2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node ...Since v3:
- Dropped patch changingstring_get_size()[2] as not needed
- Dropped patch modifying xfs allocators [3] as non needed, per Dave Chinner
- Added Reviewed-by, per Kees Cook
- Movedprepare_slab_obj_exts_hook()andalloc_slab_obj_exts()where they are used, per Vlastimil Babka
- FixedSLAB_NO_OBJ_EXTdefinition to use unused bit, per Vlastimil Babka
- Refactored patch [4] into other patches, per Vlastimil Babka
- Replacedsnprintf()withseq_buf_printf(), per Kees Cook
- Changed output to report bytes, per Andrew Morton and Pasha Tatashin
- Changed output to report [module] only for loadable modules, per Vlastimil Babka
- Movedmem_alloc_profiling_enabled()check earlier, per Vlastimil Babka
- Changed the code to handle page splitting to be more understandable, per Vlastimil Babka
- Movedalloc_tagging_slab_free_hook(),mark_objexts_empty(),mark_failed_objexts_alloc()andhandle_failed_objexts_alloc(), per Vlastimil Babka
- Fixed loss of__alloc_size(1, 2)in kvmalloc functions,
per Vlastimil Babka
- Refactored the code inshow_mem()to avoid memory allocations, per Michal Hocko
- Changed to trylock inshow_mem()to avoid blocking in atomic context, per Tetsuo Handa
- Added mm mailing list intoMAINTAINERS, per Kees Cook
- Added base commit SHA, per Andy Shevchenko
- Added a patch with documentation, per Jani Nikula
- Fixed 0day bugs
- Added benchmark results [5], per Steven Rostedt
- Rebased over Linux 6.8-rc5Quoted 4 lines
Items not yet addressed:
- Anearly_bootoption to prevent pageext overhead. We are looking into ways for using the same sysctr instead of adding additional early boot parameter.
Quoted 4 lines
I have reviewed the parts that integrate the tracking with page and slab allocators, and besides some details to improve it seems ok to me. The early boot option seems coming so that might eventually be suitable for build-time enablement in a distro kernel.
Thanks for reviewing Vlastimil!
Quoted 8 lines
The macros (and their potential spread to upper layers to keep the information useful enough) are of course ugly, but guess it can't be currently helped and I'm unable to decide whether it's worth it or not. That's up to those providing their success stories I guess. If there's at least a path ahead to replace that part with compiler support in the future, great. So I'm not against merging this. BTW, do we know Linus's opinion on the macros approach?
We haven't run it by Linus specifically but hopefully we will see a comment from him on the mailing list at some point.
Quoted 4 lines
--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].