Skip to content

RFC Code tagging framework and applications

Suren Baghdasaryan io-uring linux-arch linux-bcache linux-iommu linux-mm linux-modules lkml llvm xen-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest30 patches 107 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/30kernel/module: move find_kallsyms_symbol_value declarationinclude/linux/module.h kernel/module/internal.h+1 -10 replies
2/30lib/string_helpers: Drop space in string_get_size's outputlib/string_helpers.c+1 -20 replies
3/30Lazy percpu countersinclude/linux/lazy-percpu-counter.h lib/Kconfig lib/Makefile +1 more+213 -08 replies
4/30scripts/kallysms: Always include __start and __stop symbolsscripts/kallsyms.c+13 -00 replies
5/30lib: code tagging frameworkinclude/linux/codetag.h lib/Kconfig.debug lib/Makefile +1 more+275 -00 replies
6/30lib: code tagging module supportinclude/linux/codetag.h kernel/module/main.c lib/codetag.c+66 -10 replies
7/30lib: add support for allocation tagginginclude/asm-generic/codetag.lds.h include/asm-generic/vmlinux.lds.h include/linux/alloc_tag.h +4 more+255 -00 replies
8/30lib: introduce page allocation tagginginclude/linux/pgalloc_tag.h lib/Kconfig.debug lib/Makefile +2 more+68 -00 replies
9/30change alloc_pages name in dma_map_ops to avoid name conflictsarch/x86/kernel/amd_gart_64.c drivers/iommu/dma-iommu.c drivers/xen/grant-dma-ops.c +3 more+7 -70 replies
10/30mm: enable page allocation tagging for __get_free_pages and alloc_pagesinclude/linux/gfp.h include/linux/page_ext.h include/linux/pgalloc_tag.h +2 more+56 -96 replies
11/30mm: introduce slabobj_ext to support slab object extensionsinclude/linux/memcontrol.h init/Kconfig mm/kfence/core.c +3 more+131 -752 replies
12/30mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creationinclude/linux/gfp_types.h+10 -20 replies
13/30mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creationinclude/linux/slab.h mm/slab.c mm/slub.c+11 -30 replies
14/30mm: prevent slabobj_ext allocations for slabobj_ext and kmem_cache objectsmm/memcontrol.c mm/slab.h+8 -02 replies
15/30lib: introduce slab allocation tagginginclude/linux/memcontrol.h include/linux/slab.h include/linux/slab_def.h +3 more+77 -30 replies
16/30mm: enable slab allocation tagging for kmalloc and friendsinclude/linux/slab.h mm/slab.c mm/slab_common.c +2 more+75 -501 reply
17/30lib/string.c: strsep_no_empty()include/linux/string.h lib/string.c+20 -00 replies
18/30codetag: add codetag query helper functionsinclude/linux/codetag.h lib/codetag.c+162 -00 replies
19/30move stack capture functionality into a separate function for reuseinclude/linux/sched.h include/linux/stackdepot.h lib/stackdepot.c +1 more+77 -520 replies
20/30lib: introduce support for storing code tag contextinclude/linux/codetag.h include/linux/codetag_ctx.h lib/codetag.c+231 -10 replies
21/30lib: implement context capture support for page and slab allocatorsinclude/linux/alloc_tag.h include/linux/codetag.h lib/Kconfig.debug +2 more+273 -180 replies
22/30Code tagging based fault injectioninclude/asm-generic/codetag.lds.h include/linux/dynamic_fault.h include/linux/slab.h +3 more+463 -26 replies
23/30timekeeping: Add a missing includeinclude/linux/timekeeping.h+1 -00 replies
24/30wait: Clean up waitqueue_entry initializationinclude/linux/sbitmap.h include/linux/wait.h include/linux/wait_bit.h +1 more+27 -470 replies
25/30lib/time_stats: New library for statistics on eventsinclude/linux/time_stats.h lib/Kconfig lib/Makefile +1 more+284 -00 replies
26/30bcache: Convert to lib/time_statsdrivers/md/bcache/Kconfig drivers/md/bcache/bcache.h drivers/md/bcache/bset.c +6 more+47 -1090 replies
27/30Code tagging based latency trackinginclude/asm-generic/codetag.lds.h include/linux/codetag_time_stats.h include/linux/io_uring_types.h +5 more+233 -612 replies
28/30Improved symbolic error namesinclude/asm-generic/codetag.lds.h include/linux/err.h include/linux/errname.h +1 more+156 -22 replies
29/30dyndbg: Convert to code tagginginclude/asm-generic/codetag.lds.h include/asm-generic/vmlinux.lds.h include/linux/dynamic_debug.h +3 more+158 -3400 replies
30/30MAINTAINERS: Add entries for code tagging & relatedMAINTAINERS+34 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/30] Code tagging framework and applications

===========================
Code tagging framework
===========================
Code tag is a structure identifying a specific location in the source code which is generated at compile time and can be embedded in an application- specific structure. Several applications of code tagging are included in this RFC, such as memory allocation tracking, dynamic fault injection, latency tracking and improved error code reporting.
Basically, it takes the old trick of "define a special elf section for objects of a given type so that we can iterate over them at runtime" and creates a proper library for it.

===========================
Memory allocation tracking
===========================
The goal for using codetags for memory allocation tracking is to minimize performance and memory overhead. By recording only the call count and allocation size, the required operations are kept at the minimum while collecting statistics for every allocation in the codebase. With that information, if users are interested in mode detailed context for a specific allocation, they can enable more in-depth context tracking, which includes capturing the pid, tgid, task name, allocation size, timestamp and call stack for every allocation at the specified code location.
Memory allocation tracking is implemented in two parts:

part1: instruments page and slab allocators to record call count and total memory allocated at every allocation in the source code. Every time an allocation is performed by an instrumented allocator, the codetag at that location increments its call and size counters. Every time the memory is freed these counters are decremented. To decrement the counters upon free, allocated object needs a reference to its codetag. Page allocators use page_ext to record this reference while slab allocators use memcg_data of the slab page.
The data is exposed to the user space via a read-only debugfs file called alloc_tags.

Usage example:

$ sort -hr /sys/kernel/debug/alloc_tags|head
  153MiB     8599 mm/slub.c:1826 module:slub func:alloc_slab_page
 6.08MiB      49 mm/slab_common.c:950 module:slab_common func:_kmalloc_order
 5.09MiB     6335 mm/memcontrol.c:2814 module:memcontrol func:alloc_slab_obj_exts
 4.54MiB      78 mm/page_alloc.c:5777 module:page_alloc func:alloc_pages_exact
 1.32MiB      338 include/asm-generic/pgalloc.h:63 module:pgtable func:__pte_alloc_one
 1.16MiB      603 fs/xfs/xfs_log_priv.h:700 module:xfs func:xlog_kvmalloc
 1.00MiB      256 mm/swap_cgroup.c:48 module:swap_cgroup func:swap_cgroup_prepare
  734KiB     5380 fs/xfs/kmem.c:20 module:xfs func:kmem_alloc
  640KiB      160 kernel/rcu/tree.c:3184 module:tree func:fill_page_cache_func
  640KiB      160 drivers/char/virtio_console.c:452 module:virtio_console func:alloc_buf

part2: adds support for the user to select a specific code location to capture allocation context. A new debugfs file called alloc_tags.ctx is used to select which code location should capture allocation context and to read captured context information.

Usage example:

$ cd /sys/kernel/debug/
$ echo "file include/asm-generic/pgalloc.h line 63 enable" > alloc_tags.ctx
$ cat alloc_tags.ctx
  920KiB      230 include/asm-generic/pgalloc.h:63 module:pgtable func:__pte_alloc_one
    size: 4096
    pid: 1474
    tgid: 1474
    comm: bash
    ts: 175332940994
    call stack:
         pte_alloc_one+0xfe/0x130
         __pte_alloc+0x22/0xb0
         copy_page_range+0x842/0x1640
         dup_mm+0x42d/0x580
         copy_process+0xfb1/0x1ac0
         kernel_clone+0x92/0x3e0
         __do_sys_clone+0x66/0x90
         do_syscall_64+0x38/0x90
         entry_SYSCALL_64_after_hwframe+0x63/0xcd
...

NOTE: slab allocation tracking is not yet stable and has a leak that shows up in long-running tests. We are working on fixing it and posting the RFC early to collect some feedback and to have a reference code in public before presenting the idea at LPC2022.

===========================
Dynamic fault injection
===========================
Dynamic fault injection lets you do fault injection with a single call to dynamic_fault(), with a debugfs interface similar to dynamic_debug.

Calls to dynamic_fault are listed in debugfs and can be enabled at runtime (oneshot mode or a defined frequency are also available). This patch also uses the memory allocation wrapper macros introduced by the memory allocation tracking patches to add distinct fault injection points for every memory allocation in the kernel.

Example fault injection points, after hooking memory allocation paths:

fs/xfs/libxfs/xfs_iext_tree.c:606 module:xfs func:xfs_iext_realloc_rootclass:memory disabled " fs/xfs/libxfs/xfs_inode_fork.c:503 module:xfs func:xfs_idata_reallocclass:memory disabled " fs/xfs/libxfs/xfs_inode_fork.c:399 module:xfs func:xfs_iroot_reallocclass:memory disabled " fs/xfs/xfs_buf.c:373 module:xfs func:xfs_buf_alloc_pagesclass:memory disabled " fs/xfs/xfs_iops.c:497 module:xfs func:xfs_vn_get_linkclass:memory disabled " fs/xfs/xfs_mount.c:85 module:xfs func:xfs_uuid_mountclass:memory disabled "

===========================
Latency tracking
===========================
This lets you instrument code for measuring latency with just two calls to code_tag_time_stats_start() and code_tag_time_stats_finish(), and makes statistics available in debugfs on a per-callsite basis.

Recorded statistics include total count, frequency/rate, average duration, max duration, and event duration quantiles.

Additionally, this patch instruments prepare_to_wait() and finish_wait().

Example output:

fs/xfs/xfs_extent_busy.c:589 module:xfs func:xfs_extent_busy_flush
count: 61
rate: 0/sec
frequency: 19 sec
avg duration: 632 us
max duration: 2 ms
quantiles (us): 274 288 288 296 296 296 296 336 336 336 336 336 336 336 336

===========================
Improved error codes
===========================
Ever waste hours trying to figure out which line of code from some obscure module is returning you -EINVAL and nothing else?

What if we had... more error codes?

This patch adds ERR(), which returns a unique error code that is related to the error code that passed to it: the original error code can be recovered with error_class(), and errname() (as well as %pE) returns an error string that includes the file and line number of the ERR() call.

Example output:

VFS: Cannot open root device "sda" or unknown-block(8,0): error -EINVAL at fs/ext4/super.c:4387

===========================
Dynamic debug conversion to code tagging
===========================
There are several open coded implementations of the "define a special elf section for objects and iterate" technique that should be converted to code tagging. This series just converts dynamic debug; there are others (multiple in ftrace, in particular) that should also be converted.

===========================

The patchset applies cleanly over Linux 6.0-rc3
The tree for testing is published at:
https://github.com/surenbaghdasaryan/linux/tree/alloc_tags_rfc

The structure of the patchset is:
- code tagging framework (patches 1-6)
- page allocation tracking (patches 7-10)
- slab allocation tracking (patch 11-16)
- allocation context capture (patch 17-21)
- dynamic fault injection (patch 22)
- latency tracking (patch 23-27)
- improved error codes (patch 28)
- dynamic debug conversion to code tagging (patch 29)
- MAINTAINERS update (patch 30)

Next steps:
- track and fix slab allocator leak mentioned earlier;
- instrument more allocators: vmalloc, per-cpu allocations, others?

Kent Overstreet (14):
lib/string_helpers: Drop space in string_get_size's output Lazy percpu counters
scripts/kallysms: Always include __start and __stop symbols lib/string.c: strsep_no_empty()
codetag: add codetag query helper functions
Code tagging based fault injection
timekeeping: Add a missing include
wait: Clean up waitqueue_entry initialization lib/time_stats: New library for statistics on events
bcache: Convert to lib/time_stats
Code tagging based latency tracking
Improved symbolic error names
dyndbg: Convert to code tagging
MAINTAINERS: Add entries for code tagging & related

Suren Baghdasaryan (16):
kernel/module: move find_kallsyms_symbol_value declaration
lib: code tagging framework
lib: code tagging module support
lib: add support for allocation tagging
lib: introduce page allocation tagging
change alloc_pages name in dma_map_ops to avoid name conflicts
mm: enable page allocation tagging for __get_free_pages and
alloc_pages
mm: introduce slabobj_ext to support slab object extensions
mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext
creation
mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation
mm: prevent slabobj_ext allocations for slabobj_ext and kmem_cache
objects
lib: introduce slab allocation tagging
mm: enable slab allocation tagging for kmalloc and friends move stack capture functionality into a separate function for reuse
lib: introduce support for storing code tag context
lib: implement context capture support for page and slab allocators

 MAINTAINERS                         |  34 ++
 arch/x86/kernel/amd_gart_64.c       |   2 +-
 drivers/iommu/dma-iommu.c           |   2 +-
 drivers/md/bcache/Kconfig           |   1 +
 drivers/md/bcache/bcache.h          |   1 +
 drivers/md/bcache/bset.c            |   8 +-
 drivers/md/bcache/bset.h            |   1 +
 drivers/md/bcache/btree.c           |  12 +-
 drivers/md/bcache/super.c           |   3 +
 drivers/md/bcache/sysfs.c           |  43 ++-
 drivers/md/bcache/util.c            |  30 --
 drivers/md/bcache/util.h            |  57 ---
 drivers/xen/grant-dma-ops.c         |   2 +-
 drivers/xen/swiotlb-xen.c           |   2 +-
 include/asm-generic/codetag.lds.h   |  18 +
 include/asm-generic/vmlinux.lds.h   |   8 +-
 include/linux/alloc_tag.h           |  84 +++++
 include/linux/codetag.h             | 159 +++++++++
 include/linux/codetag_ctx.h         |  48 +++
 include/linux/codetag_time_stats.h  |  54 +++
 include/linux/dma-map-ops.h         |   2 +-
 include/linux/dynamic_debug.h       |  11 +-
 include/linux/dynamic_fault.h       |  79 +++++
 include/linux/err.h                 |   2 +-
 include/linux/errname.h             |  50 +++
 include/linux/gfp.h                 |  10 +-
 include/linux/gfp_types.h           |  12 +-
 include/linux/io_uring_types.h      |   2 +-
 include/linux/lazy-percpu-counter.h |  67 ++++
 include/linux/memcontrol.h          |  23 +-
 include/linux/module.h              |   1 +
 include/linux/page_ext.h            |   3 +-
 include/linux/pgalloc_tag.h         |  63 ++++
 include/linux/sbitmap.h             |   6 +-
 include/linux/sched.h               |   6 +-
 include/linux/slab.h                | 136 +++++---
 include/linux/slab_def.h            |   2 +-
 include/linux/slub_def.h            |   4 +-
 include/linux/stackdepot.h          |   3 +
 include/linux/string.h              |   1 +
 include/linux/time_stats.h          |  44 +++
 include/linux/timekeeping.h         |   1 +
 include/linux/wait.h                |  72 ++--
 include/linux/wait_bit.h            |   7 +-
 init/Kconfig                        |   5 +
 kernel/dma/mapping.c                |   4 +-
 kernel/module/internal.h            |   3 -
 kernel/module/main.c                |  27 +-
 kernel/sched/wait.c                 |  15 +-
 lib/Kconfig                         |   6 +
 lib/Kconfig.debug                   |  46 +++
 lib/Makefile                        |  10 +
 lib/alloc_tag.c                     | 391 +++++++++++++++++++++
 lib/codetag.c                       | 519 ++++++++++++++++++++++++++++
 lib/codetag_time_stats.c            | 143 ++++++++
 lib/dynamic_debug.c                 | 452 +++++++++---------------
 lib/dynamic_fault.c                 | 372 ++++++++++++++++++++
 lib/errname.c                       | 103 ++++++
 lib/lazy-percpu-counter.c           | 141 ++++++++
 lib/pgalloc_tag.c                   |  22 ++
 lib/stackdepot.c                    |  68 ++++
 lib/string.c                        |  19 +
 lib/string_helpers.c                |   3 +-
 lib/time_stats.c                    | 236 +++++++++++++
 mm/kfence/core.c                    |   2 +-
 mm/memcontrol.c                     |  62 ++--
 mm/mempolicy.c                      |   4 +-
 mm/page_alloc.c                     |  13 +-
 mm/page_ext.c                       |   6 +
 mm/page_owner.c                     |  54 +--
 mm/slab.c                           |   4 +-
 mm/slab.h                           | 125 ++++---
 mm/slab_common.c                    |  49 ++-
 mm/slob.c                           |   2 +
 mm/slub.c                           |   7 +-
 scripts/kallsyms.c                  |  13 +
 scripts/module.lds.S                |   7 +
 77 files changed, 3406 insertions(+), 703 deletions(-)
 create mode 100644 include/asm-generic/codetag.lds.h
 create mode 100644 include/linux/alloc_tag.h
 create mode 100644 include/linux/codetag.h
 create mode 100644 include/linux/codetag_ctx.h
 create mode 100644 include/linux/codetag_time_stats.h
 create mode 100644 include/linux/dynamic_fault.h
 create mode 100644 include/linux/lazy-percpu-counter.h
 create mode 100644 include/linux/pgalloc_tag.h
 create mode 100644 include/linux/time_stats.h
 create mode 100644 lib/alloc_tag.c
 create mode 100644 lib/codetag.c
 create mode 100644 lib/codetag_time_stats.c
 create mode 100644 lib/dynamic_fault.c
 create mode 100644 lib/lazy-percpu-counter.c
 create mode 100644 lib/pgalloc_tag.c
 create mode 100644 lib/time_stats.c
Signature · 1 line
2.37.2.672.g94769d06f0-goog