Skip to contentSkip to conversation

ftrace,bpf: Use single direct ops for bpf trampolines

RFC In review Jiri Olsa 5 participants 22 messages Jul 29, 2025 – Aug 13, 2025

Subsystems (4)
lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

cover ftrace,bpf: Use single direct ops for bpf trampolines

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

However having just single ftrace_ops object removes direct_call field from direct_call, which is needed by arm, so I'm not sure it's the right path forward.

Mark, Florent,
any idea how hard would it be to for arm to get rid of direct_call field?

thougts? thanks,
jirka

---
Jiri Olsa (10):
ftrace: Make alloc_and_copy_ftrace_hash direct friendly
ftrace: Add register_ftrace_direct_hash function
ftrace: Add unregister_ftrace_direct_hash function
ftrace: Add modify_ftrace_direct_hash function
ftrace: Export some of hash related functions
ftrace: Use direct hash interface in direct functions
bpf: Add trampoline ip hash table
ftrace: Factor ftrace_ops ops_func interface
bpf: Remove ftrace_ops from bpf_trampoline object Revert "ftrace: Store direct called addresses in their ops"

 include/linux/bpf.h           |   8 +-
 include/linux/ftrace.h        |  51 ++++++++++---
 kernel/bpf/trampoline.c       |  94 +++++++++++++-----------
 kernel/trace/ftrace.c         | 481 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------------------
 kernel/trace/trace.h          |   8 --
 kernel/trace/trace_selftest.c |   5 +-
 6 files changed, 395 insertions(+), 252 deletions(-)
author reviewer
#1 lore raw

RFC 1/10 ftrace: Make alloc_and_copy_ftrace_hash direct friendly

Make alloc_and_copy_ftrace_hash to copy also direct address for each hash entry.

---
 kernel/trace/ftrace.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/trace/ftrace.c +9 -2
index 4203fad56b6c..5b8f565a1258 100644
@@ -1190,7 +1190,7 @@ static void __add_hash_entry(struct ftrace_hash *hash,
 }
 
 static struct ftrace_func_entry *
-add_hash_entry(struct ftrace_hash *hash, unsigned long ip)
+add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned long direct)
 {
 	struct ftrace_func_entry *entry;
 
@@ -1199,11 +1199,18 @@ add_hash_entry(struct ftrace_hash *hash, unsigned long ip)
 		return NULL;
 
 	entry->ip = ip;
+	entry->direct = direct;
 	__add_hash_entry(hash, entry);
 
 	return entry;
 }
 
+static struct ftrace_func_entry *
+add_hash_entry(struct ftrace_hash *hash, unsigned long ip)
+{
+	return add_hash_entry_direct(hash, ip, 0);
+}
+
 static void
 free_hash_entry(struct ftrace_hash *hash,
 		  struct ftrace_func_entry *entry)
@@ -1376,7 +1383,7 @@ alloc_and_copy_ftrace_hash(int size_bits, struct ftrace_hash *hash)
 	size = 1 << hash->size_bits;
 	for (i = 0; i < size; i++) {
 		hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
-			if (add_hash_entry(new_hash, entry->ip) == NULL)
+			if (add_hash_entry_direct(new_hash, entry->ip, entry->direct) == NULL)
 				goto free_hash;
 		}
 	}
Signature · 1 line
2.50.1
author reviewer
#2 lore raw

RFC 2/10 ftrace: Add register_ftrace_direct_hash function

In reply to Jiri Olsa

Adding register_ftrace_direct_hash function that registers all entries (ip -> direct) provided in hash argument.

The difference to current register_ftrace_direct is
- hash argument that allows to register multiple ip -> direct entries at once
- we can call register_ftrace_direct_hash multiple times on the same ftrace_ops object, becase after first registration with register_ftrace_function_nolock, it uses ftrace_update_ops to update the ftrace_ops object

This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes.

---
 include/linux/ftrace.h |   7 +++
 kernel/trace/ftrace.c  | 123 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 130 insertions(+)
include/linux/ftrace.h +7 -0
index b672ca15f265..e45bcc9de53b 100644
@@ -526,6 +526,8 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
 int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
 int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
 
+int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
+
 void ftrace_stub_direct_tramp(void);
 
 #else
@@ -552,6 +554,11 @@ static inline int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned l
 	return -ENODEV;
 }
 
+int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
+{
+	return -ENODEV;
+}
+
 /*
  * This must be implemented by the architecture.
  * It is the way the ftrace direct_ops helper, when called
kernel/trace/ftrace.c +123 -0

129 lines not on this page · raw message

Signature · 1 line
2.50.1
#3 lore raw

Re: [RFC 02/10] ftrace: Add register_ftrace_direct_hash function

kernel test robot noticed the following build errors:

Hi Jiri,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on trace/for-next]
[also build test ERROR on bpf-next/net bpf-next/master bpf/master linus/master v6.16 next-20250729] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Jiri-Olsa/ftrace-Make-alloc_and_copy_ftrace_hash-direct-friendly/20250729-183143
base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link:    https://lore.kernel.org/r/20250729102813.1531457-3-jolsa%40kernel.org
patch subject: [RFC 02/10] ftrace: Add register_ftrace_direct_hash function
config: arc-randconfig-001-20250730 (https://download.01.org/0day-ci/archive/20250730/[email protected]/config)
compiler: arc-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250730/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

arc-linux-ld: drivers/i2c/i2c-core-smbus.o: in function `register_ftrace_direct_hash':

>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/i2c/i2c-core-base.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/i2c/i2c-core-slave.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/i2c/i2c-core-base.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/renesas/rcar-du/rcar_du_drv.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/rcar-du/rcar_du_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/renesas/rcar-du/rcar_du_encoder.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/rcar-du/rcar_du_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/renesas/rcar-du/rcar_du_group.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/rcar-du/rcar_du_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/renesas/rcar-du/rcar_du_kms.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/rcar-du/rcar_du_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/renesas/rcar-du/rcar_du_plane.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/rcar-du/rcar_du_crtc.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/meson/meson_plane.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_crtc.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_encoder_cvbs.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_venc.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_vclk.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_overlay.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_encoder_hdmi.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/meson/meson_encoder_dsi.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/meson/meson_drv.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/renesas/shmobile/shmob_drm_drv.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/shmobile/shmob_drm_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/renesas/shmobile/shmob_drm_kms.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/shmobile/shmob_drm_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/renesas/shmobile/shmob_drm_plane.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/renesas/shmobile/shmob_drm_crtc.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_crtc.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_fb.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_gem.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_plane.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_dma.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_fimd.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_dp.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_mixer.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/exynos/exynos_drm_vidi.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/exynos/exynos_drm_drv.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/pl111/pl111_versatile.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/pl111/pl111_display.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/pl111/pl111_drv.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/pl111/pl111_display.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/pl111/pl111_debugfs.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/pl111/pl111_display.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/logicvc/logicvc_drm.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/logicvc/logicvc_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/logicvc/logicvc_interface.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/logicvc/logicvc_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/logicvc/logicvc_layer.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/logicvc/logicvc_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/logicvc/logicvc_mode.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/logicvc/logicvc_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/logicvc/logicvc_of.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/logicvc/logicvc_crtc.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/arm/malidp_hw.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/arm/malidp_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/arm/malidp_planes.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/arm/malidp_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/arm/malidp_crtc.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/arm/malidp_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/arm/malidp_mw.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/arm/malidp_drv.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/mxsfb/lcdif_kms.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/mxsfb/lcdif_drv.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_kms.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_gem.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_vram.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_display.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_vq.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_fence.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_object.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_debugfs.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_plane.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_ioctl.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_prime.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_trace_points.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/virtio/virtgpu_submit.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/virtio/virtgpu_drv.o:include/linux/ftrace.h:560: first defined here
--
   arc-linux-ld: drivers/gpu/drm/kmb/kmb_drv.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/kmb/kmb_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/kmb/kmb_plane.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/kmb/kmb_crtc.o:include/linux/ftrace.h:560: first defined here
   arc-linux-ld: drivers/gpu/drm/kmb/kmb_dsi.o: in function `register_ftrace_direct_hash':
>> include/linux/ftrace.h:560: multiple definition of `register_ftrace_direct_hash'; drivers/gpu/drm/kmb/kmb_crtc.o:include/linux/ftrace.h:560: first defined here

..

vim +560 include/linux/ftrace.h

556
557 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
558 {
559 return -ENODEV;

 > 560	}

561

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#4 lore raw

RFC 3/10 ftrace: Add unregister_ftrace_direct_hash function

In reply to Jiri Olsa

Adding unregister_ftrace_direct_hash function that unregisters all entries (ip -> direct) provided in hash argument.

The difference to current unregister_ftrace_direct is
- hash argument that allows to unregister multiple ip -> direct entries at once
- we can call unregister_ftrace_direct_hash multiple times on the same ftrace_ops object, becase we do not need to unregister all entries at once, we can do it gradualy with the help of ftrace_update_ops function

This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes.

---
 include/linux/ftrace.h |  6 +++
 kernel/trace/ftrace.c  | 98 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 104 insertions(+)
include/linux/ftrace.h +6 -0
index e45bcc9de53b..7ff6004498c0 100644
@@ -527,6 +527,7 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
 int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
 
 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
+int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
 
 void ftrace_stub_direct_tramp(void);
 
@@ -559,6 +560,11 @@ int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash
 	return -ENODEV;
 }
 
+int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
+{
+	return -ENODEV;
+}
+
 /*
  * This must be implemented by the architecture.
  * It is the way the ftrace direct_ops helper, when called
kernel/trace/ftrace.c +98 -0

104 lines not on this page · raw message

Signature · 1 line
2.50.1
#5 lore raw

Re: [RFC 03/10] ftrace: Add unregister_ftrace_direct_hash function

kernel test robot noticed the following build warnings:

Hi Jiri,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on trace/for-next]
[also build test WARNING on bpf-next/net bpf-next/master bpf/master linus/master v6.16 next-20250729] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Jiri-Olsa/ftrace-Make-alloc_and_copy_ftrace_hash-direct-friendly/20250729-183143
base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link:    https://lore.kernel.org/r/20250729102813.1531457-4-jolsa%40kernel.org
patch subject: [RFC 03/10] ftrace: Add unregister_ftrace_direct_hash function
config: x86_64-buildonly-randconfig-003-20250730 (https://download.01.org/0day-ci/archive/20250730/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250730/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from kernel/events/callchain.c:11:
In file included from include/linux/perf_event.h:43:
include/linux/ftrace.h:558:5: warning: no previous prototype for function 'register_ftrace_direct_hash' [-Wmissing-prototypes]
558 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
| ^
include/linux/ftrace.h:558:1: note: declare 'static' if the function is not intended to be used outside of this translation unit

     558 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 
>> include/linux/ftrace.h:563:5: warning: no previous prototype for function 'unregister_ftrace_direct_hash' [-Wmissing-prototypes]
     563 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         |     ^
   include/linux/ftrace.h:563:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     563 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 

2 warnings generated.
--
In file included from kernel/events/core.c:34:
In file included from include/linux/syscalls.h:94:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:43:
include/linux/ftrace.h:558:5: warning: no previous prototype for function 'register_ftrace_direct_hash' [-Wmissing-prototypes]
558 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
| ^
include/linux/ftrace.h:558:1: note: declare 'static' if the function is not intended to be used outside of this translation unit

     558 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 
>> include/linux/ftrace.h:563:5: warning: no previous prototype for function 'unregister_ftrace_direct_hash' [-Wmissing-prototypes]
     563 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         |     ^
   include/linux/ftrace.h:563:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     563 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 

kernel/events/core.c:215:20: warning: unused function 'class_perf_ctx_lock_destructor' [-Wunused-function]
215 | static inline void class_perf_ctx_lock_destructor(class_perf_ctx_lock_t *_T)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
kernel/events/core.c:219:1: warning: unused function 'class_perf_ctx_lock_constructor' [-Wunused-function]
219 | class_perf_ctx_lock_constructor(struct perf_cpu_context *cpuctx,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 warnings generated.

vim +/unregister_ftrace_direct_hash +563 include/linux/ftrace.h

557

 > 558	int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
   559	{
   560		return -ENODEV;
   561	}
   562	
 > 563	int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)

564 {
565 return -ENODEV;
566 }
567

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#6 lore raw

RFC 4/10 ftrace: Add modify_ftrace_direct_hash function

In reply to Jiri Olsa

Adding modify_ftrace_direct_hash function that modifies all entries (ip -> direct) provided in hash argument.

The difference to current unregister_ftrace_direct is
- hash argument that allows to modify multiple ip -> direct entries at once

This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes.

---
 include/linux/ftrace.h |  6 +++++
 kernel/trace/ftrace.c  | 58 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
include/linux/ftrace.h +6 -0
index 7ff6004498c0..8761765d9abc 100644
@@ -528,6 +528,7 @@ int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
 
 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
 int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
+int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock);
 
 void ftrace_stub_direct_tramp(void);
 
@@ -565,6 +566,11 @@ int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *ha
 	return -ENODEV;
 }
 
+int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
+{
+	return -ENODEV;
+}
+
 /*
  * This must be implemented by the architecture.
  * It is the way the ftrace direct_ops helper, when called
kernel/trace/ftrace.c +58 -0

64 lines not on this page · raw message

Signature · 1 line
2.50.1
#7 lore raw

Re: [RFC 04/10] ftrace: Add modify_ftrace_direct_hash function

kernel test robot noticed the following build warnings:

Hi Jiri,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on trace/for-next]
[also build test WARNING on bpf-next/net bpf-next/master bpf/master linus/master v6.16 next-20250729] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Jiri-Olsa/ftrace-Make-alloc_and_copy_ftrace_hash-direct-friendly/20250729-183143
base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link:    https://lore.kernel.org/r/20250729102813.1531457-5-jolsa%40kernel.org
patch subject: [RFC 04/10] ftrace: Add modify_ftrace_direct_hash function
config: x86_64-buildonly-randconfig-003-20250730 (https://download.01.org/0day-ci/archive/20250730/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250730/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from kernel/events/callchain.c:11:
In file included from include/linux/perf_event.h:43:
include/linux/ftrace.h:559:5: warning: no previous prototype for function 'register_ftrace_direct_hash' [-Wmissing-prototypes]
559 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
| ^
include/linux/ftrace.h:559:1: note: declare 'static' if the function is not intended to be used outside of this translation unit

     559 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 
   include/linux/ftrace.h:564:5: warning: no previous prototype for function 'unregister_ftrace_direct_hash' [-Wmissing-prototypes]
     564 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         |     ^
   include/linux/ftrace.h:564:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     564 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 
>> include/linux/ftrace.h:569:5: warning: no previous prototype for function 'modify_ftrace_direct_hash' [-Wmissing-prototypes]
     569 | int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
         |     ^
   include/linux/ftrace.h:569:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     569 | int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
         | ^
         | static 

3 warnings generated.
--
In file included from kernel/events/core.c:34:
In file included from include/linux/syscalls.h:94:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:43:
include/linux/ftrace.h:559:5: warning: no previous prototype for function 'register_ftrace_direct_hash' [-Wmissing-prototypes]
559 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
| ^
include/linux/ftrace.h:559:1: note: declare 'static' if the function is not intended to be used outside of this translation unit

     559 | int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 
   include/linux/ftrace.h:564:5: warning: no previous prototype for function 'unregister_ftrace_direct_hash' [-Wmissing-prototypes]
     564 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         |     ^
   include/linux/ftrace.h:564:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     564 | int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
         | ^
         | static 
>> include/linux/ftrace.h:569:5: warning: no previous prototype for function 'modify_ftrace_direct_hash' [-Wmissing-prototypes]
     569 | int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
         |     ^
   include/linux/ftrace.h:569:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     569 | int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
         | ^
         | static 

kernel/events/core.c:215:20: warning: unused function 'class_perf_ctx_lock_destructor' [-Wunused-function]
215 | static inline void class_perf_ctx_lock_destructor(class_perf_ctx_lock_t *_T)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
kernel/events/core.c:219:1: warning: unused function 'class_perf_ctx_lock_constructor' [-Wunused-function]
219 | class_perf_ctx_lock_constructor(struct perf_cpu_context *cpuctx,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5 warnings generated.

vim +/modify_ftrace_direct_hash +569 include/linux/ftrace.h

563

 > 564	int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
   565	{
   566		return -ENODEV;
   567	}
   568	
 > 569	int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)

570 {
571 return -ENODEV;
572 }
573

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#8 lore raw

RFC 5/10 ftrace: Export some of hash related functions

In reply to Jiri Olsa

We are going to use these functions in following changes.

---
 include/linux/ftrace.h | 16 ++++++++++++++++
 kernel/trace/ftrace.c  |  7 +++----
 kernel/trace/trace.h   |  8 --------
 3 files changed, 19 insertions(+), 12 deletions(-)
include/linux/ftrace.h +16 -0
index 8761765d9abc..9a6fcdafeda2 100644
@@ -397,6 +397,22 @@ enum ftrace_ops_cmd {
 typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cmd cmd);
 
 #ifdef CONFIG_DYNAMIC_FTRACE
+
+#define FTRACE_HASH_DEFAULT_BITS 10
+
+struct ftrace_hash {
+	unsigned long		size_bits;
+	struct hlist_head	*buckets;
+	unsigned long		count;
+	unsigned long		flags;
+	struct rcu_head		rcu;
+};
+
+struct ftrace_hash *alloc_ftrace_hash(int size_bits);
+void free_ftrace_hash(struct ftrace_hash *hash);
+struct ftrace_func_entry *add_hash_entry_direct(struct ftrace_hash *hash,
+						unsigned long ip, unsigned long direct);
+
 /* The hash used to know what functions callbacks trace */
 struct ftrace_ops_hash {
 	struct ftrace_hash __rcu	*notrace_hash;
kernel/trace/ftrace.c +3 -4
index 755d5550ac44..fcb8f2d3172b 100644
@@ -68,7 +68,6 @@
 	})
 
 /* hash bits for specific function selection */
-#define FTRACE_HASH_DEFAULT_BITS 10
 #define FTRACE_HASH_MAX_BITS 12
 
 #ifdef CONFIG_DYNAMIC_FTRACE
@@ -1189,7 +1188,7 @@ static void __add_hash_entry(struct ftrace_hash *hash,
 	hash->count++;
 }
 
-static struct ftrace_func_entry *
+struct ftrace_func_entry *
 add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned long direct)
 {
 	struct ftrace_func_entry *entry;
@@ -1269,7 +1268,7 @@ static void clear_ftrace_mod_list(struct list_head *head)
 	mutex_unlock(&ftrace_lock);
 }
 
-static void free_ftrace_hash(struct ftrace_hash *hash)
+void free_ftrace_hash(struct ftrace_hash *hash)
 {
 	if (!hash || hash == EMPTY_HASH)
 		return;
@@ -1309,7 +1308,7 @@ void ftrace_free_filter(struct ftrace_ops *ops)
 }
 EXPORT_SYMBOL_GPL(ftrace_free_filter);
 
-static struct ftrace_hash *alloc_ftrace_hash(int size_bits)
+struct ftrace_hash *alloc_ftrace_hash(int size_bits)
 {
 	struct ftrace_hash *hash;
 	int size;
kernel/trace/trace.h +0 -8
index bd084953a98b..74ef7755f361 100644
@@ -899,14 +899,6 @@ enum {
 	FTRACE_HASH_FL_MOD	= (1 << 0),
 };
 
-struct ftrace_hash {
-	unsigned long		size_bits;
-	struct hlist_head	*buckets;
-	unsigned long		count;
-	unsigned long		flags;
-	struct rcu_head		rcu;
-};
-
 struct ftrace_func_entry *
 ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip);
 
Signature · 1 line
2.50.1
author reviewer
#9 lore raw

RFC 6/10 ftrace: Use direct hash interface in direct functions

In reply to Jiri Olsa

Implement current *_ftrace_direct function with their *_hash function counterparts.

---
 include/linux/ftrace.h        |  17 +--
 kernel/bpf/trampoline.c       |  10 +-
 kernel/trace/ftrace.c         | 242 +++++-----------------------------
 kernel/trace/trace_selftest.c |   5 +-
 4 files changed, 45 insertions(+), 229 deletions(-)
include/linux/ftrace.h +6 -11
index 9a6fcdafeda2..85f4ab1a1e72 100644
@@ -536,11 +536,10 @@ struct ftrace_func_entry {
 
 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
 unsigned long ftrace_find_rec_direct(unsigned long ip);
-int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
-int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
+int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr);
+int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr,
 			     bool free_filters);
-int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
-int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
+int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr, bool lock_direct_mutex);
 
 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
 int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
@@ -554,20 +553,16 @@ static inline unsigned long ftrace_find_rec_direct(unsigned long ip)
 {
 	return 0;
 }
-static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
+static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr)
 {
 	return -ENODEV;
 }
-static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
+static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr,
 					   bool free_filters)
 {
 	return -ENODEV;
 }
-static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
-{
-	return -ENODEV;
-}
-static inline int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr)
+static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr, bool lock_direct_mutex)
 {
 	return -ENODEV;
 }
kernel/bpf/trampoline.c +3 -7
index 0e364614c3a2..6bf272715f0e 100644
@@ -181,7 +181,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
 	int ret;
 
 	if (tr->func.ftrace_managed)
-		ret = unregister_ftrace_direct(tr->fops, (long)old_addr, false);
+		ret = unregister_ftrace_direct(tr->fops, (unsigned long) ip, (long)old_addr, false);
 	else
 		ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL);
 
@@ -195,10 +195,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad
 	int ret;
 
 	if (tr->func.ftrace_managed) {
-		if (lock_direct_mutex)
-			ret = modify_ftrace_direct(tr->fops, (long)new_addr);
-		else
-			ret = modify_ftrace_direct_nolock(tr->fops, (long)new_addr);
+		ret = modify_ftrace_direct(tr->fops, (unsigned long) ip, (long)new_addr, lock_direct_mutex);
 	} else {
 		ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr);
 	}
@@ -220,8 +217,7 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
 	}
 
 	if (tr->func.ftrace_managed) {
-		ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1);
-		ret = register_ftrace_direct(tr->fops, (long)new_addr);
+		ret = register_ftrace_direct(tr->fops, (unsigned long)ip, (long)new_addr);
 	} else {
 		ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr);
 	}
kernel/trace/ftrace.c +34 -208

296 lines not on this page · raw message

kernel/trace/trace_selftest.c +2 -3
index d88c44f1dfa5..37f5eb1f252b 100644
@@ -1135,8 +1135,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,
 	 * Register direct function together with graph tracer
 	 * and make sure we get graph trace.
 	 */
-	ftrace_set_filter_ip(&direct, (unsigned long)DYN_FTRACE_TEST_NAME, 0, 0);
-	ret = register_ftrace_direct(&direct,
+	ret = register_ftrace_direct(&direct, (unsigned long)DYN_FTRACE_TEST_NAME,
 				     (unsigned long)ftrace_stub_direct_tramp);
 	if (ret)
 		goto out;
@@ -1159,7 +1158,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,
 
 	unregister_ftrace_graph(&fgraph_ops);
 
-	ret = unregister_ftrace_direct(&direct,
+	ret = unregister_ftrace_direct(&direct, (unsigned long)DYN_FTRACE_TEST_NAME,
 				       (unsigned long)ftrace_stub_direct_tramp,
 				       true);
 	if (ret)
Signature · 1 line
2.50.1
author reviewer
#10 lore raw

RFC 7/10 bpf: Add trampoline ip hash table

In reply to Jiri Olsa

Following changes need to lookup trampoline based on its ip address, adding hash table for that.

---
 include/linux/bpf.h     |  7 +++++--
 kernel/bpf/trampoline.c | 30 +++++++++++++++++++-----------
 2 files changed, 24 insertions(+), 13 deletions(-)
include/linux/bpf.h +5 -2
index f9cd2164ed23..c14bde400d97 100644
@@ -1253,14 +1253,17 @@ struct bpf_tramp_image {
 };
 
 struct bpf_trampoline {
-	/* hlist for trampoline_table */
-	struct hlist_node hlist;
+	/* hlist for trampoline_key_table */
+	struct hlist_node hlist_key;
+	/* hlist for trampoline_ip_table */
+	struct hlist_node hlist_ip;
 	struct ftrace_ops *fops;
 	/* serializes access to fields of this trampoline */
 	struct mutex mutex;
 	refcount_t refcnt;
 	u32 flags;
 	u64 key;
+	unsigned long ip;
 	struct {
 		struct btf_func_model model;
 		void *addr;
kernel/bpf/trampoline.c +19 -11
index 6bf272715f0e..84bcd9f6bd74 100644
@@ -24,9 +24,10 @@ const struct bpf_prog_ops bpf_extension_prog_ops = {
 #define TRAMPOLINE_HASH_BITS 10
 #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS)
 
-static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE];
+static struct hlist_head trampoline_key_table[TRAMPOLINE_TABLE_SIZE];
+static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE];
 
-/* serializes access to trampoline_table */
+/* serializes access to trampoline tables */
 static DEFINE_MUTEX(trampoline_mutex);
 
 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
@@ -135,15 +136,15 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
 			   PAGE_SIZE, true, ksym->name);
 }
 
-static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
+static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
 {
 	struct bpf_trampoline *tr;
 	struct hlist_head *head;
 	int i;
 
 	mutex_lock(&trampoline_mutex);
-	head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
-	hlist_for_each_entry(tr, head, hlist) {
+	head = &trampoline_key_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
+	hlist_for_each_entry(tr, head, hlist_key) {
 		if (tr->key == key) {
 			refcount_inc(&tr->refcnt);
 			goto out;
@@ -164,8 +165,12 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
 #endif
 
 	tr->key = key;
-	INIT_HLIST_NODE(&tr->hlist);
-	hlist_add_head(&tr->hlist, head);
+	tr->ip = ip;
+	INIT_HLIST_NODE(&tr->hlist_key);
+	INIT_HLIST_NODE(&tr->hlist_ip);
+	hlist_add_head(&tr->hlist_key, head);
+	head = &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)];
+	hlist_add_head(&tr->hlist_ip, head);
 	refcount_set(&tr->refcnt, 1);
 	mutex_init(&tr->mutex);
 	for (i = 0; i < BPF_TRAMP_MAX; i++)
@@ -800,7 +805,7 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
 					 prog->aux->attach_btf_id);
 
 	bpf_lsm_find_cgroup_shim(prog, &bpf_func);
-	tr = bpf_trampoline_lookup(key);
+	tr = bpf_trampoline_lookup(key, 0);
 	if (WARN_ON_ONCE(!tr))
 		return;
 
@@ -820,7 +825,7 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
 {
 	struct bpf_trampoline *tr;
 
-	tr = bpf_trampoline_lookup(key);
+	tr = bpf_trampoline_lookup(key, tgt_info->tgt_addr);
 	if (!tr)
 		return NULL;
 
@@ -856,7 +861,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
 	 * fexit progs. The fentry-only trampoline will be freed via
 	 * multiple rcu callbacks.
 	 */
-	hlist_del(&tr->hlist);
+	hlist_del(&tr->hlist_key);
+	hlist_del(&tr->hlist_ip);
 	if (tr->fops) {
 		ftrace_free_filter(tr->fops);
 		kfree(tr->fops);
@@ -1135,7 +1141,9 @@ static int __init init_trampolines(void)
 	int i;
 
 	for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
-		INIT_HLIST_HEAD(&trampoline_table[i]);
+		INIT_HLIST_HEAD(&trampoline_key_table[i]);
+	for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
+		INIT_HLIST_HEAD(&trampoline_ip_table[i]);
 	return 0;
 }
 late_initcall(init_trampolines);
Signature · 1 line
2.50.1
author reviewer
#11 lore raw

RFC 8/10 ftrace: Factor ftrace_ops ops_func interface

In reply to Jiri Olsa

We are going to remove "ftrace_ops->private == bpf_trampoline" setup in following changes.

Adding ip argument to ftrace_ops_func_t callback function, so we can use it to look up the trampoline.

---
 include/linux/ftrace.h  |  2 +-
 kernel/bpf/trampoline.c | 26 ++++++++++++++++++++++++--
 kernel/trace/ftrace.c   |  6 +++---
 3 files changed, 28 insertions(+), 6 deletions(-)
include/linux/ftrace.h +1 -1
index 85f4ab1a1e72..1a61f969550d 100644
@@ -394,7 +394,7 @@ enum ftrace_ops_cmd {
  *        Negative on failure. The return value is dependent on the
  *        callback.
  */
-typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cmd cmd);
+typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, unsigned long ip, enum ftrace_ops_cmd cmd);
 
 #ifdef CONFIG_DYNAMIC_FTRACE
 
kernel/bpf/trampoline.c +24 -2
index 84bcd9f6bd74..398c1a722d83 100644
@@ -33,11 +33,33 @@ static DEFINE_MUTEX(trampoline_mutex);
 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
 static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);
 
-static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, enum ftrace_ops_cmd cmd)
+static struct bpf_trampoline *bpf_trampoline_ip_lookup(unsigned long ip)
 {
-	struct bpf_trampoline *tr = ops->private;
+	struct hlist_head *head_ip;
+	struct bpf_trampoline *tr;
+
+	mutex_lock(&trampoline_mutex);
+	head_ip = &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)];
+	hlist_for_each_entry(tr, head_ip, hlist_ip) {
+		if (tr->func.addr == (void *) ip)
+			goto out;
+	}
+	tr = NULL;
+out:
+	mutex_unlock(&trampoline_mutex);
+	return tr;
+}
+
+static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
+				     enum ftrace_ops_cmd cmd)
+{
+	struct bpf_trampoline *tr;
 	int ret = 0;
 
+	tr = bpf_trampoline_ip_lookup(ip);
+	if (!tr)
+		return -EINVAL;
+
 	if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {
 		/* This is called inside register_ftrace_direct_multi(), so
 		 * tr->mutex is already locked.
kernel/trace/ftrace.c +3 -3
index 151ca94f496a..943feabdd5e6 100644
@@ -2040,7 +2040,7 @@ static int __ftrace_hash_update_ipmodify(struct ftrace_ops *ops,
 				 */
 				if (!ops->ops_func)
 					return -EBUSY;
-				ret = ops->ops_func(ops, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF);
+				ret = ops->ops_func(ops, rec->ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF);
 				if (ret)
 					return ret;
 			} else if (is_ipmodify) {
@@ -8746,7 +8746,7 @@ static int prepare_direct_functions_for_ipmodify(struct ftrace_ops *ops)
 				if (!op->ops_func)
 					return -EBUSY;
 
-				ret = op->ops_func(op, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER);
+				ret = op->ops_func(op, ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER);
 				if (ret)
 					return ret;
 			}
@@ -8793,7 +8793,7 @@ static void cleanup_direct_functions_after_ipmodify(struct ftrace_ops *ops)
 
 			/* The cleanup is optional, ignore any errors */
 			if (found_op && op->ops_func)
-				op->ops_func(op, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER);
+				op->ops_func(op, ip, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER);
 		}
 	}
 	mutex_unlock(&direct_mutex);
Signature · 1 line
2.50.1
author reviewer
#12 lore raw

RFC 9/10 bpf: Remove ftrace_ops from bpf_trampoline object

In reply to Jiri Olsa

We no longer need ftrace_ops in each bpf_trampoline object, we can manage with just single ftrace_ops for all direct trampoline attachments.

---
 include/linux/bpf.h     |  1 -
 kernel/bpf/trampoline.c | 34 ++++++++++------------------------
 2 files changed, 10 insertions(+), 25 deletions(-)
include/linux/bpf.h +0 -1
index c14bde400d97..bad29fe38a12 100644
@@ -1257,7 +1257,6 @@ struct bpf_trampoline {
 	struct hlist_node hlist_key;
 	/* hlist for trampoline_ip_table */
 	struct hlist_node hlist_ip;
-	struct ftrace_ops *fops;
 	/* serializes access to fields of this trampoline */
 	struct mutex mutex;
 	refcount_t refcnt;
kernel/bpf/trampoline.c +10 -24
index 398c1a722d83..e6a0e7b20bb6 100644
@@ -175,16 +175,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
 	tr = kzalloc(sizeof(*tr), GFP_KERNEL);
 	if (!tr)
 		goto out;
-#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
-	tr->fops = kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL);
-	if (!tr->fops) {
-		kfree(tr);
-		tr = NULL;
-		goto out;
-	}
-	tr->fops->private = tr;
-	tr->fops->ops_func = bpf_tramp_ftrace_ops_func;
-#endif
 
 	tr->key = key;
 	tr->ip = ip;
@@ -202,13 +192,19 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
 	return tr;
 }
 
+struct ftrace_ops direct_ops = {
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+       .ops_func = bpf_tramp_ftrace_ops_func,
+#endif
+};
+
 static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
 {
 	void *ip = tr->func.addr;
 	int ret;
 
 	if (tr->func.ftrace_managed)
-		ret = unregister_ftrace_direct(tr->fops, (unsigned long) ip, (long)old_addr, false);
+		ret = unregister_ftrace_direct(&direct_ops, (unsigned long) ip, (long)old_addr, false);
 	else
 		ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL);
 
@@ -222,7 +218,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad
 	int ret;
 
 	if (tr->func.ftrace_managed) {
-		ret = modify_ftrace_direct(tr->fops, (unsigned long) ip, (long)new_addr, lock_direct_mutex);
+		ret = modify_ftrace_direct(&direct_ops, (unsigned long) ip, (long)new_addr, lock_direct_mutex);
 	} else {
 		ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr);
 	}
@@ -237,14 +233,11 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
 	int ret;
 
 	faddr = ftrace_location((unsigned long)ip);
-	if (faddr) {
-		if (!tr->fops)
-			return -ENOTSUPP;
+	if (faddr)
 		tr->func.ftrace_managed = true;
-	}
 
 	if (tr->func.ftrace_managed) {
-		ret = register_ftrace_direct(tr->fops, (unsigned long)ip, (long)new_addr);
+		ret = register_ftrace_direct(&direct_ops, (unsigned long)ip, (long)new_addr);
 	} else {
 		ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr);
 	}
@@ -502,9 +495,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		 * BPF_TRAMP_F_SHARE_IPMODIFY is set, we can generate the
 		 * trampoline again, and retry register.
 		 */
-		/* reset fops->func and fops->trampoline for re-register */
-		tr->fops->func = NULL;
-		tr->fops->trampoline = 0;
 
 		/* free im memory and reallocate later */
 		bpf_tramp_image_free(im);
@@ -885,10 +875,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
 	 */
 	hlist_del(&tr->hlist_key);
 	hlist_del(&tr->hlist_ip);
-	if (tr->fops) {
-		ftrace_free_filter(tr->fops);
-		kfree(tr->fops);
-	}
 	kfree(tr);
 out:
 	mutex_unlock(&trampoline_mutex);
Signature · 1 line
2.50.1
author reviewer
#13 lore raw

RFC 10/10 Revert "ftrace: Store direct called addresses in their ops"

In reply to Jiri Olsa

This reverts commit dbaccb618fabde8b8596e341f8d76da63a9b0c2f.

Current code uses ip address to lookup the trampoline and we need the ops to point multiple trampolines, hence this is no longer needed.

TODO this probably breaks arm.

---
 include/linux/ftrace.h | 3 ---
 1 file changed, 3 deletions(-)
include/linux/ftrace.h +0 -3
index 1a61f969550d..27b26a87231c 100644
@@ -457,9 +457,6 @@ struct ftrace_ops {
 	struct list_head		subop_list;
 	ftrace_ops_func_t		ops_func;
 	struct ftrace_ops		*managed;
-#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
-	unsigned long			direct_call;
-#endif
 #endif
 };
 
Signature · 1 line
2.50.1
reviewer
#14 lore raw
In reply to Jiri Olsa

Hi Jiri,

[adding some powerpc and riscv folk, see below]

Quoted 6 lines · Jiri Olsa

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

How important is that, and what sort of speedup does this result in? I ask due to potential performance hits noted below, and I'm lacking context as to why we want to do this in the first place -- what is this intended to enable/improve?

Quoted 3 lines

However having just single ftrace_ops object removes direct_call field from direct_call, which is needed by arm, so I'm not sure it's the right path forward.

It's also needed by powerpc and riscv since commits:

a52f6043a2238d65 ("powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS") b21cdb9523e5561b ("riscv: ftrace: support direct call using call_ops")

Quoted 2 lines

Mark, Florent,
any idea how hard would it be to for arm to get rid of direct_call field?

For architectures which follow the arm64 style of implementation, it's pretty hard to get rid of it without introducing a performance hit to the call and/or a hit to attachment/detachment/modification. It would also end up being a fair amount more complicated.

There's some historical rationale at:

https://lore.kernel.org/lkml/ZfBbxPDd0rz6FN2T@FVFF77S0Q05N/

... but the gist is that for several reasons we want the ops pointer in the callsite, and for atomic modification of this to switch everything dependent on that ops atomically, as this keeps the call logic and attachment/detachment/modification logic simple and pretty fast.

If we remove the direct_call pointer from the ftrace_ops, then IIUC our options include:

* Point the callsite pointer at some intermediate structure which points to the ops (e.g. the dyn_ftrace for the callsite). That introduces an additional dependent load per call that needs the ops, and introduces potential incoherency with other fields in that structure, requiring more synchronization overhead for attachment/detachment/modification.

* Point the callsite pointer at a trampoline which can generate the ops pointer. This requires that every ops has a trampoline even for non-direct usage, which then requires more memory / I$, has more potential failure points, and is generally more complicated. The performance here will vary by architecture and platform, on some this might be faster, on some it might be slower.

Note that we probably still need to bounce through an intermediary trampoline here to actually load from the callsite pointer and indirectly branch to it.

... but I'm not really keen on either unless we really have to remove the ftrace_ops::direct_call field, since they come with a substantial jump in complexity.

Mark.

Quoted 25 lines

thougts? thanks,
jirka

---
Jiri Olsa (10):
ftrace: Make alloc_and_copy_ftrace_hash direct friendly
ftrace: Add register_ftrace_direct_hash function
ftrace: Add unregister_ftrace_direct_hash function
ftrace: Add modify_ftrace_direct_hash function
ftrace: Export some of hash related functions
ftrace: Use direct hash interface in direct functions
bpf: Add trampoline ip hash table
ftrace: Factor ftrace_ops ops_func interface
bpf: Remove ftrace_ops from bpf_trampoline object Revert "ftrace: Store direct called addresses in their ops"

 include/linux/bpf.h           |   8 +-
 include/linux/ftrace.h        |  51 ++++++++++---
 kernel/bpf/trampoline.c       |  94 +++++++++++++-----------
 kernel/trace/ftrace.c         | 481 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------------------
 kernel/trace/trace.h          |   8 --
 kernel/trace/trace_selftest.c |   5 +-
 6 files changed, 395 insertions(+), 252 deletions(-)
#15 lore raw
Quoted 9 lines · Mark Rutland

Hi Jiri,

[adding some powerpc and riscv folk, see below]

Quoted 4 lines · Jiri Olsa

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

Quoted 7 lines
Quoted 2 lines

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

How important is that, and what sort of speedup does this result in? I ask due to potential performance hits noted below, and I'm lacking context as to why we want to do this in the first place -- what is this intended to enable/improve?

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

there's ongoing development by Menglong [1] to organize such attachment for multiple functions and trampolines, but still at the end we have to use ftrace direct interface to do the attachment for each involved ftrace_ops

so at the point of attachment it helps to have as few ftrace_ops objects as possible, in my test code I ended up with just single ftrace_ops object and I see attachment time for 20k functions to be around 3 seconds

IIUC Menglong's change needs 12 ftrace_ops objects so we need to do around 12 direct ftrace_ops direct calls .. so probably not that bad, but still it would be faster with just single ftrace_ops involved

[1] https://lore.kernel.org/bpf/[email protected]/

Quoted 43 lines
Quoted 4 lines

However having just single ftrace_ops object removes direct_call field from direct_call, which is needed by arm, so I'm not sure it's the right path forward.

It's also needed by powerpc and riscv since commits:

a52f6043a2238d65 ("powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS") b21cdb9523e5561b ("riscv: ftrace: support direct call using call_ops")

Quoted 2 lines

Mark, Florent,
any idea how hard would it be to for arm to get rid of direct_call field?

For architectures which follow the arm64 style of implementation, it's pretty hard to get rid of it without introducing a performance hit to the call and/or a hit to attachment/detachment/modification. It would also end up being a fair amount more complicated.

There's some historical rationale at:

https://lore.kernel.org/lkml/ZfBbxPDd0rz6FN2T@FVFF77S0Q05N/

... but the gist is that for several reasons we want the ops pointer in the callsite, and for atomic modification of this to switch everything dependent on that ops atomically, as this keeps the call logic and attachment/detachment/modification logic simple and pretty fast.

If we remove the direct_call pointer from the ftrace_ops, then IIUC our options include:

* Point the callsite pointer at some intermediate structure which points to the ops (e.g. the dyn_ftrace for the callsite). That introduces an additional dependent load per call that needs the ops, and introduces potential incoherency with other fields in that structure, requiring more synchronization overhead for attachment/detachment/modification.

* Point the callsite pointer at a trampoline which can generate the ops pointer. This requires that every ops has a trampoline even for non-direct usage, which then requires more memory / I$, has more potential failure points, and is generally more complicated. The performance here will vary by architecture and platform, on some this might be faster, on some it might be slower.

Quoted 7 lines

Note that we probably still need to bounce through an intermediary trampoline here to actually load from the callsite pointer and indirectly branch to it.

... but I'm not really keen on either unless we really have to remove the ftrace_ops::direct_call field, since they come with a substantial jump in complexity.

ok, that sounds bad.. thanks for the details

Steven, please correct me if/when I'm wrong ;-)

IIUC in x86_64, IF there's just single ftrace_ops defined for the function, it will bypass ftrace trampoline and call directly the direct trampoline for the function, like:

<foo>:

     call direct_trampoline
     ...

IF there are other ftrace_ops 'users' on the same function, we execute each of them like:

<foo>:

    call ftrace_trampoline
      call ftrace_ops_1->func
      call ftrace_ops_2->func
      ...

with our direct ftrace_ops->func currently using ftrace_ops->direct_call to return direct trampoline for the function:

-static void call_direct_funcs(unsigned long ip, unsigned long pip,
- struct ftrace_ops *ops, struct ftrace_regs *fregs)
-{
- unsigned long addr = READ_ONCE(ops->direct_call);
-
- if (!addr)
- return;
-
- arch_ftrace_set_direct_caller(fregs, addr);
-}

in the new changes it will do hash lookup (based on ip) for the direct trampoline we want to execute:

+static void call_direct_funcs_hash(unsigned long ip, unsigned long pip,
+ struct ftrace_ops *ops, struct ftrace_regs *fregs)
+{
+ unsigned long addr;
+
+ addr = ftrace_find_rec_direct(ip);
+ if (!addr)
+ return;
+
+ arch_ftrace_set_direct_caller(fregs, addr);
+}

still this is the slow path for the case where multiple ftrace_ops objects use same function.. for the fast path we have the direct attachment as described above

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

thanks,
jirka

maintainer
#16 lore raw
Quoted 4 lines · On Wed, 30 Jul 2025 13:19:51 +0200 Jiri Olsa <[email protected]> wrote:

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

Sounds like you are reinventing the ftrace mechanism itself. Which I warned against when I first introduced direct trampolines, which were purposely designed to do a few functions, not thousands. But, oh well.

Quoted 9 lines

Steven, please correct me if/when I'm wrong ;-)

IIUC in x86_64, IF there's just single ftrace_ops defined for the function, it will bypass ftrace trampoline and call directly the direct trampoline for the function, like:

   <foo>:
     call direct_trampoline
     ...

Yes.

And it will also do the same for normal ftrace functions. If you have:

struct ftrace_ops {
	.func = myfunc;
};

It will create a trampoline that has:

      <tramp>
	...
	call myfunc
	...
	ret

On x86, I believe the ftrace_ops for myfunc is added to the trampoline, where as in arm, it's part of the function header. To modify it, it requires converting to the list operation (which ignores the ops parameter), then the ops at the function gets changed before it goes to the new function.

And if it is the only ops attached to a function foo, the function foo would have:

      <foo>
	call tramp
	...

But what's nice about this is that if you have 12 different ftrace_ops that each attach to a 1000 different functions, but no two ftrace_ops attach to the same function, they all do the above. No hash needed!

Quoted 27 lines

IF there are other ftrace_ops 'users' on the same function, we execute each of them like:

<foo>:

    call ftrace_trampoline
      call ftrace_ops_1->func
      call ftrace_ops_2->func
      ...

with our direct ftrace_ops->func currently using ftrace_ops->direct_call to return direct trampoline for the function:

-static void call_direct_funcs(unsigned long ip, unsigned long pip,
- struct ftrace_ops *ops, struct ftrace_regs *fregs)
-{
- unsigned long addr = READ_ONCE(ops->direct_call);
-
- if (!addr)
- return;
-
- arch_ftrace_set_direct_caller(fregs, addr);
-}

in the new changes it will do hash lookup (based on ip) for the direct trampoline we want to execute:

Quoted 11 lines

+static void call_direct_funcs_hash(unsigned long ip, unsigned long pip,
+ struct ftrace_ops *ops, struct ftrace_regs *fregs)
+{
+ unsigned long addr;
+
+ addr = ftrace_find_rec_direct(ip);
+ if (!addr)
+ return;
+
+ arch_ftrace_set_direct_caller(fregs, addr);
+}

I think the above will work.

Quoted 6 lines

still this is the slow path for the case where multiple ftrace_ops objects use same function.. for the fast path we have the direct attachment as described above

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

That's a question for Mark, right?

-- Steve

#17 lore raw
Quoted 98 lines · Steven Rostedt
Quoted 4 lines · On Wed, 30 Jul 2025 13:19:51 +0200 Jiri Olsa <[email protected]> wrote:

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

Sounds like you are reinventing the ftrace mechanism itself. Which I warned against when I first introduced direct trampolines, which were purposely designed to do a few functions, not thousands. But, oh well.

Quoted 9 lines

Steven, please correct me if/when I'm wrong ;-)

IIUC in x86_64, IF there's just single ftrace_ops defined for the function, it will bypass ftrace trampoline and call directly the direct trampoline for the function, like:

   <foo>:
     call direct_trampoline
     ...

Yes.

And it will also do the same for normal ftrace functions. If you have:

struct ftrace_ops {
	.func = myfunc;
};

It will create a trampoline that has:

      <tramp>
	...
	call myfunc
	...
	ret

On x86, I believe the ftrace_ops for myfunc is added to the trampoline, where as in arm, it's part of the function header. To modify it, it requires converting to the list operation (which ignores the ops parameter), then the ops at the function gets changed before it goes to the new function.

And if it is the only ops attached to a function foo, the function foo would have:

      <foo>
	call tramp
	...

But what's nice about this is that if you have 12 different ftrace_ops that each attach to a 1000 different functions, but no two ftrace_ops attach to the same function, they all do the above. No hash needed!

Quoted 27 lines

IF there are other ftrace_ops 'users' on the same function, we execute each of them like:

<foo>:

    call ftrace_trampoline
      call ftrace_ops_1->func
      call ftrace_ops_2->func
      ...

with our direct ftrace_ops->func currently using ftrace_ops->direct_call to return direct trampoline for the function:

-static void call_direct_funcs(unsigned long ip, unsigned long pip,
- struct ftrace_ops *ops, struct ftrace_regs *fregs)
-{
- unsigned long addr = READ_ONCE(ops->direct_call);
-
- if (!addr)
- return;
-
- arch_ftrace_set_direct_caller(fregs, addr);
-}

in the new changes it will do hash lookup (based on ip) for the direct trampoline we want to execute:

Quoted 11 lines

+static void call_direct_funcs_hash(unsigned long ip, unsigned long pip,
+ struct ftrace_ops *ops, struct ftrace_regs *fregs)
+{
+ unsigned long addr;
+
+ addr = ftrace_find_rec_direct(ip);
+ if (!addr)
+ return;
+
+ arch_ftrace_set_direct_caller(fregs, addr);
+}

I think the above will work.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

still this is the slow path for the case where multiple ftrace_ops objects use same function.. for the fast path we have the direct attachment as described above

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

That's a question for Mark, right?

yes, thanks for the other details

jirka

reviewer
#18 lore raw
In reply to Jiri Olsa
Quoted 18 lines · Jiri Olsa
Quoted 9 lines · Mark Rutland

Hi Jiri,

[adding some powerpc and riscv folk, see below]

Quoted 4 lines · Jiri Olsa

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

Quoted 7 lines
Quoted 2 lines

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

How important is that, and what sort of speedup does this result in? I ask due to potential performance hits noted below, and I'm lacking context as to why we want to do this in the first place -- what is this intended to enable/improve?

Quoted 4 lines

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

Do you know which aspect of that is slow? e.g. is that becuase you have to update each ftrace_ops independently, and pay the synchronization overhead per-ops?

I ask because it might be possible to do some more batching there, at least for architectures like arm64 that use the CALL_OPS approach.

Quoted 69 lines

there's ongoing development by Menglong [1] to organize such attachment for multiple functions and trampolines, but still at the end we have to use ftrace direct interface to do the attachment for each involved ftrace_ops

so at the point of attachment it helps to have as few ftrace_ops objects as possible, in my test code I ended up with just single ftrace_ops object and I see attachment time for 20k functions to be around 3 seconds

IIUC Menglong's change needs 12 ftrace_ops objects so we need to do around 12 direct ftrace_ops direct calls .. so probably not that bad, but still it would be faster with just single ftrace_ops involved

[1] https://lore.kernel.org/bpf/[email protected]/

Quoted 43 lines
Quoted 4 lines

However having just single ftrace_ops object removes direct_call field from direct_call, which is needed by arm, so I'm not sure it's the right path forward.

It's also needed by powerpc and riscv since commits:

a52f6043a2238d65 ("powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS") b21cdb9523e5561b ("riscv: ftrace: support direct call using call_ops")

Quoted 2 lines

Mark, Florent,
any idea how hard would it be to for arm to get rid of direct_call field?

For architectures which follow the arm64 style of implementation, it's pretty hard to get rid of it without introducing a performance hit to the call and/or a hit to attachment/detachment/modification. It would also end up being a fair amount more complicated.

There's some historical rationale at:

https://lore.kernel.org/lkml/ZfBbxPDd0rz6FN2T@FVFF77S0Q05N/

... but the gist is that for several reasons we want the ops pointer in the callsite, and for atomic modification of this to switch everything dependent on that ops atomically, as this keeps the call logic and attachment/detachment/modification logic simple and pretty fast.

If we remove the direct_call pointer from the ftrace_ops, then IIUC our options include:

* Point the callsite pointer at some intermediate structure which points to the ops (e.g. the dyn_ftrace for the callsite). That introduces an additional dependent load per call that needs the ops, and introduces potential incoherency with other fields in that structure, requiring more synchronization overhead for attachment/detachment/modification.

* Point the callsite pointer at a trampoline which can generate the ops pointer. This requires that every ops has a trampoline even for non-direct usage, which then requires more memory / I$, has more potential failure points, and is generally more complicated. The performance here will vary by architecture and platform, on some this might be faster, on some it might be slower.

Quoted 7 lines

Note that we probably still need to bounce through an intermediary trampoline here to actually load from the callsite pointer and indirectly branch to it.

... but I'm not really keen on either unless we really have to remove the ftrace_ops::direct_call field, since they come with a substantial jump in complexity.

ok, that sounds bad.. thanks for the details

Steven, please correct me if/when I'm wrong ;-)

Quoted 7 lines

IIUC in x86_64, IF there's just single ftrace_ops defined for the function, it will bypass ftrace trampoline and call directly the direct trampoline for the function, like:

   <foo>:
     call direct_trampoline
     ...

More details at the end of this reply; arm64 can sometimes do this, but not always, and even when there's a single ftrace_ops we may need to bounce through a common trampoline (which can still be cheap).

Quoted 8 lines

IF there are other ftrace_ops 'users' on the same function, we execute each of them like:

<foo>:

    call ftrace_trampoline
      call ftrace_ops_1->func
      call ftrace_ops_2->func
      ...

More details at the end of this reply; arm64 does essentially the same thing via the ftrace_list_ops and ftrace_ops_list_func().

Quoted 13 lines

with our direct ftrace_ops->func currently using ftrace_ops->direct_call to return direct trampoline for the function:

-static void call_direct_funcs(unsigned long ip, unsigned long pip,
- struct ftrace_ops *ops, struct ftrace_regs *fregs)
-{
- unsigned long addr = READ_ONCE(ops->direct_call);
-
- if (!addr)
- return;
-
- arch_ftrace_set_direct_caller(fregs, addr);
-}

More details at the end of this reply; at present, when an instrumented function has a single ops, arm64 can call ops->direct_call directly from the common trampoline, and only needs to fall back to call_direct_funcs() when there are multiple ops.

Quoted 15 lines

in the new changes it will do hash lookup (based on ip) for the direct trampoline we want to execute:

+static void call_direct_funcs_hash(unsigned long ip, unsigned long pip,
+ struct ftrace_ops *ops, struct ftrace_regs *fregs)
+{
+ unsigned long addr;
+
+ addr = ftrace_find_rec_direct(ip);
+ if (!addr)
+ return;
+
+ arch_ftrace_set_direct_caller(fregs, addr);
+}

Quoted 5 lines

still this is the slow path for the case where multiple ftrace_ops objects use same function.. for the fast path we have the direct attachment as described above

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

On arm64 we have a fast path, BUT branch range limitations means that we cannot always branch directly from the instrumented function to the direct func with a single branch instruction. We use ops->direct_call to handle that case within a common trampoline, which is significantly cheaper that iterating over the ops and/or looking up the direct func from a hash.

With CALL_OPS, we place a pointer to the ops immediately before the instrumented function, and have the instrumented function branch to a common trampoline which can load that pointer (and can then branch to any direct func as necessary).

The instrumented function looks like:

# Aligned to 8 bytes
func - 8:
< pointer to ops >
func:
BTI // optional
MOV X9, LR // save original return address
NOP // patched to BL ftrace_caller
func_body:

... and then in ftrace_caller we can recover the 'ops' pointer with:

	BIC	<tmp>, LR, 0x7					// align down (skips BTI)
	LDR	<ops>, [<tmp>, #-16]				// load ops pointer

	LDR	<direct>, [<ops>, #FTRACE_OPS_DIRECT_CALL]	// load ops->direct_call
	CBNZ	<direct>, ftrace_caller_direct			// if !NULL, make direct call

	< fall through to non-direct func case here >

Having the ops (and ops->direct_call) means that getting to the direct func is significantly cheaper than having to lookup the direct func via the hash.

Where an instrumented function has a single ops, this can get to the direct func with a low constant overhead, significantly cheaper than looking up the direct func via the hash.

Where an instrumented function has multiple ops, the ops pointer is pointed at a common ftrace_list_ops, where ftrace_ops_list_func() iterates over all the other relevant ops.

Mark.

#19 lore raw
Quoted 24 lines · Mark Rutland
Quoted 18 lines · Jiri Olsa
Quoted 9 lines · Mark Rutland

Hi Jiri,

[adding some powerpc and riscv folk, see below]

Quoted 4 lines · Jiri Olsa

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

Quoted 7 lines
Quoted 2 lines

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

How important is that, and what sort of speedup does this result in? I ask due to potential performance hits noted below, and I'm lacking context as to why we want to do this in the first place -- what is this intended to enable/improve?

Quoted 4 lines

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

Quoted 6 lines

Do you know which aspect of that is slow? e.g. is that becuase you have to update each ftrace_ops independently, and pay the synchronization overhead per-ops?

I ask because it might be possible to do some more batching there, at least for architectures like arm64 that use the CALL_OPS approach.

IIRC it's the rcu sync in register_ftrace_direct and ftrace_shutdown I'll try to profile that case again, there might have been changes since the last time we did that

Quoted 146 lines
Quoted 70 lines

there's ongoing development by Menglong [1] to organize such attachment for multiple functions and trampolines, but still at the end we have to use ftrace direct interface to do the attachment for each involved ftrace_ops

so at the point of attachment it helps to have as few ftrace_ops objects as possible, in my test code I ended up with just single ftrace_ops object and I see attachment time for 20k functions to be around 3 seconds

IIUC Menglong's change needs 12 ftrace_ops objects so we need to do around 12 direct ftrace_ops direct calls .. so probably not that bad, but still it would be faster with just single ftrace_ops involved

[1] https://lore.kernel.org/bpf/[email protected]/

Quoted 43 lines
Quoted 4 lines

However having just single ftrace_ops object removes direct_call field from direct_call, which is needed by arm, so I'm not sure it's the right path forward.

It's also needed by powerpc and riscv since commits:

a52f6043a2238d65 ("powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS") b21cdb9523e5561b ("riscv: ftrace: support direct call using call_ops")

Quoted 2 lines

Mark, Florent,
any idea how hard would it be to for arm to get rid of direct_call field?

For architectures which follow the arm64 style of implementation, it's pretty hard to get rid of it without introducing a performance hit to the call and/or a hit to attachment/detachment/modification. It would also end up being a fair amount more complicated.

There's some historical rationale at:

https://lore.kernel.org/lkml/ZfBbxPDd0rz6FN2T@FVFF77S0Q05N/

... but the gist is that for several reasons we want the ops pointer in the callsite, and for atomic modification of this to switch everything dependent on that ops atomically, as this keeps the call logic and attachment/detachment/modification logic simple and pretty fast.

If we remove the direct_call pointer from the ftrace_ops, then IIUC our options include:

* Point the callsite pointer at some intermediate structure which points to the ops (e.g. the dyn_ftrace for the callsite). That introduces an additional dependent load per call that needs the ops, and introduces potential incoherency with other fields in that structure, requiring more synchronization overhead for attachment/detachment/modification.

* Point the callsite pointer at a trampoline which can generate the ops pointer. This requires that every ops has a trampoline even for non-direct usage, which then requires more memory / I$, has more potential failure points, and is generally more complicated. The performance here will vary by architecture and platform, on some this might be faster, on some it might be slower.

Quoted 7 lines

Note that we probably still need to bounce through an intermediary trampoline here to actually load from the callsite pointer and indirectly branch to it.

... but I'm not really keen on either unless we really have to remove the ftrace_ops::direct_call field, since they come with a substantial jump in complexity.

ok, that sounds bad.. thanks for the details

Steven, please correct me if/when I'm wrong ;-)

Quoted 7 lines

IIUC in x86_64, IF there's just single ftrace_ops defined for the function, it will bypass ftrace trampoline and call directly the direct trampoline for the function, like:

   <foo>:
     call direct_trampoline
     ...

More details at the end of this reply; arm64 can sometimes do this, but not always, and even when there's a single ftrace_ops we may need to bounce through a common trampoline (which can still be cheap).

Quoted 8 lines

IF there are other ftrace_ops 'users' on the same function, we execute each of them like:

<foo>:

    call ftrace_trampoline
      call ftrace_ops_1->func
      call ftrace_ops_2->func
      ...

More details at the end of this reply; arm64 does essentially the same thing via the ftrace_list_ops and ftrace_ops_list_func().

Quoted 13 lines

with our direct ftrace_ops->func currently using ftrace_ops->direct_call to return direct trampoline for the function:

-static void call_direct_funcs(unsigned long ip, unsigned long pip,
- struct ftrace_ops *ops, struct ftrace_regs *fregs)
-{
- unsigned long addr = READ_ONCE(ops->direct_call);
-
- if (!addr)
- return;
-
- arch_ftrace_set_direct_caller(fregs, addr);
-}

More details at the end of this reply; at present, when an instrumented function has a single ops, arm64 can call ops->direct_call directly from the common trampoline, and only needs to fall back to call_direct_funcs() when there are multiple ops.

Quoted 15 lines

in the new changes it will do hash lookup (based on ip) for the direct trampoline we want to execute:

+static void call_direct_funcs_hash(unsigned long ip, unsigned long pip,
+ struct ftrace_ops *ops, struct ftrace_regs *fregs)
+{
+ unsigned long addr;
+
+ addr = ftrace_find_rec_direct(ip);
+ if (!addr)
+ return;
+
+ arch_ftrace_set_direct_caller(fregs, addr);
+}

Quoted 5 lines

still this is the slow path for the case where multiple ftrace_ops objects use same function.. for the fast path we have the direct attachment as described above

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

On arm64 we have a fast path, BUT branch range limitations means that we cannot always branch directly from the instrumented function to the direct func with a single branch instruction. We use ops->direct_call to handle that case within a common trampoline, which is significantly cheaper that iterating over the ops and/or looking up the direct func from a hash.

With CALL_OPS, we place a pointer to the ops immediately before the instrumented function, and have the instrumented function branch to a common trampoline which can load that pointer (and can then branch to any direct func as necessary).

Quoted 5 lines

The instrumented function looks like:

# Aligned to 8 bytes
func - 8:
< pointer to ops >

stupid question.. so there's ftrace_ops pointer stored for each function at 'func - 8` ? why not store the func's direct trampoline address in there?

Quoted 20 lines

func:
BTI // optional
MOV X9, LR // save original return address
NOP // patched to BL ftrace_caller
func_body:

... and then in ftrace_caller we can recover the 'ops' pointer with:

	BIC	<tmp>, LR, 0x7					// align down (skips BTI)
	LDR	<ops>, [<tmp>, #-16]				// load ops pointer

	LDR	<direct>, [<ops>, #FTRACE_OPS_DIRECT_CALL]	// load ops->direct_call
	CBNZ	<direct>, ftrace_caller_direct			// if !NULL, make direct call

	< fall through to non-direct func case here >

Having the ops (and ops->direct_call) means that getting to the direct func is significantly cheaper than having to lookup the direct func via the hash.

Quoted 7 lines

Where an instrumented function has a single ops, this can get to the direct func with a low constant overhead, significantly cheaper than looking up the direct func via the hash.

Where an instrumented function has multiple ops, the ops pointer is pointed at a common ftrace_list_ops, where ftrace_ops_list_func() iterates over all the other relevant ops.

thanks for all the details, I'll check if both the new change and ops->direct_call could live together for x86 and other arch, but it will probably complicate things a lot more

jirka

reviewer
#20 lore raw
Quoted 26 lines · Jiri Olsa
Quoted 25 lines · Mark Rutland
Quoted 15 lines · Jiri Olsa
Quoted 6 lines · Mark Rutland
Quoted 4 lines · Jiri Olsa

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

Quoted 7 lines
Quoted 2 lines

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

How important is that, and what sort of speedup does this result in? I ask due to potential performance hits noted below, and I'm lacking context as to why we want to do this in the first place -- what is this intended to enable/improve?

Quoted 4 lines

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

Do you know which aspect of that is slow? e.g. is that becuase you have to update each ftrace_ops independently, and pay the synchronization overhead per-ops?

Quoted 6 lines
Quoted 2 lines

I ask because it might be possible to do some more batching there, at least for architectures like arm64 that use the CALL_OPS approach.

IIRC it's the rcu sync in register_ftrace_direct and ftrace_shutdown I'll try to profile that case again, there might have been changes since the last time we did that

Do you mean synchronize_rcu_tasks()?

The call in register_ftrace_direct() was removed in commit:

33f137143e651321 ("ftrace: Use asynchronous grace period for register_ftrace_direct()")

... but in ftrace_shutdown() we still have a call to synchronize_rcu_tasks(), and to synchronize_rcu_tasks_rude().

The call to synchronize_rcu_tasks() is still necessary, but we might be abel to batch that better with API changes.

I think we might be able to remove the call to synchronize_rcu_tasks_rude() on architectures with ARCH_WANTS_NO_INSTR, since there shouldn't be any instrumentable functions called with RCU not watching. That'd need to be checked.

[...]

Quoted 15 lines
Quoted 15 lines
Quoted 2 lines

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

On arm64 we have a fast path, BUT branch range limitations means that we cannot always branch directly from the instrumented function to the direct func with a single branch instruction. We use ops->direct_call to handle that case within a common trampoline, which is significantly cheaper that iterating over the ops and/or looking up the direct func from a hash.

With CALL_OPS, we place a pointer to the ops immediately before the instrumented function, and have the instrumented function branch to a common trampoline which can load that pointer (and can then branch to any direct func as necessary).

Quoted 8 lines
Quoted 5 lines

The instrumented function looks like:

# Aligned to 8 bytes
func - 8:
< pointer to ops >

stupid question.. so there's ftrace_ops pointer stored for each function at 'func - 8` ? why not store the func's direct trampoline address in there?

Once reason is that today we don't have trampolines for all ops. Since branch range limitations can require bouncing through the common ops, it's simpler/better to bounce from that to the regular call than to bounce from that to a trampoline which makes the regular call.

We *could* consider adding trampolines, but that comes with a jump in complexity that we originally tried to avoid, and a potential performance hit for regular ftrace calls. IIUC that will require similar synchronization to what we have today, so it's not clearly a win generally.

I'd like to better understand what the real bottleneck is; AFAICT it's the tasks-rcu synchronization, and sharing the hash means that you only need to do that once. I think that it should be possible to share that synchronization across multiple ops updates with some API changes (e.g. something like the batching of text_poke on x86).

If we can do that, it might benefit other users too (e.g. live-patching), even if trampolines aren't being used, and would keep the arch bits simple/maintainable.

[...]

Quoted 3 lines

thanks for all the details, I'll check if both the new change and ops->direct_call could live together for x86 and other arch, but it will probably complicate things a lot more

Thanks; please let me know if there's any challenges there!

Mark.

#21 lore raw
Quoted 88 lines · Mark Rutland
Quoted 26 lines · Jiri Olsa
Quoted 25 lines · Mark Rutland
Quoted 15 lines · Jiri Olsa
Quoted 6 lines · Mark Rutland
Quoted 4 lines · Jiri Olsa

hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops object to attach all trampolines.

Quoted 7 lines
Quoted 2 lines

This change allows to use less direct API calls during the attachment changes in the future code, so in effect speeding up the attachment.

How important is that, and what sort of speedup does this result in? I ask due to potential performance hits noted below, and I'm lacking context as to why we want to do this in the first place -- what is this intended to enable/improve?

Quoted 4 lines

so it's all work on PoC stage, the idea is to be able to attach many (like 20,30,40k) functions to their trampolines quickly, which at the moment is slow because all the involved interfaces work with just single function/tracempoline relation

Do you know which aspect of that is slow? e.g. is that becuase you have to update each ftrace_ops independently, and pay the synchronization overhead per-ops?

Quoted 6 lines
Quoted 2 lines

I ask because it might be possible to do some more batching there, at least for architectures like arm64 that use the CALL_OPS approach.

IIRC it's the rcu sync in register_ftrace_direct and ftrace_shutdown I'll try to profile that case again, there might have been changes since the last time we did that

Do you mean synchronize_rcu_tasks()?

The call in register_ftrace_direct() was removed in commit:

33f137143e651321 ("ftrace: Use asynchronous grace period for register_ftrace_direct()")

... but in ftrace_shutdown() we still have a call to synchronize_rcu_tasks(), and to synchronize_rcu_tasks_rude().

The call to synchronize_rcu_tasks() is still necessary, but we might be abel to batch that better with API changes.

I think we might be able to remove the call to synchronize_rcu_tasks_rude() on architectures with ARCH_WANTS_NO_INSTR, since there shouldn't be any instrumentable functions called with RCU not watching. That'd need to be checked.

[...]

Quoted 15 lines
Quoted 15 lines
Quoted 2 lines

sorry I probably forgot/missed discussion on this, but doing the fast path like in x86_64 is not an option in arm, right?

On arm64 we have a fast path, BUT branch range limitations means that we cannot always branch directly from the instrumented function to the direct func with a single branch instruction. We use ops->direct_call to handle that case within a common trampoline, which is significantly cheaper that iterating over the ops and/or looking up the direct func from a hash.

With CALL_OPS, we place a pointer to the ops immediately before the instrumented function, and have the instrumented function branch to a common trampoline which can load that pointer (and can then branch to any direct func as necessary).

Quoted 8 lines
Quoted 5 lines

The instrumented function looks like:

# Aligned to 8 bytes
func - 8:
< pointer to ops >

stupid question.. so there's ftrace_ops pointer stored for each function at 'func - 8` ? why not store the func's direct trampoline address in there?

Once reason is that today we don't have trampolines for all ops. Since branch range limitations can require bouncing through the common ops, it's simpler/better to bounce from that to the regular call than to bounce from that to a trampoline which makes the regular call.

We *could* consider adding trampolines, but that comes with a jump in complexity that we originally tried to avoid, and a potential performance hit for regular ftrace calls. IIUC that will require similar synchronization to what we have today, so it's not clearly a win generally.

Quoted 5 lines

I'd like to better understand what the real bottleneck is; AFAICT it's the tasks-rcu synchronization, and sharing the hash means that you only need to do that once. I think that it should be possible to share that synchronization across multiple ops updates with some API changes (e.g. something like the batching of text_poke on x86).

yea, so rcu does not seem to be the cause anymore (IIRC that was the case some time ago) it looks like now the time is spent in the ftrace internals that iterate and update call sites

the test was loop on attach/detach of fentry program

    31.48%  test_progs       [kernel.kallsyms]                               [k] ftrace_replace_code
    10.98%  test_progs       [kernel.kallsyms]                               [k] __ftrace_hash_update_ipmodify
     6.41%  test_progs       [kernel.kallsyms]                               [k] __ftrace_hash_rec_update
     4.69%  test_progs       [kernel.kallsyms]                               [k] ftrace_check_record
     4.59%  test_progs       [kernel.kallsyms]                               [k] ftrace_lookup_ip
     3.65%  swapper          [kernel.kallsyms]                               [k] acpi_os_read_port
     3.40%  test_progs       [kernel.kallsyms]                               [k] srso_alias_return_thunk
     2.97%  test_progs       [kernel.kallsyms]                               [k] srso_alias_safe_ret
     2.67%  test_progs       [kernel.kallsyms]                               [k] ftrace_rec_iter_record
     2.05%  test_progs       [kernel.kallsyms]                               [k] ftrace_test_record
     1.83%  test_progs       [kernel.kallsyms]                               [k] ftrace_rec_iter_next
     1.76%  test_progs       [kernel.kallsyms]                               [k] smp_call_function_many_cond
     1.05%  rcu_tasks_kthre  [kernel.kallsyms]                               [k] rcu_tasks_pertask
     0.70%  test_progs       [kernel.kallsyms]                               [k] btf_find_by_name_kind
     0.61%  swapper          [kernel.kallsyms]                               [k] srso_alias_safe_ret
     0.55%  swapper          [kernel.kallsyms]                               [k] io_idle

so by sharing the hash we do that (iterate and update functions) just once

jirka

---

    31.48%  test_progs       [kernel.kallsyms]                               [k] ftrace_replace_code
            |          
            |--11.54%--ftrace_replace_code
            |          ftrace_modify_all_code
            |          |          
            |          |--6.06%--ftrace_shutdown.part.0
            |          |          unregister_ftrace_function
            |          |          unregister_ftrace_direct
            |          |          bpf_trampoline_update
            |          |          bpf_trampoline_unlink_prog
            |          |          bpf_tracing_link_release
            |          |          bpf_link_free
            |          |          bpf_link_release
            |          |          __fput
            |          |          __x64_sys_close
            |          |          do_syscall_64
            |          |          entry_SYSCALL_64_after_hwframe
            |          |          __syscall_cancel_arch_end
            |          |          __syscall_cancel
            |          |          __close
            |          |          fentry_test_common
            |          |          fentry_test
            |          |          test_fentry_test
            |          |          run_one_test
            |          |          main
            |          |          __libc_start_call_main
            |          |          __libc_start_main@@GLIBC_2.34
            |          |          _start
            |          |          
            |           --5.47%--ftrace_startup
            |                     register_ftrace_function_nolock
            |                     register_ftrace_direct
            |                     bpf_trampoline_update
            |                     __bpf_trampoline_link_prog
            |                     bpf_trampoline_link_prog
            |                     bpf_tracing_prog_attach
            |                     bpf_raw_tp_link_attach
            |                     __sys_bpf
            |                     __x64_sys_bpf
            |                     do_syscall_64
            |                     entry_SYSCALL_64_after_hwframe
            |                     syscall
            |                     skel_raw_tracepoint_open
            |                     fentry_test_lskel__test1__attach
            |                     fentry_test_common
            |                     fentry_test
            |                     test_fentry_test
            |                     run_one_test
            |                     main
            |                     __libc_start_call_main
            |                     __libc_start_main@@GLIBC_2.34
            |                     _start
            |          
            |--8.81%--ftrace_check_record
            |          ftrace_replace_code
            |          ftrace_modify_all_code
            |          |          
            |          |--4.72%--ftrace_shutdown.part.0
            |          |          unregister_ftrace_function
            |          |          unregister_ftrace_direct
            |          |          bpf_trampoline_update
            |          |          bpf_trampoline_unlink_prog
            |          |          bpf_tracing_link_release
            |          |          bpf_link_free
            |          |          bpf_link_release
            |          |          __fput
            |          |          __x64_sys_close
            |          |          do_syscall_64
            |          |          entry_SYSCALL_64_after_hwframe
            |          |          __syscall_cancel_arch_end
            |          |          __syscall_cancel
            |          |          __close
            |          |          fentry_test_common
            |          |          fentry_test
            |          |          test_fentry_test
            |          |          run_one_test
            |          |          main
            |          |          __libc_start_call_main
            |          |          __libc_start_main@@GLIBC_2.34
            |          |          _start
            |          |          
            |           --4.10%--ftrace_startup
            |                     register_ftrace_function_nolock
            |                     register_ftrace_direct
            |                     bpf_trampoline_update
            |                     __bpf_trampoline_link_prog
            |                     bpf_trampoline_link_prog
            |                     bpf_tracing_prog_attach
            |                     bpf_raw_tp_link_attach
            |                     __sys_bpf
            |                     __x64_sys_bpf
            |                     do_syscall_64
            |                     entry_SYSCALL_64_after_hwframe
            |                     syscall
            |                     skel_raw_tracepoint_open
            |                     fentry_test_lskel__test1__attach
            |                     fentry_test_common
            |                     fentry_test
            |                     test_fentry_test
            |                     run_one_test
            |                     main
            |                     __libc_start_call_main
            |                     __libc_start_main@@GLIBC_2.34
            |                     _start
            |          
            |--3.60%--ftrace_rec_iter_record
            |          ftrace_replace_code
            |          ftrace_modify_all_code
            |          |          
            |          |--1.91%--ftrace_shutdown.part.0
            |          |          unregister_ftrace_function
            |          |          unregister_ftrace_direct
            |          |          bpf_trampoline_update
            |          |          bpf_trampoline_unlink_prog
            |          |          bpf_tracing_link_release
            |          |          bpf_link_free
            |          |          bpf_link_release
            |          |          __fput
            |          |          __x64_sys_close
            |          |          do_syscall_64
            |          |          entry_SYSCALL_64_after_hwframe
            |          |          __syscall_cancel_arch_end
            |          |          __syscall_cancel
            |          |          __close
            |          |          fentry_test_common
            |          |          fentry_test
            |          |          test_fentry_test
            |          |          run_one_test
            |          |          main
            |          |          __libc_start_call_main
            |          |          __libc_start_main@@GLIBC_2.34
            |          |          _start
            |          |          
            |           --1.69%--ftrace_startup
            |                     register_ftrace_function_nolock
            |                     register_ftrace_direct
            |                     bpf_trampoline_update
            |                     __bpf_trampoline_link_prog
            |                     bpf_trampoline_link_prog
            |                     bpf_tracing_prog_attach
            |                     bpf_raw_tp_link_attach
            |                     __sys_bpf
            |                     __x64_sys_bpf
            |                     do_syscall_64
            |                     entry_SYSCALL_64_after_hwframe
            |                     syscall
            |                     skel_raw_tracepoint_open
            |                     fentry_test_lskel__test1__attach
            |                     fentry_test_common
            |                     fentry_test
            |                     test_fentry_test
            |                     run_one_test
            |                     main
            |                     __libc_start_call_main
            |                     __libc_start_main@@GLIBC_2.34
            |                     _start
            |          
            |--3.50%--ftrace_rec_iter_next
            |          ftrace_replace_code
            |          ftrace_modify_all_code
            |          |          
            |          |--2.08%--ftrace_startup
            |          |          register_ftrace_function_nolock
            |          |          register_ftrace_direct
            |          |          bpf_trampoline_update
            |          |          __bpf_trampoline_link_prog
            |          |          bpf_trampoline_link_prog
            |          |          bpf_tracing_prog_attach
            |          |          bpf_raw_tp_link_attach
            |          |          __sys_bpf
            |          |          __x64_sys_bpf
            |          |          do_syscall_64
            |          |          entry_SYSCALL_64_after_hwframe
            |          |          syscall
            |          |          skel_raw_tracepoint_open
            |          |          fentry_test_lskel__test1__attach
            |          |          fentry_test_common
            |          |          fentry_test
            |          |          test_fentry_test
            |          |          run_one_test
            |          |          main
            |          |          __libc_start_call_main
            |          |          __libc_start_main@@GLIBC_2.34
            |          |          _start
            |          |          
            |           --1.42%--ftrace_shutdown.part.0
            |                     unregister_ftrace_function
            |                     unregister_ftrace_direct
            |                     bpf_trampoline_update
            |                     bpf_trampoline_unlink_prog
            |                     bpf_tracing_link_release
            |                     bpf_link_free
            |                     bpf_link_release
            |                     __fput
            |                     __x64_sys_close
            |                     do_syscall_64
            |                     entry_SYSCALL_64_after_hwframe
            |                     __syscall_cancel_arch_end
            |                     __syscall_cancel
            |                     __close
            |                     fentry_test_common
            |                     fentry_test
            |                     test_fentry_test
            |                     run_one_test
            |                     main
            |                     __libc_start_call_main
            |                     __libc_start_main@@GLIBC_2.34
            |                     _start
            |          
            |--2.44%--srso_alias_safe_ret
            |          srso_alias_return_thunk
            |          ftrace_replace_code
            |          ftrace_modify_all_code
            |          |          
            |          |--1.36%--ftrace_shutdown.part.0
            |          |          unregister_ftrace_function
            |          |          unregister_ftrace_direct
            |          |          bpf_trampoline_update
            |          |          bpf_trampoline_unlink_prog
            |          |          bpf_tracing_link_release
            |          |          bpf_link_free
            |          |          bpf_link_release
            |          |          __fput
            |          |          __x64_sys_close
            |          |          do_syscall_64
            |          |          entry_SYSCALL_64_after_hwframe
            |          |          __syscall_cancel_arch_end
            |          |          __syscall_cancel
            |          |          __close
            |          |          fentry_test_common
            |          |          fentry_test
            |          |          test_fentry_test
            |          |          run_one_test
            |          |          main
            |          |          __libc_start_call_main
            |          |          __libc_start_main@@GLIBC_2.34
            |          |          _start
            |          |          
            |           --1.07%--ftrace_startup
            |                     register_ftrace_function_nolock
            |                     register_ftrace_direct
            |                     bpf_trampoline_update
            |                     __bpf_trampoline_link_prog
            |                     bpf_trampoline_link_prog
            |                     bpf_tracing_prog_attach
            |                     bpf_raw_tp_link_attach
            |                     __sys_bpf
            |                     __x64_sys_bpf
            |                     do_syscall_64
            |                     entry_SYSCALL_64_after_hwframe
            |                     syscall
            |                     skel_raw_tracepoint_open
            |                     fentry_test_lskel__test1__attach
            |                     fentry_test_common
            |                     fentry_test
            |                     test_fentry_test
            |                     run_one_test
            |                     main
            |                     __libc_start_call_main
            |                     __libc_start_main@@GLIBC_2.34
            |                     _start
            |          

--1.59%--ftrace_test_record
ftrace_replace_code ftrace_modify_all_code
|
|--0.87%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.72%--ftrace_shutdown.part.0
unregister_ftrace_function unregister_ftrace_direct bpf_trampoline_update bpf_trampoline_unlink_prog bpf_tracing_link_release bpf_link_free bpf_link_release __fput __x64_sys_close do_syscall_64 entry_SYSCALL_64_after_hwframe __syscall_cancel_arch_end __syscall_cancel __close fentry_test_common fentry_test test_fentry_test run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start

10.98% test_progs [kernel.kallsyms] [k] __ftrace_hash_update_ipmodify
|
|--7.90%--__ftrace_hash_update_ipmodify
| |
| |--4.27%--ftrace_shutdown.part.0
| | unregister_ftrace_function
| | unregister_ftrace_direct
| | bpf_trampoline_update
| | bpf_trampoline_unlink_prog
| | bpf_tracing_link_release
| | bpf_link_free
| | bpf_link_release
| | __fput
| | __x64_sys_close
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | __syscall_cancel_arch_end
| | __syscall_cancel
| | __close
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --3.63%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--3.06%--ftrace_lookup_ip
__ftrace_hash_update_ipmodify
|
|--1.92%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--1.14%--ftrace_shutdown.part.0
unregister_ftrace_function unregister_ftrace_direct bpf_trampoline_update bpf_trampoline_unlink_prog bpf_tracing_link_release bpf_link_free bpf_link_release __fput __x64_sys_close do_syscall_64 entry_SYSCALL_64_after_hwframe __syscall_cancel_arch_end __syscall_cancel __close fentry_test_common fentry_test test_fentry_test run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start

6.41% test_progs [kernel.kallsyms] [k] __ftrace_hash_rec_update
|
|--3.37%--__ftrace_hash_rec_update
| |
| |--1.90%--ftrace_startup
| | register_ftrace_function_nolock
| | register_ftrace_direct
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_link_prog
| | bpf_tracing_prog_attach
| | bpf_raw_tp_link_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | skel_raw_tracepoint_open
| | fentry_test_lskel__test1__attach
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --1.47%--ftrace_shutdown.part.0
| unregister_ftrace_function
| unregister_ftrace_direct
| bpf_trampoline_update
| bpf_trampoline_unlink_prog
| bpf_tracing_link_release
| bpf_link_free
| bpf_link_release
| __fput
| __x64_sys_close
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| __syscall_cancel_arch_end
| __syscall_cancel
| __close
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
|--2.16%--ftrace_lookup_ip
| __ftrace_hash_rec_update
| |
| |--1.16%--ftrace_shutdown.part.0
| | unregister_ftrace_function
| | unregister_ftrace_direct
| | bpf_trampoline_update
| | bpf_trampoline_unlink_prog
| | bpf_tracing_link_release
| | bpf_link_free
| | bpf_link_release
| | __fput
| | __x64_sys_close
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | __syscall_cancel_arch_end
| | __syscall_cancel
| | __close
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --0.99%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.88%--srso_alias_safe_ret
|
--0.79%--__ftrace_hash_rec_update
|
--0.52%--ftrace_shutdown.part.0
unregister_ftrace_function unregister_ftrace_direct bpf_trampoline_update bpf_trampoline_unlink_prog bpf_tracing_link_release bpf_link_free bpf_link_release __fput __x64_sys_close do_syscall_64 entry_SYSCALL_64_after_hwframe __syscall_cancel_arch_end __syscall_cancel __close fentry_test_common fentry_test test_fentry_test run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start

4.69% test_progs [kernel.kallsyms] [k] ftrace_check_record
|
|--2.04%--ftrace_check_record
| ftrace_replace_code
| ftrace_modify_all_code
| |
| |--1.06%--ftrace_startup
| | register_ftrace_function_nolock
| | register_ftrace_direct
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_link_prog
| | bpf_tracing_prog_attach
| | bpf_raw_tp_link_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | skel_raw_tracepoint_open
| | fentry_test_lskel__test1__attach
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --0.98%--ftrace_shutdown.part.0
| unregister_ftrace_function
| unregister_ftrace_direct
| bpf_trampoline_update
| bpf_trampoline_unlink_prog
| bpf_tracing_link_release
| bpf_link_free
| bpf_link_release
| __fput
| __x64_sys_close
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| __syscall_cancel_arch_end
| __syscall_cancel
| __close
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--1.28%--ftrace_replace_code
ftrace_modify_all_code
|
--0.81%--ftrace_startup
register_ftrace_function_nolock register_ftrace_direct bpf_trampoline_update __bpf_trampoline_link_prog bpf_trampoline_link_prog bpf_tracing_prog_attach bpf_raw_tp_link_attach __sys_bpf __x64_sys_bpf do_syscall_64 entry_SYSCALL_64_after_hwframe syscall skel_raw_tracepoint_open fentry_test_lskel__test1__attach fentry_test_common fentry_test test_fentry_test run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start

4.59% test_progs [kernel.kallsyms] [k] ftrace_lookup_ip
| |--1.99%--__ftrace_hash_update_ipmodify
| |
| |--1.03%--ftrace_shutdown.part.0
| | unregister_ftrace_function
| | unregister_ftrace_direct
| | bpf_trampoline_update
| | bpf_trampoline_unlink_prog
| | bpf_tracing_link_release
| | bpf_link_free
| | bpf_link_release
| | __fput
| | __x64_sys_close
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | __syscall_cancel_arch_end
| | __syscall_cancel
| | __close
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --0.96%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
|--1.67%--ftrace_lookup_ip
| |
| --1.19%--__ftrace_hash_update_ipmodify
| |
| |--0.60%--ftrace_shutdown.part.0
| | unregister_ftrace_function
| | unregister_ftrace_direct
| | bpf_trampoline_update
| | bpf_trampoline_unlink_prog
| | bpf_tracing_link_release
| | bpf_link_free
| | bpf_link_release
| | __fput
| | __x64_sys_close
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | __syscall_cancel_arch_end
| | __syscall_cancel
| | __close
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --0.59%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.81%--__ftrace_hash_rec_update

3.65% swapper [kernel.kallsyms] [k] acpi_os_read_port
|
|--1.03%--acpi_os_read_port
| acpi_hw_read_port
| acpi_hw_read
| acpi_hw_register_read
| acpi_read_bit_register
| acpi_idle_enter_bm
| cpuidle_enter_state
| cpuidle_enter
| do_idle
| cpu_startup_entry
| |
| --0.97%--start_secondary
| common_startup_64
|
|--0.82%--srso_alias_safe_ret
|
--0.74%--acpi_hw_read
acpi_hw_register_read acpi_read_bit_register acpi_idle_enter_bm cpuidle_enter_state cpuidle_enter do_idle cpu_startup_entry
|
--0.74%--start_secondary
common_startup_64

3.40% test_progs [kernel.kallsyms] [k] srso_alias_return_thunk
|
|--0.85%--ftrace_replace_code
| ftrace_modify_all_code
| |
| --0.51%--ftrace_startup
| register_ftrace_function_nolock
| register_ftrace_direct
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_link_prog
| bpf_tracing_prog_attach
| bpf_raw_tp_link_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| skel_raw_tracepoint_open
| fentry_test_lskel__test1__attach
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.64%--ftrace_check_record
ftrace_replace_code
ftrace_modify_all_code

2.97% test_progs [kernel.kallsyms] [k] srso_alias_safe_ret
|
|--0.73%--ftrace_check_record
| ftrace_replace_code
| ftrace_modify_all_code
|
--0.69%--ftrace_replace_code
ftrace_modify_all_code

2.67% test_progs [kernel.kallsyms] [k] ftrace_rec_iter_record
|
|--1.19%--ftrace_replace_code
| ftrace_modify_all_code
| |
| |--0.68%--ftrace_startup
| | register_ftrace_function_nolock
| | register_ftrace_direct
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_link_prog
| | bpf_tracing_prog_attach
| | bpf_raw_tp_link_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | skel_raw_tracepoint_open
| | fentry_test_lskel__test1__attach
| | fentry_test_common
| | fentry_test
| | test_fentry_test
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --0.51%--ftrace_shutdown.part.0
| unregister_ftrace_function
| unregister_ftrace_direct
| bpf_trampoline_update
| bpf_trampoline_unlink_prog
| bpf_tracing_link_release
| bpf_link_free
| bpf_link_release
| __fput
| __x64_sys_close
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| __syscall_cancel_arch_end
| __syscall_cancel
| __close
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.69%--ftrace_check_record
ftrace_replace_code
ftrace_modify_all_code

2.05% test_progs [kernel.kallsyms] [k] ftrace_test_record
|
--0.79%--ftrace_replace_code
ftrace_modify_all_code

1.83% test_progs [kernel.kallsyms] [k] ftrace_rec_iter_next
|
--0.87%--ftrace_replace_code
ftrace_modify_all_code
|
--0.51%--ftrace_startup
register_ftrace_function_nolock register_ftrace_direct bpf_trampoline_update __bpf_trampoline_link_prog bpf_trampoline_link_prog bpf_tracing_prog_attach bpf_raw_tp_link_attach __sys_bpf __x64_sys_bpf do_syscall_64 entry_SYSCALL_64_after_hwframe syscall skel_raw_tracepoint_open fentry_test_lskel__test1__attach fentry_test_common fentry_test test_fentry_test run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start

1.76% test_progs [kernel.kallsyms] [k] smp_call_function_many_cond
|
--1.73%--smp_call_function_many_cond
on_each_cpu_cond_mask
|
--1.57%--smp_text_poke_batch_finish
|
--1.55%--ftrace_modify_all_code
| |--0.91%--ftrace_shutdown.part.0
| unregister_ftrace_function
| unregister_ftrace_direct
| bpf_trampoline_update
| bpf_trampoline_unlink_prog
| bpf_tracing_link_release
| bpf_link_free
| bpf_link_release
| __fput
| __x64_sys_close
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| __syscall_cancel_arch_end
| __syscall_cancel
| __close
| fentry_test_common
| fentry_test
| test_fentry_test
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.64%--ftrace_startup
register_ftrace_function_nolock register_ftrace_direct bpf_trampoline_update __bpf_trampoline_link_prog bpf_trampoline_link_prog bpf_tracing_prog_attach bpf_raw_tp_link_attach __sys_bpf __x64_sys_bpf do_syscall_64 entry_SYSCALL_64_after_hwframe syscall skel_raw_tracepoint_open fentry_test_lskel__test1__attach fentry_test_common fentry_test test_fentry_test run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start

1.05% rcu_tasks_kthre [kernel.kallsyms] [k] rcu_tasks_pertask
|
--0.65%--rcu_tasks_wait_gp
rcu_tasks_one_gp
rcu_tasks_kthread
kthread
ret_from_fork
ret_from_fork_asm

0.70% test_progs [kernel.kallsyms] [k] btf_find_by_name_kind
|
--0.59%--btf_find_by_name_kind