kernel-hacking: introduce CONFIG_NO_AUTO_INLINE
Subsystems (12)
v2 cover kernel-hacking: introduce CONFIG_NO_AUTO_INLINE
This series introduces a new kernel configuration option NO_AUTO_INLINE, which can be used to disable the automatic inlining of functions.
This will allow the function tracer to trace more functions because it only traces functions that the compiler has not inlined.
Previous discussions can be found at
And v1 of this series can be found at
This patch depends on
[PATCH] drm/i915/pxp: fix undefined reference to
`intel_pxp_gsccs_is_ready_for_sessions'
which can be found at
https://lore.kernel.org/all/[email protected]/
as well as
[RFC PATCH 5/7] RDMA/hns: initialize db in update_srq_db()
which can be found at
https://lore.kernel.org/all/FF922C77946229B6+20250411105459.90782-5-chenlinxuan@uniontech.com/
---
Winston Wen (5):
nvme: add __always_inline for nvme_pci_npages_prp
mm: add __always_inline for page_contains_unaccepted vfio/virtio: add __always_inline for virtiovf_get_device_config_size
tpm: add __always_inline for tpm_is_hwrng_enabled lib/Kconfig.debug: introduce CONFIG_NO_AUTO_INLINE
Makefile | 6 ++++++ drivers/char/tpm/tpm-chip.c | 2 +- drivers/nvme/host/pci.c | 2 +- drivers/vfio/pci/virtio/legacy_io.c | 2 +- lib/Kconfig.debug | 15 +++++++++++++++ mm/page_alloc.c | 2 +- 6 files changed, 25 insertions(+), 4 deletions(-)
---
base-commit: 1a1d569a75f3ab2923cb62daf356d102e4df2b86
change-id: 20250416-noautoinline-8e9b9e535452
Best regards,
Signature · 1 line
Chen Linxuan <[email protected]>
RFC PATCH v2 1/5 nvme: add __always_inline for nvme_pci_npages_prp
From: Winston Wen <[email protected]>
On x86_64 with gcc version 13.3.0, I build drivers/nvme/host/pci.c with:
make defconfig
./scripts/kconfig/merge_config.sh .config <(
echo CONFIG_BLK_DEV_NVME=m
)
make KCFLAGS="-fno-inline-small-functions -fno-inline-functions-called-once" \
drivers/nvme/host/pci.oThen I get a compile error:
CALL scripts/checksyscalls.sh
DESCEND objtool
INSTALL libsubcmd_headers
CC drivers/nvme/host/pci.o
In file included from <command-line>:
drivers/nvme/host/pci.c: In function 'nvme_init':
././include/linux/compiler_types.h:557:45: error: call to '__compiletime_assert_878' declared with attribute error: BUILD_BUG_ON failed: nvme_pci_npages_prp() > NVME_MAX_NR_ALLOCATIONS
557 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
././include/linux/compiler_types.h:538:25: note: in definition of macro '__compiletime_assert'
538 | prefix ## suffix(); \
| ^~~~~~
././include/linux/compiler_types.h:557:9: note: in expansion of macro '_compiletime_assert'
557 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
| ^~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:50:9: note: in expansion of macro 'BUILD_BUG_ON_MSG'
50 | BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
| ^~~~~~~~~~~~~~~~
drivers/nvme/host/pci.c:3804:9: note: in expansion of macro 'BUILD_BUG_ON'
3804 | BUILD_BUG_ON(nvme_pci_npages_prp() > NVME_MAX_NR_ALLOCATIONS);
| ^~~~~~~~~~~~Mark nvme_pci_npages_prp() with __always_inline make it can be computed at compile time.
--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +1 -1
@@ -390,7 +390,7 @@ static bool nvme_dbbuf_update_and_check_event(u16 value, __le32 *dbbuf_db,* as it only leads to a small amount of wasted memory for the lifetime of * the I/O. */ -static int nvme_pci_npages_prp (void ) +static __always_inline int nvme_pci_npages_prp(void) {unsigned max_bytes = (NVME_MAX_KB_SZ *1024 ) +NVME_CTRL_PAGE_SIZE ;unsigned nprps =DIV_ROUND_UP (max_bytes,NVME_CTRL_PAGE_SIZE );
Signature · 1 line
2.48.1
RFC PATCH v2 2/5 mm: add __always_inline for page_contains_unaccepted
In reply to Chen LinxuanFrom: Winston Wen <[email protected]>
On x86_64 with gcc version 13.3.0, I compile mm/page_alloc.c with:
make defconfig
make KCFLAGS="-fno-inline-small-functions -fno-inline-functions-called-once" \
mm/page_alloc.oThen I get a compile error:
CALL scripts/checksyscalls.sh
DESCEND objtool
INSTALL libsubcmd_headers
CC mm/page_alloc.o
In file included from <command-line>:
mm/page_alloc.c: In function '__free_unaccepted.isra':
././include/linux/compiler_types.h:557:45: error: call to '__compiletime_assert_1013' declared with attribute error: BUILD_BUG failed
557 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
././include/linux/compiler_types.h:538:25: note: in definition of macro '__compiletime_assert'
538 | prefix ## suffix(); \
| ^~~~~~
././include/linux/compiler_types.h:557:9: note: in expansion of macro '_compiletime_assert'
557 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
| ^~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:59:21: note: in expansion of macro 'BUILD_BUG_ON_MSG'
59 | #define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed")
| ^~~~~~~~~~~~~~~~
mm/page_alloc.c:7301:9: note: in expansion of macro 'BUILD_BUG'
7301 | BUILD_BUG();
| ^~~~~~~~~Marking page_contains_unaccepted with __always_inline and let dead code elimination remove reference to __free_unaccepted() here.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -7286,7 +7286,7 @@ static bool __free_unaccepted(struct page *page)#else -static bool page_contains_unaccepted (struct page *page,unsigned int order) +static __always_inline bool page_contains_unaccepted(struct page *page, unsigned int order) {return false; }
Signature · 1 line
2.48.1
RFC PATCH v2 3/5 vfio/virtio: add __always_inline for virtiovf_get_device_config_size
In reply to Chen LinxuanFrom: Winston Wen <[email protected]>
Presume that kernel is compiled for x86_64 with gcc version 13.3.0:
make defconfig
./scripts/kconfig/merge_config.sh .config <(
echo CONFIG_VFIO=m
echo CONFIG_VIRTIO_PCI=y
echo CONFIG_VIRTIO_PCI_LIB_LEGACY=y
echo CONFIG_VIRTIO_VFIO_PCI=m
echo CONFIG_VIRTIO_VFIO_PCI_ADMIN_LEGACY=y
)
make KCFLAGS="-fno-inline-small-functions -fno-inline-functions-called-once" \
drivers/vfio/pci/virtio/legacy_io.oThis results a compile error:
CALL scripts/checksyscalls.sh
DESCEND objtool
INSTALL libsubcmd_headers
CC drivers/vfio/pci/virtio/legacy_io.o
In file included from <command-line>:
drivers/vfio/pci/virtio/legacy_io.c: In function 'virtiovf_init_legacy_io':
././include/linux/compiler_types.h:557:45: error: call to '__compiletime_assert_889' declared with attribute error: BUILD_BUG_ON failed: !is_power_of_2(virtvdev->bar0_virtual_buf_size)
557 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
././include/linux/compiler_types.h:538:25: note: in definition of macro '__compiletime_assert'
538 | prefix ## suffix(); \
| ^~~~~~
././include/linux/compiler_types.h:557:9: note: in expansion of macro '_compiletime_assert'
557 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
| ^~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:50:9: note: in expansion of macro 'BUILD_BUG_ON_MSG'
50 | BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
| ^~~~~~~~~~~~~~~~
drivers/vfio/pci/virtio/legacy_io.c:401:9: note: in expansion of macro 'BUILD_BUG_ON'
401 | BUILD_BUG_ON(!is_power_of_2(virtvdev->bar0_virtual_buf_size));
| ^~~~~~~~~~~~BUILD_BUG_ON needs virtvdev->bar0_virtual_buf_size to be computed at compile time. So we should mark virtiovf_get_device_config_size() with __always_inline here.
--- drivers/vfio/pci/virtio/legacy_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/pci/virtio/legacy_io.c +1 -1
@@ -350,7 +350,7 @@ int virtiovf_open_legacy_io(struct virtiovf_pci_core_device *virtvdev)return virtiovf_set_notify_addr (virtvdev); } -static int virtiovf_get_device_config_size (unsigned short device) +static __always_inline int virtiovf_get_device_config_size(unsigned short device) {/* Network card */ return offsetofend(struct virtio_net_config, status);
Signature · 1 line
2.48.1
RFC PATCH v2 4/5 tpm: add __always_inline for tpm_is_hwrng_enabled
In reply to Chen LinxuanFrom: Winston Wen <[email protected]>
Presume that kernel is compiled for x86_64 with gcc version 13.3.0:
make defconfig
./scripts/kconfig/merge_config.sh .config <(
echo CONFIG_TCG_TPM=y
echo CONFIG_HW_RANDOM=m
)
make KCFLAGS="-fno-inline-small-functions -fno-inline-functions-called-once"
This results a link error:
ld: vmlinux.o: in function `tpm_add_hwrng':
tpm-chip.c:(.text+0x6c5924): undefined reference to `hwrng_register'
ld: vmlinux.o: in function `tpm_chip_unregister':
(.text+0x6c5bc9): undefined reference to `hwrng_unregister'
ld: vmlinux.o: in function `tpm_chip_register':
(.text+0x6c5c9b): undefined reference to `hwrng_unregister'
With CONFIG_TCG_TPM=y and CONFIG_HW_RANDOM=m,
the functions tpm_add_hwrng, tpm_chip_unregister, and tpm_chip_register are compiled into vmlinux.o
and reference the symbols hwrng_register and hwrng_unregister. These symbols, however, are compiled into rng-core.ko, which results in the linking error.
I am not sure but I think this weird linking error only arises when auto inlining is disabled because of some dead code elimination.
CONFIG_TCG_TPM=y and CONFIG_HW_RANDOM=m set CONFIG_HW_RANDOM_TPM=n. This causes the function tpm_is_hwrng_enabled to always return false, as shown below:
static bool tpm_is_hwrng_enabled(struct tpm_chip *chip)
{
if (!IS_ENABLED(CONFIG_HW_RANDOM_TPM))
return false;
if (tpm_is_firmware_upgrade(chip))
return false;
if (chip->flags & TPM_CHIP_FLAG_HWRNG_DISABLED)
return false;
return true;
}When tpm_is_hwrng_enabled is inlined, dead code elimination optimizations are applied and the reference to the hwrng_* functions will been removed.
For instance, in the tpm_chip_unregister function:
void tpm_chip_unregister(struct tpm_chip *chip)
{
#ifdef CONFIG_TCG_TPM2_HMAC
int rc; rc = tpm_try_get_ops(chip);
if (!rc) {
tpm2_end_auth_session(chip);
tpm_put_ops(chip);
}#endif
tpm_del_legacy_sysfs(chip);
if (tpm_is_hwrng_enabled(chip))
hwrng_unregister(&chip->hwrng);
tpm_bios_log_teardown(chip);
if (chip->flags & TPM_CHIP_FLAG_TPM2 && !tpm_is_firmware_upgrade(chip))
tpm_devs_remove(chip);
tpm_del_char_device(chip);}
When tpm_is_hwrng_enabled is inlined and always returns false, the call to hwrng_unregister is effectively part of a if (false) block, which I guess that will be then optimized out.
However, when the -fno-inline-small-functions and -fno-inline-functions-called-once flags are used, tpm_is_hwrng_enabled is not inline.
And this optimization some how cannot occur, leading to the undefined reference errors during linking.
Adding the __always_inline attribute ensures that tpm_is_hwrng_enabled is inlined regardless of the compiler flags. This allows the dead code elimination to proceed as expected, resolving the linking issue.
--- drivers/char/tpm/tpm-chip.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/char/tpm/tpm-chip.c +1 -1
@@ -534,7 +534,7 @@ static int tpm_hwrng_read(struct hwrng *rng, void *data, size_t max, bool wait)return tpm_get_random (chip, data, max); } -static bool tpm_is_hwrng_enabled (struct tpm_chip *chip) +static __always_inline bool tpm_is_hwrng_enabled(struct tpm_chip *chip) {if (!IS_ENABLED (CONFIG_HW_RANDOM_TPM ))return false;
Signature · 1 line
2.48.1
Re: [PATCH RFC v2 4/5] tpm: add __always_inline for tpm_is_hwrng_enabled
Quoted 113 lines · On Wed, Apr 16, 2025 at 05:44:50PM +0800, Chen Linxuan via B4 Relay wrote:
From: Winston Wen <[email protected]>
Presume that kernel is compiled for
x86_64with gcc version 13.3.0:make defconfig
./scripts/kconfig/merge_config.sh.config <(
echoCONFIG_TCG_TPM=y
echoCONFIG_HW_RANDOM=m
)
make KCFLAGS="-fno-inline-small-functions -fno-inline-functions-called-once"This results a link error:
ld: vmlinux.o: in function `tpm_add_hwrng':
tpm-chip.c:(.text+0x6c5924): undefined reference to `hwrng_register'
ld: vmlinux.o: in function `tpm_chip_unregister':
(.text+0x6c5bc9): undefined reference to `hwrng_unregister'
ld: vmlinux.o: in function `tpm_chip_register':
(.text+0x6c5c9b): undefined reference to `hwrng_unregister'With
CONFIG_TCG_TPM=yandCONFIG_HW_RANDOM=m,
the functionstpm_add_hwrng,tpm_chip_unregister, andtpm_chip_registerare compiled intovmlinux.o
and reference the symbolshwrng_registerandhwrng_unregister. These symbols, however, are compiled intorng-core.ko, which results in the linking error.I am not sure but I think this weird linking error only arises when auto inlining is disabled because of some dead code elimination.
CONFIG_TCG_TPM=yandCONFIG_HW_RANDOM=msetCONFIG_HW_RANDOM_TPM=n. This causes the functiontpm_is_hwrng_enabledto always returnfalse, as shown below:static bool tpm_is_hwrng_enabled(struct tpm_chip *chip) { if (!IS_ENABLED(CONFIG_HW_RANDOM_TPM)) return false; if (tpm_is_firmware_upgrade(chip)) return false; if (chip->flags & TPM_CHIP_FLAG_HWRNG_DISABLED) return false; return true; }When
tpm_is_hwrng_enabledis inlined, dead code elimination optimizations are applied and the reference to thehwrng_*functions will been removed.
For instance, in thetpm_chip_unregisterfunction:void tpm_chip_unregister(struct tpm_chip *chip) { #ifdef CONFIG_TCG_TPM2_HMAC int rc;rc = tpm_try_get_ops(chip); if (!rc) { tpm2_end_auth_session(chip); tpm_put_ops(chip); }#endif
tpm_del_legacy_sysfs(chip); if (tpm_is_hwrng_enabled(chip)) hwrng_unregister(&chip->hwrng); tpm_bios_log_teardown(chip); if (chip->flags & TPM_CHIP_FLAG_TPM2 && !tpm_is_firmware_upgrade(chip)) tpm_devs_remove(chip); tpm_del_char_device(chip);}
When
tpm_is_hwrng_enabledis inlined and always returnsfalse, the call tohwrng_unregisteris effectively part of aif (false)block, which I guess that will be then optimized out.However, when the
-fno-inline-small-functionsand-fno-inline-functions-called-onceflags are used,tpm_is_hwrng_enabledis not inline.And this optimization some how cannot occur, leading to the undefined reference errors during linking.
Adding the
__always_inlineattribute ensures thattpm_is_hwrng_enabledis inlined regardless of the compiler flags. This allows the dead code elimination to proceed as expected, resolving the linking issue.Co-developed-by Chen Linxuan <[email protected]>Signed-off-by Chen Linxuan <[email protected]>Signed-off-by Winston Wen <[email protected]>--- drivers/char/tpm/tpm-chip.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/char/tpm/tpm-chip.c +1
-1@@ -534,7 +534,7 @@ static int tpm_hwrng_read(struct hwrng *rng, void *data, size_t max, bool wait)return tpm_get_random (chip, data, max); } -static bool tpm_is_hwrng_enabled (struct tpm_chip *chip) +static __always_inline bool tpm_is_hwrng_enabled(struct tpm_chip *chip) {if (!IS_ENABLED (CONFIG_HW_RANDOM_TPM ))return false;Signature · 2 lines
-- 2.48.1
Thank you.
BR, Jarkko
RFC PATCH v2 5/5 lib/Kconfig.debug: introduce CONFIG_NO_AUTO_INLINE
In reply to Chen LinxuanFrom: Winston Wen <[email protected]>
Add a new kernel hacking option CONFIG_NO_AUTO_INLINE that prevents the compiler from auto-inlining functions not explicitly marked with the 'inline' keyword.
This enhancement improves function tracer capabilities as it can only trace functions that haven't been inlined by the compiler.
Previous discussions:
This patch is modified from commit 917fad29febd ("kernel hacking: add a config option to disable compiler auto-inlining") which can be founded in linux-next-history:
Unlike the original implementation, this patch:
1. Make it depends on CC_IS_GCC,
as Clang 18.1.3 break test_bitmap_const_eval() in lib/test_bitmap.c
2. Make it depends on X86 and LOONGARCH,
as I haven't test other architectures
3. Removes unnecessary cc-option checks per commit 7d73c3e9c514 ("Makefile: remove stale cc-option checks").
4. Update help information.
--- Makefile | 6 ++++++ lib/Kconfig.debug | 15 +++++++++++++++ 2 files changed, 21 insertions(+)
Makefile +6 -0
@@ -1071,6 +1071,12 @@ endif # Ensure compilers do not transform certain loops into calls to wcslen() KBUILD_CFLAGS += -fno-builtin-wcslen +ifdef CONFIG_NO_AUTO_INLINE +KBUILD_CFLAGS += -fno-inline-functions \ + -fno-inline-small-functions \ + -fno-inline-functions-called-once +endif + # change __FILE__ to the relative path to the source directory ifdef building_out_of_srctree KBUILD_CPPFLAGS += $(call cc-option,-ffile-prefix-map=$(srcroot)/=)
lib/Kconfig.debug +15 -0
@@ -436,8 +436,23 @@ config GDB_SCRIPTS instance. See Documentation/process/debugging/gdb-kernel-debugging.rst for further details. + endif # DEBUG_INFO +config NO_AUTO_INLINE + bool "Disable compiler auto-inline optimizations (EXPERIMENTAL)" + default n + depends on CC_IS_GCC && (X86 || LOONGARCH) + help + This will prevent the compiler from optimizing the kernel by + auto-inlining functions not marked with the inline keyword. + With this option, only functions explicitly marked with + "inline" will be inlined. This will allow the function tracer + to trace more functions because it only traces functions that + the compiler has not inlined. + + If unsure, select N. + config FRAME_WARN int "Warn for stack frames larger than" range 0 8192
Signature · 1 line
2.48.1
Re: [PATCH RFC v2 5/5] lib/Kconfig.debug: introduce CONFIG_NO_AUTO_INLINE
Quoted 2 lines · On 4/16/25 2:44 AM, Chen Linxuan via B4 Relay wrote:
2. Make it depends on X86 and LOONGARCH,
as I haven't test other architectures
That sounds weird to me. Shouldn't this option be made architecture- independent?
lib/Kconfig.debug config GDB_SCRIPTS · line 445
Quoted 4 lines
lib/Kconfig.debug +4
-0+config NO_AUTO_INLINE + bool "Disable compiler auto-inline optimizations (EXPERIMENTAL)" + default n + depends on CC_IS_GCC && (X86 || LOONGARCH)
Why "depends on CC_IS_GCC"? Please make sure that both gcc and clang are supported.
Thanks,
Bart.
Re: [PATCH RFC v2 5/5] lib/Kconfig.debug: introduce CONFIG_NO_AUTO_INLINE
Quoted 6 lines · Bart Van Assche
Quoted 2 lines · On 4/16/25 2:44 AM, Chen Linxuan via B4 Relay wrote:
2. Make it depends on X86 and LOONGARCH,
as I haven't test other architecturesThat sounds weird to me. Shouldn't this option be made architecture- independent?
It should, but I have only tested it on X86 and LOONGARCH.
Quoted 8 lines
lib/Kconfig.debug config GDB_SCRIPTS · line 445
Quoted 5 lines
lib/Kconfig.debug +4
-0+config NO_AUTO_INLINE + bool "Disable compiler auto-inline optimizations (EXPERIMENTAL)" + default n + depends on CC_IS_GCC && (X86 || LOONGARCH)Why "depends on
CC_IS_GCC"? Please make sure that both gcc and clang are supported.
I make it depends on CC_IS_GCC because
1. Clang do not have -fno-inline-small-functions and -fno-inline-functions-called-once.
2. If we wrap those options with cc-option,
Clang 18.1.3 fails to compile test_bitmap_const_eval(), with config KASAN and TEST_BITMAP is enabled.
Comments above test_bitmap_const_eval() says that:
Quoted 4 lines
/*
* FIXME: Clang breaks compile-time evaluations when KASAN and GCOV are enabled.
* To workaround it, GCOV is force-disabled inMakefilefor this configuration. */
It seems there are some issues with Clang's compile-time evaluations.
So I think there are some ways to workaround this problem:
1. Make NO_AUTO_INLINE depends on CC_IS_GCC;
2. Make NO_AUTO_INLINE depends on KASAN=n;
3. Disable NO_AUTO_INLINE for TEST_BITMAP
like how we handle CC_IS_CLANG && KASAN && GCOV in lib/Makefile:
Quoted 4 lines
ifeq ($(CONFIG_CC_IS_CLANG)$(CONFIG_KASAN),yy)
# FIXME: Clang breakstest_bitmap_const_evalwhen KASAN and GCOV are enabled GCOV_PROFILE_test_bitmap.o := n
endif
Which one do you prefer or do you have any other suggestions?
Quoted 4 lines
Thanks,
Bart.