Skip to contentSkip to conversation

infiniband/hw/ocrdma: increase frame warning limit in verifier when using KASAN or KCSAN

PATCH In review [email protected] 3 participants 6 messages Jul 9, 2024 – Jul 10, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 infiniband/hw/ocrdma: increase frame warning limit in verifier when using KASAN or KCSAN

From: Peng Hao <[email protected]>

When building kernel with clang, which will typically
have sanitizers enabled, there is a warning about a large stack frame.

drivers/infiniband/hw/ocrdma/ocrdma_stats.c:686:16: error: stack frame size (20664) exceeds limit (8192) in 'ocrdma_dbgfs_ops_read' [-Werror,-Wframe-larger-than] static ssize_t ocrdma_dbgfs_ops_read(struct file *filp, char __user *buffer,

               ^

Increase the frame size by 20% to set.

---
 drivers/infiniband/hw/ocrdma/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/hw/ocrdma/Makefile +6 -0
index 14fba95021d8..a1e9fcc04751 100644
@@ -3,4 +3,10 @@ ccflags-y := -I $(srctree)/drivers/net/ethernet/emulex/benet
 
 obj-$(CONFIG_INFINIBAND_OCRDMA)	+= ocrdma.o
 
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(filter y,$(CONFIG_KASAN)$(CONFIG_KCSAN)),y)
+CFLAGS_ocrdma_stats.o = -Wframe-larger-than=22664
+endif
+endif
+
 ocrdma-y :=	ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o ocrdma_stats.o
Signature · 1 line
2.27.0
#1 lore raw
Quoted 8 lines · On Tue, Jul 09, 2024 at 06:52:42PM +0800, [email protected] wrote:

From: Peng Hao <[email protected]>

When building kernel with clang, which will typically
have sanitizers enabled, there is a warning about a large stack frame.

drivers/infiniband/hw/ocrdma/ocrdma_stats.c:686:16: error: stack frame size (20664) exceeds limit (8192) in 'ocrdma_dbgfs_ops_read' [-Werror,-Wframe-larger-than] static ssize_t ocrdma_dbgfs_ops_read(struct file *filp, char __user *buffer,

               ^

Please fix it, not hide it.

Thanks

Quoted 27 lines

Increase the frame size by 20% to set.

---
 drivers/infiniband/hw/ocrdma/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/hw/ocrdma/Makefile +6 -0
index 14fba95021d8..a1e9fcc04751 100644
@@ -3,4 +3,10 @@ ccflags-y := -I $(srctree)/drivers/net/ethernet/emulex/benet
 
 obj-$(CONFIG_INFINIBAND_OCRDMA)	+= ocrdma.o
 
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(filter y,$(CONFIG_KASAN)$(CONFIG_KCSAN)),y)
+CFLAGS_ocrdma_stats.o = -Wframe-larger-than=22664
+endif
+endif
+
 ocrdma-y :=	ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o ocrdma_stats.o
Signature · 1 line
2.27.0
#2 lore raw
Quoted 11 lines · Leon Romanovsky
Quoted 8 lines · On Tue, Jul 09, 2024 at 06:52:42PM +0800, [email protected] wrote:

From: Peng Hao <[email protected]>

When building kernel with clang, which will typically
have sanitizers enabled, there is a warning about a large stack frame.

drivers/infiniband/hw/ocrdma/ocrdma_stats.c:686:16: error: stack frame size (20664) exceeds limit (8192) in 'ocrdma_dbgfs_ops_read' [-Werror,-Wframe-larger-than] static ssize_t ocrdma_dbgfs_ops_read(struct file *filp, char __user *buffer,

               ^

Please fix it, not hide it.

Agreed, this is far from an acceptable solution. No details were provided around compiler, architecture, or configuration, so I can only speculate what is happening here. From reading the code, I suspect that ocrdma_add_stat() is getting inlined into all of its callsites but the stack slot for buff[128] is not getting reused, which may be related to a missing lifetime marker like [1] or sanitizer instrumentation. I am guessing that marking ocrdma_dbgfs_ops_read() as noinline_for_stack would resolve this.

static noinline_for_stack int ocrdma_add_stat(char *start, char *pcur,

If this is not tolerable for all configurations, it could be made more pointed with something like

static
#if defined(CONFIG_CC_IS_CLANG) && (defined(CONFIG_KASAN) || defined(CONFIG_KCSAN)) noinline_for_stack
#endif
int ocrdma_add_stat(char *start, char *pcur,

but I am aware that is quite ugly.

[1]: https://github.com/llvm/llvm-project/issues/38157#issuecomment-1756321571

Cheers,
Nathan

Quoted 26 lines
Quoted 26 lines

Increase the frame size by 20% to set.

---
 drivers/infiniband/hw/ocrdma/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/hw/ocrdma/Makefile +6 -0
index 14fba95021d8..a1e9fcc04751 100644
@@ -3,4 +3,10 @@ ccflags-y := -I $(srctree)/drivers/net/ethernet/emulex/benet
 
 obj-$(CONFIG_INFINIBAND_OCRDMA)	+= ocrdma.o
 
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(filter y,$(CONFIG_KASAN)$(CONFIG_KCSAN)),y)
+CFLAGS_ocrdma_stats.o = -Wframe-larger-than=22664
+endif
+endif
+
 ocrdma-y :=	ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o ocrdma_stats.o
Signature · 1 line
2.27.0
#3 lore raw
Quoted 22 lines · Nathan Chancellor
Quoted 11 lines · Leon Romanovsky
Quoted 8 lines · On Tue, Jul 09, 2024 at 06:52:42PM +0800, [email protected] wrote:

From: Peng Hao <[email protected]>

When building kernel with clang, which will typically
have sanitizers enabled, there is a warning about a large stack frame.

drivers/infiniband/hw/ocrdma/ocrdma_stats.c:686:16: error: stack frame size (20664) exceeds limit (8192) in 'ocrdma_dbgfs_ops_read' [-Werror,-Wframe-larger-than] static ssize_t ocrdma_dbgfs_ops_read(struct file *filp, char __user *buffer,

               ^

Please fix it, not hide it.

Agreed, this is far from an acceptable solution. No details were provided around compiler, architecture, or configuration, so I can only speculate what is happening here. From reading the code, I suspect that ocrdma_add_stat() is getting inlined into all of its callsites but the stack slot for buff[128] is not getting reused, which may be related to a missing lifetime marker like [1] or sanitizer instrumentation. I am guessing that marking ocrdma_dbgfs_ops_read() as noinline_for_stack would resolve this.

Quoted 1 line

static noinline_for_stack int ocrdma_add_stat(char *start, char *pcur,

This is a good solution, but first let's see if the author can provide more details.

Quoted 43 lines

If this is not tolerable for all configurations, it could be made more pointed with something like

static
#if defined(CONFIG_CC_IS_CLANG) && (defined(CONFIG_KASAN) || defined(CONFIG_KCSAN)) noinline_for_stack
#endif
int ocrdma_add_stat(char *start, char *pcur,

but I am aware that is quite ugly.

[1]: https://github.com/llvm/llvm-project/issues/38157#issuecomment-1756321571

Cheers,
Nathan

Quoted 26 lines
Quoted 26 lines

Increase the frame size by 20% to set.

---
 drivers/infiniband/hw/ocrdma/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/hw/ocrdma/Makefile +6 -0
index 14fba95021d8..a1e9fcc04751 100644
@@ -3,4 +3,10 @@ ccflags-y := -I $(srctree)/drivers/net/ethernet/emulex/benet
 
 obj-$(CONFIG_INFINIBAND_OCRDMA)	+= ocrdma.o
 
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(filter y,$(CONFIG_KASAN)$(CONFIG_KCSAN)),y)
+CFLAGS_ocrdma_stats.o = -Wframe-larger-than=22664
+endif
+endif
+
 ocrdma-y :=	ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o ocrdma_stats.o
Signature · 1 line
2.27.0
author
#4 lore raw
Quoted 15 lines · Leon Romanovsky
Quoted 13 lines · Nathan Chancellor
Quoted 11 lines · Leon Romanovsky
Quoted 8 lines · On Tue, Jul 09, 2024 at 06:52:42PM +0800, [email protected] wrote:

From: Peng Hao <[email protected]>

When building kernel with clang, which will typically
have sanitizers enabled, there is a warning about a large stack frame.

drivers/infiniband/hw/ocrdma/ocrdma_stats.c:686:16: error: stack frame size (20664) exceeds limit (8192) in 'ocrdma_dbgfs_ops_read' [-Werror,-Wframe-larger-than] static ssize_t ocrdma_dbgfs_ops_read(struct file *filp, char __user *buffer,

               ^

Please fix it, not hide it.

Quoted 9 lines
Quoted 9 lines

Agreed, this is far from an acceptable solution. No details were provided around compiler, architecture, or configuration, so I can only speculate what is happening here. From reading the code, I suspect that ocrdma_add_stat() is getting inlined into all of its callsites but the stack slot for buff[128] is not getting reused, which may be related to a missing lifetime marker like [1] or sanitizer instrumentation. I am guessing that marking ocrdma_dbgfs_ops_read() as noinline_for_stack would resolve this.

KASAN is generally enabled for testing environments; noinline_for_stack is used to solve the problem of needing to
track deeper call chains. I also tried to use noinline_for_stack and
issued a new patch, but I think the v1 method is better. My environment is:
x86_64, clang version 15.0.7
CONFIG_FRAME_WARN=8192

Quoted 48 lines
Quoted 1 line

static noinline_for_stack int ocrdma_add_stat(char *start, char *pcur,

This is a good solution, but first let's see if the author can provide more details.

Quoted 43 lines

If this is not tolerable for all configurations, it could be made more pointed with something like

static
#if defined(CONFIG_CC_IS_CLANG) && (defined(CONFIG_KASAN) || defined(CONFIG_KCSAN)) noinline_for_stack
#endif
int ocrdma_add_stat(char *start, char *pcur,

but I am aware that is quite ugly.

[1]: https://github.com/llvm/llvm-project/issues/38157#issuecomment-1756321571

Cheers,
Nathan

Quoted 26 lines
Quoted 26 lines

Increase the frame size by 20% to set.

---
 drivers/infiniband/hw/ocrdma/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/hw/ocrdma/Makefile +6 -1
index 14fba95021d8..a1e9fcc04751 100644
@@ -3,4 +3,10 @@ ccflags-y := -I $(srctree)/drivers/net/ethernet/emulex/benet
 
 obj-$(CONFIG_INFINIBAND_OCRDMA)  += ocrdma.o
 
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(filter y,$(CONFIG_KASAN)$(CONFIG_KCSAN)),y)
+CFLAGS_ocrdma_stats.o = -Wframe-larger-than=22664
+endif
+endif
+
 ocrdma-y :=      ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o ocrdma_stats.o
--

2.27.0

#5 lore raw
Quoted 16 lines · Hao Peng
Quoted 15 lines · Leon Romanovsky
Quoted 13 lines · Nathan Chancellor
Quoted 11 lines · Leon Romanovsky
Quoted 8 lines · On Tue, Jul 09, 2024 at 06:52:42PM +0800, [email protected] wrote:

From: Peng Hao <[email protected]>

When building kernel with clang, which will typically
have sanitizers enabled, there is a warning about a large stack frame.

drivers/infiniband/hw/ocrdma/ocrdma_stats.c:686:16: error: stack frame size (20664) exceeds limit (8192) in 'ocrdma_dbgfs_ops_read' [-Werror,-Wframe-larger-than] static ssize_t ocrdma_dbgfs_ops_read(struct file *filp, char __user *buffer,

               ^

Please fix it, not hide it.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Agreed, this is far from an acceptable solution. No details were provided around compiler, architecture, or configuration, so I can only speculate what is happening here. From reading the code, I suspect that ocrdma_add_stat() is getting inlined into all of its callsites but the stack slot for buff[128] is not getting reused, which may be related to a missing lifetime marker like [1] or sanitizer instrumentation. I am guessing that marking ocrdma_dbgfs_ops_read() as noinline_for_stack

This should have been ocrdma_add_stat()...

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

would resolve this.

KASAN is generally enabled for testing environments; noinline_for_stack is used to solve the problem of needing to
track deeper call chains. I also tried to use noinline_for_stack and
issued a new patch, but I think the v1 method is better.

As far as I understand it, this stack usage would be a problem at runtime if running without VMAP_STACK, so I am going to disagree. v2 gets more at the heart of the problem (although I wonder if that just moves the high stack usage to those other functions under the limit).

Quoted 3 lines

My environment is:
x86_64, clang version 15.0.7
CONFIG_FRAME_WARN=8192

Is CONFIG_KASAN_STACK enabled? I realized that could be another reason you see this warning, as that option is known to have high stack usage with clang.

Quoted 48 lines
Quoted 48 lines
Quoted 1 line

static noinline_for_stack int ocrdma_add_stat(char *start, char *pcur,

This is a good solution, but first let's see if the author can provide more details.

Quoted 43 lines

If this is not tolerable for all configurations, it could be made more pointed with something like

static
#if defined(CONFIG_CC_IS_CLANG) && (defined(CONFIG_KASAN) || defined(CONFIG_KCSAN)) noinline_for_stack
#endif
int ocrdma_add_stat(char *start, char *pcur,

but I am aware that is quite ugly.

[1]: https://github.com/llvm/llvm-project/issues/38157#issuecomment-1756321571

Cheers,
Nathan

Quoted 26 lines
Quoted 26 lines

Increase the frame size by 20% to set.

---
 drivers/infiniband/hw/ocrdma/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/hw/ocrdma/Makefile +6 -1
index 14fba95021d8..a1e9fcc04751 100644
@@ -3,4 +3,10 @@ ccflags-y := -I $(srctree)/drivers/net/ethernet/emulex/benet
 
 obj-$(CONFIG_INFINIBAND_OCRDMA)  += ocrdma.o
 
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(filter y,$(CONFIG_KASAN)$(CONFIG_KCSAN)),y)
+CFLAGS_ocrdma_stats.o = -Wframe-larger-than=22664
+endif
+endif
+
 ocrdma-y :=      ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o ocrdma_stats.o
--

2.27.0