Skip to contentSkip to conversation

bpf: force inc_active()/dec_active() to be inline functions

PATCH In review Arnd Bergmann 4 participants 11 messages Jul 22, 2023 – Jul 25, 2023 BPF

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bpf: force inc_active()/dec_active() to be inline functions

From: Arnd Bergmann <[email protected]>

Splitting these out into separate helper functions means that we actually pass an uninitialized variable into another function call if dec_active() happens to not be inlined, and CONFIG_PREEMPT_RT is disabled:

kernel/bpf/memalloc.c: In function 'add_obj_to_free_list':
kernel/bpf/memalloc.c:200:9: error: 'flags' is used uninitialized [-Werror=uninitialized]
200 | dec_active(c, flags);

Mark the two functions as __always_inline so gcc can see through this regardless of optimizations and other options that may prevent it otherwise.

Fixes 18e027b1c7c6d ("bpf: Factor out inc/dec of active flag into helpers.")
---
 kernel/bpf/memalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/memalloc.c +2 -2
index 51d6389e5152e..23906325298da 100644
@@ -165,7 +165,7 @@ static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c)
 #endif
 }
 
-static void inc_active(struct bpf_mem_cache *c, unsigned long *flags)
+static __always_inline void inc_active(struct bpf_mem_cache *c, unsigned long *flags)
 {
 	if (IS_ENABLED(CONFIG_PREEMPT_RT))
 		/* In RT irq_work runs in per-cpu kthread, so disable
@@ -183,7 +183,7 @@ static void inc_active(struct bpf_mem_cache *c, unsigned long *flags)
 	WARN_ON_ONCE(local_inc_return(&c->active) != 1);
 }
 
-static void dec_active(struct bpf_mem_cache *c, unsigned long flags)
+static __always_inline void dec_active(struct bpf_mem_cache *c, unsigned long flags)
 {
 	local_dec(&c->active);
 	if (IS_ENABLED(CONFIG_PREEMPT_RT))
Signature · 1 line
2.39.2
#1 lore raw
Quoted 7 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

Splitting these out into separate helper functions means that we actually pass an uninitialized variable into another function call if dec_active() happens to not be inlined, and CONFIG_PREEMPT_RT is disabled:

Do you mean that the compiler can remove the flags automatically when dec_active() is inlined, but can't remove it automatically when dec_active() is not inlined ?
If so, why can't we improve the compiler ?

If we have to change the kernel, what about the change below?

index 51d6389..17191ee 100644

kernel/bpf/memalloc.c +0 -0
@@ -165,15 +165,17 @@ static struct mem_cgroup *get_memcg(const struct

bpf_mem_cache *c)
#endif
}

-static void inc_active(struct bpf_mem_cache *c, unsigned long *flags) +static unsigned long inc_active(struct bpf_mem_cache *c)
{
+ unsigned long flags = 0;
+
if (IS_ENABLED(CONFIG_PREEMPT_RT))
/* In RT irq_work runs in per-cpu kthread, so disable
* interrupts to avoid preemption and interrupts and
* reduce the chance of bpf prog executing on this cpu
* when active counter is busy.
*/
- local_irq_save(*flags);
+ local_irq_save(flags);
/* alloc_bulk runs from irq_work which will not preempt a bpf
* program that does unit_alloc/unit_free since IRQs are
* disabled there. There is no race to increment 'active'
@@ -181,6 +183,7 @@ static void inc_active(struct bpf_mem_cache *c, unsigned long *flags)

         * bpf prog preempted this loop.
         */
        WARN_ON_ONCE(local_inc_return(&c->active) != 1);

+ return flags;
}

Quoted 41 lines

kernel/bpf/memalloc.c: In function 'add_obj_to_free_list':
kernel/bpf/memalloc.c:200:9: error: 'flags' is used uninitialized [-Werror=uninitialized]
200 | dec_active(c, flags);

Mark the two functions as __always_inline so gcc can see through this regardless of optimizations and other options that may prevent it otherwise.

Fixes 18e027b1c7c6d ("bpf: Factor out inc/dec of active flag into helpers.")
---
 kernel/bpf/memalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/memalloc.c +2 -3
index 51d6389e5152e..23906325298da 100644
@@ -165,7 +165,7 @@ static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c)
 #endif
 }
 
-static void inc_active(struct bpf_mem_cache *c, unsigned long *flags)
+static __always_inline void inc_active(struct bpf_mem_cache *c, unsigned long *flags)
 {
        if (IS_ENABLED(CONFIG_PREEMPT_RT))
                /* In RT irq_work runs in per-cpu kthread, so disable
@@ -183,7 +183,7 @@ static void inc_active(struct bpf_mem_cache *c, unsigned long *flags)
        WARN_ON_ONCE(local_inc_return(&c->active) != 1);
 }
 
-static void dec_active(struct bpf_mem_cache *c, unsigned long flags)
+static __always_inline void dec_active(struct bpf_mem_cache *c, unsigned long flags)
 {
        local_dec(&c->active);
        if (IS_ENABLED(CONFIG_PREEMPT_RT))
--

2.39.2

Signature · 2 lines
Regards
Yafang
#2 lore raw
Quoted 10 lines · Yafang Shao
Quoted 7 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

Splitting these out into separate helper functions means that we actually pass an uninitialized variable into another function call if dec_active() happens to not be inlined, and CONFIG_PREEMPT_RT is disabled:

Quoted 4 lines

Do you mean that the compiler can remove the flags automatically when dec_active() is inlined, but can't remove it automatically when dec_active() is not inlined ?
If so, why can't we improve the compiler ?

Agree.
Sounds like a compiler bug.

Quoted 1 line

If we have to change the kernel, what about the change below?

To workaround the compiler bug we can simply init flag=0 to silence the warn, but even that is silly. Passing flag=0 into irqrestore is buggy.

#3 lore raw
Quoted 4 lines · Alexei Starovoitov
Quoted 3 lines · Yafang Shao
Quoted 2 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Splitting these out into separate helper functions means that we actually pass an uninitialized variable into another function call if dec_active() happens to not be inlined, and CONFIG_PREEMPT_RT is disabled:

Do you mean that the compiler can remove the flags automatically when dec_active() is inlined, but can't remove it automatically when dec_active() is not inlined ?

My educated guess is that it's fine when neither of them are inlined, since then gcc can assume that 'flags' gets initialized by inc_active(), and it's fine when both are inlined since dead code elimination then gets rid of both the initialization and the use.

The only broken case should be when inc_active() is inlined and gcc can tell that there is never an initialization, but dec_active() is not inlined, so gcc assumes it is actually used.

Quoted 4 lines
Quoted 1 line

If so, why can't we improve the compiler ?

Agree.
Sounds like a compiler bug.

I don't know what you might want to change in the compiler to avoid this. Compilers are free to decide which functions to inline in the absence of noinline or always_inline flags.

One difference between gcc and clang is that gcc tries to be smart about warnings by using information from inlining to produce better warnings, while clang never uses information across function boundaries for generated warnings, so it won't find this one, but also would ignore an unconditional use of the uninitialized variable.

Quoted 4 lines
Quoted 1 line

If we have to change the kernel, what about the change below?

To workaround the compiler bug we can simply init flag=0 to silence the warn, but even that is silly. Passing flag=0 into irqrestore is buggy.

Maybe inc_active() could return the flags instead of modifying the stack variable? that would also result in slightly better code when it's not inlined.

Arnd

#4 lore raw
Quoted 9 lines · Arnd Bergmann
Quoted 5 lines
Quoted 2 lines

If so, why can't we improve the compiler ?

Agree.
Sounds like a compiler bug.

I don't know what you might want to change in the compiler to avoid this. Compilers are free to decide which functions to inline in the absence of noinline or always_inline flags.

Clearly a compiler bug.
Compilers should not produce false positive warnings regardless how inlining went and optimizations performed.

Quoted 7 lines

One difference between gcc and clang is that gcc tries to be smart about warnings by using information from inlining to produce better warnings, while clang never uses information across function boundaries for generated warnings, so it won't find this one, but also would ignore an unconditional use of the uninitialized variable.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If we have to change the kernel, what about the change below?

To workaround the compiler bug we can simply init flag=0 to silence the warn, but even that is silly. Passing flag=0 into irqrestore is buggy.

Maybe inc_active() could return the flags instead of modifying the stack variable? that would also result in slightly better code when it's not inlined.

Which gcc are we talking about here that is so buggy?

#5 lore raw
Quoted 7 lines · Alexei Starovoitov
Quoted 6 lines · Arnd Bergmann
Quoted 5 lines
Quoted 2 lines

If so, why can't we improve the compiler ?

Agree.
Sounds like a compiler bug.

Quoted 7 lines
Quoted 3 lines

I don't know what you might want to change in the compiler to avoid this. Compilers are free to decide which functions to inline in the absence of noinline or always_inline flags.

Clearly a compiler bug.
Compilers should not produce false positive warnings regardless how inlining went and optimizations performed.

That would be a nice idea, but until we force everyone to migrate to clang, that's not something in our power. gcc is well known to throw tons of warnings that depend on inlining:
-Wnull-dereference, -Wmaybe-uninitialized, -Wdiv-by-zero and other inherently depend on how much gcc can infer from inlining and dead code elimination.

In this case, it doesn't even require a lot of imagination, the code is literally written as undefined behavior when the first call is inlined and the second one is not, I don't see what one would do in gcc to /not/ warn about passing an uninitialized register into a function call, other than moving the warning before inlining and DCE as clang does.

Quoted 9 lines
Quoted 9 lines

One difference between gcc and clang is that gcc tries to be smart about warnings by using information from inlining to produce better warnings, while clang never uses information across function boundaries for generated warnings, so it won't find this one, but also would ignore an unconditional use of the uninitialized variable.

Quoted 2 lines
Quoted 1 line

If we have to change the kernel, what about the change below?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

To workaround the compiler bug we can simply init flag=0 to silence the warn, but even that is silly. Passing flag=0 into irqrestore is buggy.

Maybe inc_active() could return the flags instead of modifying the stack variable? that would also result in slightly better code when it's not inlined.

Which gcc are we talking about here that is so buggy?

I think I only tried versions 8 through 13 for this one, but can check others as well.

Arnd

#6 lore raw
Quoted 12 lines · Arnd Bergmann
Quoted 12 lines
Quoted 12 lines

One difference between gcc and clang is that gcc tries to be smart about warnings by using information from inlining to produce better warnings, while clang never uses information across function boundaries for generated warnings, so it won't find this one, but also would ignore an unconditional use of the uninitialized variable.

Quoted 4 lines
Quoted 1 line

If we have to change the kernel, what about the change below?

To workaround the compiler bug we can simply init flag=0 to silence the warn, but even that is silly. Passing flag=0 into irqrestore is buggy.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Maybe inc_active() could return the flags instead of modifying the stack variable? that would also result in slightly better code when it's not inlined.

Which gcc are we talking about here that is so buggy?

I think I only tried versions 8 through 13 for this one, but can check others as well.

I have a minimized test case at https://godbolt.org/z/hK4ev17fv
that shows the problem happening with all versions of gcc (4.1 through 14.0) if I force the dec_active() function to be inline and force inc_active() to be non-inline.

With clang, I only see the warning if I turn dec_active() into a macro instead of an inline function. This is the expected behavior in clang.

Arnd

#7 lore raw
Quoted 21 lines · Arnd Bergmann
Quoted 18 lines · Arnd Bergmann
Quoted 17 lines
Quoted 7 lines

One difference between gcc and clang is that gcc tries to be smart about warnings by using information from inlining to produce better warnings, while clang never uses information across function boundaries for generated warnings, so it won't find this one, but also would ignore an unconditional use of the uninitialized variable.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If we have to change the kernel, what about the change below?

To workaround the compiler bug we can simply init flag=0 to silence the warn, but even that is silly. Passing flag=0 into irqrestore is buggy.

Maybe inc_active() could return the flags instead of modifying the stack variable? that would also result in slightly better code when it's not inlined.

Which gcc are we talking about here that is so buggy?

Quoted 7 lines
Quoted 2 lines

I think I only tried versions 8 through 13 for this one, but can check others as well.

I have a minimized test case at https://godbolt.org/z/hK4ev17fv
that shows the problem happening with all versions of gcc (4.1 through 14.0) if I force the dec_active() function to be inline and force inc_active() to be non-inline.

That's a bit of cheating, but I see your point now. How about we do:

kernel/bpf/memalloc.c +0 -0
index 51d6389e5152..3fa0944cb975 100644
@@ -183,11 +183,11 @@ static void inc_active(struct bpf_mem_cache *c,

unsigned long *flags)

        WARN_ON_ONCE(local_inc_return(&c->active) != 1);

}

-static void dec_active(struct bpf_mem_cache *c, unsigned long flags) +static void dec_active(struct bpf_mem_cache *c, unsigned long *flags)

 {
        local_dec(&c->active);
        if (IS_ENABLED(CONFIG_PREEMPT_RT))

- local_irq_restore(flags);
+ local_irq_restore(*flags);
}

static void add_obj_to_free_list(struct bpf_mem_cache *c, void *obj)
@@ -197,7 +197,7 @@ static void add_obj_to_free_list(struct bpf_mem_cache *c, void *obj)

        inc_active(c, &flags);
        __llist_add(obj, &c->free_llist);
        c->free_cnt++;

- dec_active(c, flags);
+ dec_active(c, &flags);

It's symmetrical and there is no 'flags = 0' ugliness.

#8 lore raw
Quoted 17 lines · Alexei Starovoitov
Quoted 6 lines · Arnd Bergmann

On Mon, Jul 24, 2023, at 20:13, Arnd Bergmann wrote:

I have a minimized test case at https://godbolt.org/z/hK4ev17fv
that shows the problem happening with all versions of gcc (4.1 through 14.0) if I force the dec_active() function to be inline and force inc_active() to be non-inline.

That's a bit of cheating, but I see your point now. How about we do:

kernel/bpf/memalloc.c +0 -0
index 51d6389e5152..3fa0944cb975 100644
@@ -183,11 +183,11 @@ static void inc_active(struct bpf_mem_cache *c,
 unsigned long *flags)
        WARN_ON_ONCE(local_inc_return(&c->active) != 1);

kernel/bpf/memalloc.c static void inc_active(struct bpf_mem_cache *c, · line 192

Quoted 10 lines
kernel/bpf/memalloc.c +2 -2
 }
 
-static void dec_active(struct bpf_mem_cache *c, unsigned long flags)
+static void dec_active(struct bpf_mem_cache *c, unsigned long *flags)
 {
        local_dec(&c->active);
        if (IS_ENABLED(CONFIG_PREEMPT_RT))
-               local_irq_restore(flags);
+               local_irq_restore(*flags);
 }

Sure, that's fine. Between this and the two suggestions I had (__always_inline or passing the flags from inc_active as a return code), I don't have a strong preference, so pick whichever you like.

Arnd

#9 lore raw
Quoted 21 lines · Arnd Bergmann
Quoted 19 lines · Alexei Starovoitov
Quoted 6 lines · Arnd Bergmann

On Mon, Jul 24, 2023, at 20:13, Arnd Bergmann wrote:

I have a minimized test case at https://godbolt.org/z/hK4ev17fv
that shows the problem happening with all versions of gcc (4.1 through 14.0) if I force the dec_active() function to be inline and force inc_active() to be non-inline.

That's a bit of cheating, but I see your point now. How about we do:

kernel/bpf/memalloc.c +0 -0
index 51d6389e5152..3fa0944cb975 100644
@@ -183,11 +183,11 @@ static void inc_active(struct bpf_mem_cache *c,
 unsigned long *flags)
        WARN_ON_ONCE(local_inc_return(&c->active) != 1);
 }
 
Quoted 14 lines

kernel/bpf/memalloc.c static void inc_active(struct bpf_mem_cache *c, · line 192

Quoted 8 lines
kernel/bpf/memalloc.c +2 -2
-static void dec_active(struct bpf_mem_cache *c, unsigned long flags)
+static void dec_active(struct bpf_mem_cache *c, unsigned long *flags)
 {
        local_dec(&c->active);
        if (IS_ENABLED(CONFIG_PREEMPT_RT))
-               local_irq_restore(flags);
+               local_irq_restore(*flags);
 }

Sure, that's fine. Between this and the two suggestions I had (__always_inline or passing the flags from inc_active as a return code), I don't have a strong preference, so pick whichever you like.

I think:
static void dec_active(struct bpf_mem_cache *c, unsigned long *flags) is cleaner.
Could you send a patch?

#10 lore raw
Quoted 11 lines · Alexei Starovoitov
Quoted 5 lines · Arnd Bergmann

Sure, that's fine. Between this and the two suggestions I had (__always_inline or passing the flags from inc_active as a return code), I don't have a strong preference, so pick whichever you like.

I think:
static void dec_active(struct bpf_mem_cache *c, unsigned long *flags) is cleaner.
Could you send a patch?

Ok, done,

Arnd