Skip to contentSkip to conversation

clang asm goto issue small reproducer

DISCUSSION Nick Desaulniers 2 participants 6 messages Dec 15, 2021 – Dec 16, 2021 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 6 lines · Mathieu Desnoyers

Hi Nick,

I have a small reproducer for the asm goto issue we discussed over irc. I have axed away all code that is not the problematic function by hand. Hopefully this won't change the overall optimization context too much.

If I cut the compilation pipeline off at after "middle-end" optimizations, I can see that this_cpu_list_pop returns NULL for test.c while in test-asm.c, it's a phi node of either NULL or head (which looks correct; for the test-asm.c case).

"GVNPass" seems to be converting

return: ; preds =
%cleanup.thread19, %cleanup.thread
%retval.118 = phi %struct.percpu_list_node* [ %9, %cleanup.thread ],
[ null, %cleanup.thread19 ]
ret %struct.percpu_list_node* %retval.118

into:

return: ; preds =
%cleanup.thread19, %cleanup.thread
ret %struct.percpu_list_node* null

Which drops the
%9 = load %struct.percpu_list_node*, %struct.percpu_list_node**
%head, align 8, !tbaa !9

I think GVN may be eliminating a phi that it should not be. Specifically, it seems that GVN thinks the default/fallthrough successor is unreachable, which seems incorrect to me. The predecessor lists all look correct to me, so GVN may be stumbling on callbr itself.

$ clang -O2 test.c -emit-llvm -S -o - -Xclang -disable-llvm-passes > test.ll
$ opt -O2 -print-before=gvn -print-after=gvn
-filter-print-funcs=this_cpu_list_pop test.ll -o /dev/null 2>
test.gvn.ll

Then using this reproducer, I was able to narrow it down a little further:

$ opt -O2 -print-before=gvn -print-module-scope
--filter-print-funcs=this_cpu_list_pop test.ll -S 2>test.pre-gvn.ll
$ llvm-extract --func=this_cpu_list_pop test.pre-gvn.ll -o
test.pre-gvn.this_cpu_list_pop.bc --recursive
$ llvm-dis test.pre-gvn.this_cpu_list_pop.bc
$ cat repro.sh
#!/usr/bin/env sh
set -e
grep -q "%retval.118 = phi %struct.percpu_list_node\* \[ %9,
%cleanup.thread \], \[ null, %cleanup.thread19 \]" $1
grep -q "ret %struct.percpu_list_node\* %retval.118" $1
OUT=$(opt -passes=gvn $1 -S -o -)
echo $OUT | grep -q "ret %struct.percpu_list_node\* null"
$ llvm-reduce --test=./repro.sh test.pre-gvn.this_cpu_list_pop.ll

Then pared down further manually, see reduced.ll

I've filed https://github.com/llvm/llvm-project/issues/52735, let's discuss more there?

Quoted 17 lines

on x86-64:

Buggy code:
clang-13 -O2 -o test.o test.c

OK code (added a dummy asm("")):
clang-13 -O2 -o test-asm.o test-asm.c

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 2 lines
Thanks,
~Nick Desaulniers
#2 lore raw
Quoted 25 lines · Nick Desaulniers
Quoted 6 lines · Mathieu Desnoyers

Hi Nick,

I have a small reproducer for the asm goto issue we discussed over irc. I have axed away all code that is not the problematic function by hand. Hopefully this won't change the overall optimization context too much.

If I cut the compilation pipeline off at after "middle-end" optimizations, I can see that this_cpu_list_pop returns NULL for test.c while in test-asm.c, it's a phi node of either NULL or head (which looks correct; for the test-asm.c case).

"GVNPass" seems to be converting

return: ; preds =
%cleanup.thread19, %cleanup.thread
%retval.118 = phi %struct.percpu_list_node* [ %9, %cleanup.thread ],
[ null, %cleanup.thread19 ]
ret %struct.percpu_list_node* %retval.118

into:

Quoted 7 lines

return: ; preds =
%cleanup.thread19, %cleanup.thread
ret %struct.percpu_list_node* null

Which drops the
%9 = load %struct.percpu_list_node*, %struct.percpu_list_node**
%head, align 8, !tbaa !9

Ah, Eli pointed out that I'm chasing the wrong squirrel. From test.c we have:

struct percpu_list_node *head;
load = (intptr_t *)&head;
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,

     offset, load, cpu);

then in rseq_cmpnev_storeoffp_load():
__asm__ __volatile__ goto (
:
:
[load] "m" (*load));

Did we ever initialize head? Isn't this equivalent to:

int* foo (void) {
int *head;
int **load;
load = &head;
asm(""::"m"(*load):"memory");
return head;
}

Isn't that equivalent to:

int* foo (void) {
int *head;
asm(""::"m"(head):"memory");
return head;
}

For which neither compiler warns for -Wuninitialized? They would if head was an argument to a function call...

Quoted 54 lines

I think GVN may be eliminating a phi that it should not be. Specifically, it seems that GVN thinks the default/fallthrough successor is unreachable, which seems incorrect to me. The predecessor lists all look correct to me, so GVN may be stumbling on callbr itself.

$ clang -O2 test.c -emit-llvm -S -o - -Xclang -disable-llvm-passes > test.ll
$ opt -O2 -print-before=gvn -print-after=gvn
-filter-print-funcs=this_cpu_list_pop test.ll -o /dev/null 2>
test.gvn.ll

Then using this reproducer, I was able to narrow it down a little further:

$ opt -O2 -print-before=gvn -print-module-scope
--filter-print-funcs=this_cpu_list_pop test.ll -S 2>test.pre-gvn.ll
$ llvm-extract --func=this_cpu_list_pop test.pre-gvn.ll -o
test.pre-gvn.this_cpu_list_pop.bc --recursive
$ llvm-dis test.pre-gvn.this_cpu_list_pop.bc
$ cat repro.sh
#!/usr/bin/env sh
set -e
grep -q "%retval.118 = phi %struct.percpu_list_node\* \[ %9,
%cleanup.thread \], \[ null, %cleanup.thread19 \]" $1
grep -q "ret %struct.percpu_list_node\* %retval.118" $1
OUT=$(opt -passes=gvn $1 -S -o -)
echo $OUT | grep -q "ret %struct.percpu_list_node\* null"
$ llvm-reduce --test=./repro.sh test.pre-gvn.this_cpu_list_pop.ll

Then pared down further manually, see reduced.ll

I've filed https://github.com/llvm/llvm-project/issues/52735, let's discuss more there?

Quoted 9 lines

on x86-64:

Buggy code:
clang-13 -O2 -o test.o test.c

OK code (added a dummy asm("")):
clang-13 -O2 -o test-asm.o test-asm.c

Quoted 8 lines

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 3 lines

--
Thanks,
~Nick Desaulniers

--
Thanks,
~Nick Desaulniers

#3 lore raw
Quoted 61 lines · Nick Desaulniers
Quoted 25 lines · Nick Desaulniers
Quoted 6 lines · Mathieu Desnoyers

Hi Nick,

I have a small reproducer for the asm goto issue we discussed over irc. I have axed away all code that is not the problematic function by hand. Hopefully this won't change the overall optimization context too much.

If I cut the compilation pipeline off at after "middle-end" optimizations, I can see that this_cpu_list_pop returns NULL for test.c while in test-asm.c, it's a phi node of either NULL or head (which looks correct; for the test-asm.c case).

"GVNPass" seems to be converting

return: ; preds =
%cleanup.thread19, %cleanup.thread
%retval.118 = phi %struct.percpu_list_node* [ %9, %cleanup.thread ],
[ null, %cleanup.thread19 ]
ret %struct.percpu_list_node* %retval.118

into:

Quoted 7 lines

return: ; preds =
%cleanup.thread19, %cleanup.thread
ret %struct.percpu_list_node* null

Which drops the
%9 = load %struct.percpu_list_node*, %struct.percpu_list_node**
%head, align 8, !tbaa !9

Ah, Eli pointed out that I'm chasing the wrong squirrel. From test.c we have:

struct percpu_list_node *head;
load = (intptr_t *)&head;
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,

     offset, load, cpu);

then in rseq_cmpnev_storeoffp_load():
__asm__ __volatile__ goto (
:
:
[load] "m" (*load));

Did we ever initialize head? Isn't this equivalent to:

int* foo (void) {
int *head;
int **load;
load = &head;
asm(""::"m"(*load):"memory");
return head;
}

Isn't that equivalent to:

Quoted 8 lines

int* foo (void) {
int *head;
asm(""::"m"(head):"memory");
return head;
}

For which neither compiler warns for -Wuninitialized? They would if head was an argument to a function call...

What's going on here...

$ cat y.c
void x (void) {
  int y;
  asm(""::"r"(y));
}
void z (void) {
  int y;
  asm(""::"m"(y));
}
$ clang -Wuninitialized y.c -c
y.c:3:15: warning: variable 'y' is uninitialized when used here
[-Wuninitialized]
  asm(""::"r"(y));
              ^
y.c:2:8: note: initialize the variable 'y' to silence this warning
  int y;
       ^
        = 0
1 warning generated.
$ gcc -Wuninitialized y.c -c
y.c: In function ‘x’:
y.c:3:3: warning: ‘y’ is used uninitialized [-Wuninitialized]
    3 |   asm(""::"r"(y));
      |   ^~~

Only 1 warning each? I expected 2.

Is there something special about "m" constraints and -Wuninitialized?

Signature · 2 lines
Thanks,
~Nick Desaulniers
#4 lore raw
Quoted 99 lines · ----- On Dec 15, 2021, at 7:25 PM, ndesaulniers [email protected] wrote:
Quoted 61 lines · Nick Desaulniers
Quoted 25 lines · Nick Desaulniers
Quoted 6 lines · Mathieu Desnoyers

Hi Nick,

I have a small reproducer for the asm goto issue we discussed over irc. I have axed away all code that is not the problematic function by hand. Hopefully this won't change the overall optimization context too much.

If I cut the compilation pipeline off at after "middle-end" optimizations, I can see that this_cpu_list_pop returns NULL for test.c while in test-asm.c, it's a phi node of either NULL or head (which looks correct; for the test-asm.c case).

"GVNPass" seems to be converting

return: ; preds =
%cleanup.thread19, %cleanup.thread
%retval.118 = phi %struct.percpu_list_node* [ %9, %cleanup.thread ],
[ null, %cleanup.thread19 ]
ret %struct.percpu_list_node* %retval.118

into:

Quoted 7 lines

return: ; preds =
%cleanup.thread19, %cleanup.thread
ret %struct.percpu_list_node* null

Which drops the
%9 = load %struct.percpu_list_node*, %struct.percpu_list_node**
%head, align 8, !tbaa !9

Ah, Eli pointed out that I'm chasing the wrong squirrel. From test.c we have:

struct percpu_list_node *head;
load = (intptr_t *)&head;
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,

     offset, load, cpu);

then in rseq_cmpnev_storeoffp_load():
__asm__ __volatile__ goto (
:
:
[load] "m" (*load));

Did we ever initialize head? Isn't this equivalent to:

int* foo (void) {
int *head;
int **load;
load = &head;
asm(""::"m"(*load):"memory");
return head;
}

Isn't that equivalent to:

Quoted 8 lines

int* foo (void) {
int *head;
asm(""::"m"(head):"memory");
return head;
}

For which neither compiler warns for -Wuninitialized? They would if head was an argument to a function call...

What's going on here...

$ cat y.c
void x (void) {
 int y;
 asm(""::"r"(y));
}
void z (void) {
 int y;
 asm(""::"m"(y));
}
$ clang -Wuninitialized y.c -c
y.c:3:15: warning: variable 'y' is uninitialized when used here
[-Wuninitialized]
 asm(""::"r"(y));
             ^
y.c:2:8: note: initialize the variable 'y' to silence this warning
 int y;
      ^
       = 0
1 warning generated.
$ gcc -Wuninitialized y.c -c
y.c: In function ‘x’:
y.c:3:3: warning: ‘y’ is used uninitialized [-Wuninitialized]
   3 |   asm(""::"r"(y));
     |   ^~~
Quoted 3 lines

Only 1 warning each? I expected 2.

Is there something special about "m" constraints and -Wuninitialized?

In my work-arounds for the lack of output operands with asm goto, I've used either of those tricks across various architectures:

- "m" (y) input operand with "memory" clobber, or
- "r" (&y) input operand with "memory" clobber

to achieve writing to a memory area from assembler without having output operands available.

It may be stretching the definitions, but those were my (perhaps ill advised) expectations.

Thanks,

Mathieu

Quoted 4 lines

--
Thanks,
~Nick Desaulniers

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#6 lore raw
In reply to Nick Desaulniers
Quoted 60 lines · ----- On Dec 15, 2021, at 7:18 PM, ndesaulniers [email protected] wrote:
Quoted 25 lines · Nick Desaulniers
Quoted 6 lines · Mathieu Desnoyers

Hi Nick,

I have a small reproducer for the asm goto issue we discussed over irc. I have axed away all code that is not the problematic function by hand. Hopefully this won't change the overall optimization context too much.

If I cut the compilation pipeline off at after "middle-end" optimizations, I can see that this_cpu_list_pop returns NULL for test.c while in test-asm.c, it's a phi node of either NULL or head (which looks correct; for the test-asm.c case).

"GVNPass" seems to be converting

return: ; preds =
%cleanup.thread19, %cleanup.thread
%retval.118 = phi %struct.percpu_list_node* [ %9, %cleanup.thread ],
[ null, %cleanup.thread19 ]
ret %struct.percpu_list_node* %retval.118

into:

Quoted 7 lines

return: ; preds =
%cleanup.thread19, %cleanup.thread
ret %struct.percpu_list_node* null

Which drops the
%9 = load %struct.percpu_list_node*, %struct.percpu_list_node**
%head, align 8, !tbaa !9

Ah, Eli pointed out that I'm chasing the wrong squirrel. From test.c we have:

struct percpu_list_node *head;
load = (intptr_t *)&head;
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,

    offset, load, cpu);

then in rseq_cmpnev_storeoffp_load():
__asm__ __volatile__ goto (
:
:
[load] "m" (*load));

Did we ever initialize head? Isn't this equivalent to:

int* foo (void) {
int *head;
int **load;
load = &head;
asm(""::"m"(*load):"memory");
return head;
}

Isn't that equivalent to:

Quoted 8 lines

int* foo (void) {
int *head;
asm(""::"m"(head):"memory");
return head;
}

For which neither compiler warns for -Wuninitialized? They would if head was an argument to a function call...

This is a situation where I had to work around the fact that older gcc did not support output operands. I expected that having both a "m" input operand and a "memory" clobber would allow the asm to write to the memory location pointed to by the "m" input operand, and the "memory" clobber was expected to prevent the compiler from presuming anything about memory state across the asm.

But I suspect I was expecting too much here. What would be the best course of action to fix these asm statements ? Do I have any way to do this without requiring a dependency on gcc/clang that support asm goto with output operands ?

Thanks,

Mathieu

Quoted 64 lines
Quoted 55 lines

I think GVN may be eliminating a phi that it should not be. Specifically, it seems that GVN thinks the default/fallthrough successor is unreachable, which seems incorrect to me. The predecessor lists all look correct to me, so GVN may be stumbling on callbr itself.

$ clang -O2 test.c -emit-llvm -S -o - -Xclang -disable-llvm-passes > test.ll
$ opt -O2 -print-before=gvn -print-after=gvn
-filter-print-funcs=this_cpu_list_pop test.ll -o /dev/null 2>
test.gvn.ll

Then using this reproducer, I was able to narrow it down a little further:

$ opt -O2 -print-before=gvn -print-module-scope
--filter-print-funcs=this_cpu_list_pop test.ll -S 2>test.pre-gvn.ll
$ llvm-extract --func=this_cpu_list_pop test.pre-gvn.ll -o
test.pre-gvn.this_cpu_list_pop.bc --recursive
$ llvm-dis test.pre-gvn.this_cpu_list_pop.bc
$ cat repro.sh
#!/usr/bin/env sh
set -e
grep -q "%retval.118 = phi %struct.percpu_list_node\* \[ %9,
%cleanup.thread \], \[ null, %cleanup.thread19 \]" $1
grep -q "ret %struct.percpu_list_node\* %retval.118" $1
OUT=$(opt -passes=gvn $1 -S -o -)
echo $OUT | grep -q "ret %struct.percpu_list_node\* null"
$ llvm-reduce --test=./repro.sh test.pre-gvn.this_cpu_list_pop.ll

Then pared down further manually, see reduced.ll

I've filed https://github.com/llvm/llvm-project/issues/52735, let's discuss more there?

Quoted 9 lines

on x86-64:

Buggy code:
clang-13 -O2 -o test.o test.c

OK code (added a dummy asm("")):
clang-13 -O2 -o test-asm.o test-asm.c

Quoted 8 lines

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 3 lines

--
Thanks,
~Nick Desaulniers

--
Thanks,
~Nick Desaulniers

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#5 lore raw
In reply to Nick Desaulniers

Hi Nick,

Here is the result of:

opt-13 -passes='gvn,gvn-hoist,gvn-sink,print<memoryssa>' -filter-print-funcs=this_cpu_list_pop test-asm.ll -o /dev/null 2> test-asm.gvn.ll opt-13 -passes='gvn,gvn-hoist,gvn-sink,print<memoryssa>' -filter-print-funcs=this_cpu_list_pop test.ll -o /dev/null 2> test.gvn.ll

I notice a missing MemoryPhi in the case without the extra asm("").

Thanks,

Mathieu

Quoted 106 lines · ----- On Dec 15, 2021, at 7:25 PM, ndesaulniers [email protected] wrote:
Quoted 61 lines · Nick Desaulniers
Quoted 25 lines · Nick Desaulniers
Quoted 6 lines · Mathieu Desnoyers

Hi Nick,

I have a small reproducer for the asm goto issue we discussed over irc. I have axed away all code that is not the problematic function by hand. Hopefully this won't change the overall optimization context too much.

If I cut the compilation pipeline off at after "middle-end" optimizations, I can see that this_cpu_list_pop returns NULL for test.c while in test-asm.c, it's a phi node of either NULL or head (which looks correct; for the test-asm.c case).

"GVNPass" seems to be converting

return: ; preds =
%cleanup.thread19, %cleanup.thread
%retval.118 = phi %struct.percpu_list_node* [ %9, %cleanup.thread ],
[ null, %cleanup.thread19 ]
ret %struct.percpu_list_node* %retval.118

into:

Quoted 7 lines

return: ; preds =
%cleanup.thread19, %cleanup.thread
ret %struct.percpu_list_node* null

Which drops the
%9 = load %struct.percpu_list_node*, %struct.percpu_list_node**
%head, align 8, !tbaa !9

Ah, Eli pointed out that I'm chasing the wrong squirrel. From test.c we have:

struct percpu_list_node *head;
load = (intptr_t *)&head;
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,

     offset, load, cpu);

then in rseq_cmpnev_storeoffp_load():
__asm__ __volatile__ goto (
:
:
[load] "m" (*load));

Did we ever initialize head? Isn't this equivalent to:

int* foo (void) {
int *head;
int **load;
load = &head;
asm(""::"m"(*load):"memory");
return head;
}

Isn't that equivalent to:

Quoted 8 lines

int* foo (void) {
int *head;
asm(""::"m"(head):"memory");
return head;
}

For which neither compiler warns for -Wuninitialized? They would if head was an argument to a function call...

What's going on here...

$ cat y.c
void x (void) {
 int y;
 asm(""::"r"(y));
}
void z (void) {
 int y;
 asm(""::"m"(y));
}
$ clang -Wuninitialized y.c -c
y.c:3:15: warning: variable 'y' is uninitialized when used here
[-Wuninitialized]
 asm(""::"r"(y));
             ^
y.c:2:8: note: initialize the variable 'y' to silence this warning
 int y;
      ^
       = 0
1 warning generated.
$ gcc -Wuninitialized y.c -c
y.c: In function ‘x’:
y.c:3:3: warning: ‘y’ is used uninitialized [-Wuninitialized]
   3 |   asm(""::"r"(y));
     |   ^~~

Only 1 warning each? I expected 2.

Is there something special about "m" constraints and -Wuninitialized?

--
Thanks,
~Nick Desaulniers

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com