Skip to contentSkip to conversation

syscalls/signal06: add volatile to loop variable

PATCH In review Edward Liaw via ltp 4 participants 18 messages Jul 12, 2022 – Aug 19, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 syscalls/signal06: add volatile to loop variable

On Android compiled with clang, the loop variable will be optimized out unless it is tagged with volatile.

---
 testcases/kernel/syscalls/signal/signal06.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/syscalls/signal/signal06.c +1 -1
index 64f886ee3..b40ff3e40 100644
@@ -65,7 +65,7 @@ char altstack[4096 * 10] __attribute__((aligned(4096)));
 
 void test(void)
 {
-	int loop = 0;
+	volatile int loop = 0;
 	int pid = getpid();
 
 	D = VALUE;
Signature · 5 lines
2.37.0.144.g8ac04bfd2-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/1
#1 lore raw

Hi Edward,

Quoted 2 lines

On Android compiled with clang, the loop variable will be optimized out unless it is tagged with volatile.

Thanks!

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
In reply to Edward Liaw via ltp

Hi!

Quoted 2 lines

On Android compiled with clang, the loop variable will be optimized out unless it is tagged with volatile.

Looking at the code it looks strange that it's optimized out since we use the value for the loop as:

	loop = 0;

	D = VALUE;
	while (D == VALUE && loop < LOOPS) {
		asm(...);
		loop++;
	}

And the D variable is properly marked as volatile so it's not like the loop can be expected to iterate preciselly LOOPS iterations.

It looks to me like the compiler actually forgets about the volatility of D for some reason and then assumes that the loop does LOOPS iterations.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw

Hey Cyril, sorry for the late reply, I was on vacation.

Quoted 16 lines · Cyril Hrubis

Hi!

Quoted 2 lines

On Android compiled with clang, the loop variable will be optimized out unless it is tagged with volatile.

Looking at the code it looks strange that it's optimized out since we use the value for the loop as:

        loop = 0;

        D = VALUE;
        while (D == VALUE && loop < LOOPS) {
                asm(...);
                loop++;
        }
Quoted 6 lines

And the D variable is properly marked as volatile so it's not like the loop can be expected to iterate preciselly LOOPS iterations.

It looks to me like the compiler actually forgets about the volatility of D for some reason and then assumes that the loop does LOOPS iterations.

I'm not totally sure how the compiler is working in this case.

What I saw with Android is that it fails with:

signal06    0  TINFO  :  loop = 2076174312
signal06    1  TFAIL  :

external/ltp/testcases/kernel/syscalls/signal/signal06.c:87: Bug Reproduced!

It makes one iteration, then loop is set to a random large int and the loop terminates. Printing the value of loop inside the for loop actually caused it to iterate 30000 times and succeed.

Compared to a successful run, which looks like:

signal06    0  TINFO  :  loop = 30000
signal06    1  TPASS  :  signal06 call succeeded

Thanks,
Edward

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
Quoted 3 lines
Quoted 3 lines

It looks to me like the compiler actually forgets about the volatility of D for some reason and then assumes that the loop does LOOPS iterations.

Hey Cyril,
I think I finally understand what you mean by this now; it is rather strange that the volatility of D does not protect loop from being optimized away by the compiler. I don't have a good explanation as to why it's happening but I'm not sure how to evaluate what's going on either. Should I do anything to move this patch forward?

Thanks,
Edward

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Hi!

Quoted 5 lines

I think I finally understand what you mean by this now; it is rather strange that the volatility of D does not protect loop from being optimized away by the compiler. I don't have a good explanation as to why it's happening but I'm not sure how to evaluate what's going on either. Should I do anything to move this patch forward?

It all boils down if we want to work around something that looks like a compiler bug in tests or not. I would be inclined not to do so since LTP was littered with quite a lot of workarounds for glibc/compiler bugs and we spend quite some time cleaning that mess up. But in this case I can agree that this is a borderland issue so I'm not strongly against that either.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw

Hi Edward,

Quoted 7 lines

Hi!

Quoted 5 lines

I think I finally understand what you mean by this now; it is rather strange that the volatility of D does not protect loop from being optimized away by the compiler. I don't have a good explanation as to why it's happening but I'm not sure how to evaluate what's going on either. Should I do anything to move this patch forward?

Quoted 6 lines

It all boils down if we want to work around something that looks like a compiler bug in tests or not. I would be inclined not to do so since LTP was littered with quite a lot of workarounds for glibc/compiler bugs and we spend quite some time cleaning that mess up. But in this case I can agree that this is a borderland issue so I'm not strongly against that either.

Edward, which which clang version requires it? It'd be nice to document it, so that it can be removed in the future.
Is there any bug report for it?

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw

We are currently building with clang 14.0.6. I haven't filed a bug report with llvm, will work on doing that.

Quoted 24 lines · Petr Vorel

Hi Edward,

Quoted 7 lines

Hi!

Quoted 5 lines

I think I finally understand what you mean by this now; it is rather strange that the volatility of D does not protect loop from being optimized away by the compiler. I don't have a good explanation as to why it's happening but I'm not sure how to evaluate what's going on either. Should I do anything to move this patch forward?

Quoted 6 lines

It all boils down if we want to work around something that looks like a compiler bug in tests or not. I would be inclined not to do so since LTP was littered with quite a lot of workarounds for glibc/compiler bugs and we spend quite some time cleaning that mess up. But in this case I can agree that this is a borderland issue so I'm not strongly against that either.

Edward, which which clang version requires it? It'd be nice to document it, so
that it can be removed in the future.
Is there any bug report for it?

Kind regards,
Petr

Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Tested-by
#8 lore raw

Hi Edward,

Quoted 2 lines

We are currently building with clang 14.0.6. I haven't filed a bug report with llvm, will work on doing that.

Thanks for info. I expected it'd be for aarch64 arch, but I can reproduce it on the same clang version on x86_64 on openSUSE Tumbleweed.

Would you mind we delay merging after you fill the bug in llvm, so that we can add it to git commit message?

Kind regards,
Petr

Quoted 25 lines · Petr Vorel
Quoted 24 lines

Hi Edward,

Quoted 7 lines

Hi!

Quoted 5 lines

I think I finally understand what you mean by this now; it is rather strange that the volatility of D does not protect loop from being optimized away by the compiler. I don't have a good explanation as to why it's happening but I'm not sure how to evaluate what's going on either. Should I do anything to move this patch forward?

Quoted 6 lines

It all boils down if we want to work around something that looks like a compiler bug in tests or not. I would be inclined not to do so since LTP was littered with quite a lot of workarounds for glibc/compiler bugs and we spend quite some time cleaning that mess up. But in this case I can agree that this is a borderland issue so I'm not strongly against that either.

Edward, which which clang version requires it? It'd be nice to document it, so
that it can be removed in the future.
Is there any bug report for it?

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#9 lore raw
Quoted 4 lines · Petr Vorel

Hi Edward,

Quoted 1 line

We are currently building with clang 14.0.6. I haven't filed a bug

report

Quoted 9 lines
Quoted 1 line

with llvm, will work on doing that.

Thanks for info. I expected it'd be for aarch64 arch, but I can reproduce it on
the same clang version on x86_64 on openSUSE Tumbleweed.

Would you mind we delay merging after you fill the bug in llvm, so that we can
add it to git commit message?

Sure thing, not a problem.

Quoted 1 line
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#11 lore raw
Quoted 5 lines
Quoted 4 lines · Petr Vorel
Quoted 2 lines

Hi Edward,

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Would you mind we delay merging after you fill the bug in llvm, so that we can
add it to git commit message?

Here's the link to the bug on llvm:
https://github.com/llvm/llvm-project/issues/57234

Thanks! The bug was closed as 'adding "cx" worked'. Reading topperc's comment it looks like there no other way to fix the issue on clang than workaround with volatile. Does it mean that it's a syscall problem and clang can do nothing about it?

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#12 lore raw

Hi!

Quoted 4 lines

Thanks! The bug was closed as 'adding "cx" worked'. Reading topperc's comment it looks like there no other way to fix the issue on clang than workaround with volatile. Does it mean that it's a syscall problem and clang can do nothing about it?

It's problem with the inline assembly in the body of the while loop, the call to the syscall changes the register value that is used for the D variable in the case of clang, so the loop exits prematurely. We have to add cx register to the clobber list for that asm statement so that compiler knows that it's changed by the assembly.

Interfacing assembly with C is a bit tricky since you have to explain to compiler which registers are changed from the assembly otherwise the results are undefined.

The patch should look like:

testcases/kernel/syscalls/signal/signal06.c +1 -1
index 64f886ee3..78efd0fb9 100644
@@ -73,7 +73,7 @@ void test(void)
                /* sys_tkill(pid, SIGHUP); asm to avoid save/reload
                 * fp regs around c call */
                asm ("" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP));
-               asm ("syscall" : : : "ax");
+               asm ("syscall" : : : "ax", "cx");
 
                loop++;
        }

Although it may not be a complete as the llwm issue suggests we should have a look at calling conventions for the syscall and check if we need to add any other registers to the clobber list.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#13 lore raw
Quoted 32 lines · Cyril Hrubis

Hi!

Quoted 4 lines

Thanks! The bug was closed as 'adding "cx" worked'. Reading topperc's comment it looks like there no other way to fix the issue on clang than workaround with volatile. Does it mean that it's a syscall problem and clang can do nothing about it?

It's problem with the inline assembly in the body of the while loop, the call to the syscall changes the register value that is used for the D variable in the case of clang, so the loop exits prematurely. We have to add cx register to the clobber list for that asm statement so that compiler knows that it's changed by the assembly.

Interfacing assembly with C is a bit tricky since you have to explain to compiler which registers are changed from the assembly otherwise the results are undefined.

The patch should look like:

testcases/kernel/syscalls/signal/signal06.c +1 -1
index 64f886ee3..78efd0fb9 100644
@@ -73,7 +73,7 @@ void test(void)
                /* sys_tkill(pid, SIGHUP); asm to avoid save/reload
                 * fp regs around c call */
                asm ("" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP));
-               asm ("syscall" : : : "ax");
+               asm ("syscall" : : : "ax", "cx");
 
                loop++;
        }
Quoted 3 lines

Although it may not be a complete as the llwm issue suggests we should have a look at calling conventions for the syscall and check if we need to add any other registers to the clobber list.

Looking at: https://en.wikibooks.org/wiki/X86_Assembly/Interfacing_with_Linux

We may as well add r11 as suggested by the clang issue comment and that should be complete.

Edward will you send a patch?

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#14 lore raw
In reply to Cyril Hrubis

Hi,

Am 8/19/2022 um 10:41 AM schrieb Cyril Hrubis:

Quoted 23 lines

Hi!

Quoted 4 lines

Thanks! The bug was closed as 'adding "cx" worked'. Reading topperc's comment it looks like there no other way to fix the issue on clang than workaround with volatile. Does it mean that it's a syscall problem and clang can do nothing about it?

It's problem with the inline assembly in the body of the while loop, the call to the syscall changes the register value that is used for the D variable in the case of clang, so the loop exits prematurely. We have to add cx register to the clobber list for that asm statement so that compiler knows that it's changed by the assembly.

Interfacing assembly with C is a bit tricky since you have to explain to compiler which registers are changed from the assembly otherwise the results are undefined.

The patch should look like:

testcases/kernel/syscalls/signal/signal06.c +0 -0
index 64f886ee3..78efd0fb9 100644

testcases/kernel/syscalls/signal/signal06.c void test(void) · line 76

Quoted 5 lines
testcases/kernel/syscalls/signal/signal06.c +1 -1
@@ -73,7 +73,7 @@ void test(void)
                /* sys_tkill(pid, SIGHUP); asm to avoid save/reload
                 * fp regs around c call */
                asm ("" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP));
-               asm ("syscall" : : : "ax");
+               asm ("syscall" : : : "ax", "cx");

Why is this even split up into two asm instructions?
I guess there is nothing, that prevents the compiler from reordering the asm instructions, because it does not know, that they have side effects (they are not marked volatile).

asm volatile ("syscall" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP):
"rax", "rcx", "r11");

I am not sure if there is any good reason, to split this up into two asm instructions and if there is a good reason, to use the short names of the registers.

Joerg

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#15 lore raw

Hi!

Quoted 27 lines
Quoted 17 lines

It's problem with the inline assembly in the body of the while loop, the call to the syscall changes the register value that is used for the D variable in the case of clang, so the loop exits prematurely. We have to add cx register to the clobber list for that asm statement so that compiler knows that it's changed by the assembly.

Interfacing assembly with C is a bit tricky since you have to explain to compiler which registers are changed from the assembly otherwise the results are undefined.

The patch should look like:

testcases/kernel/syscalls/signal/signal06.c +0 -0
index 64f886ee3..78efd0fb9 100644

testcases/kernel/syscalls/signal/signal06.c void test(void) · line 76

Quoted 5 lines
testcases/kernel/syscalls/signal/signal06.c +1 -1
@@ -73,7 +73,7 @@ void test(void)
                /* sys_tkill(pid, SIGHUP); asm to avoid save/reload
                 * fp regs around c call */
                asm ("" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP));
-               asm ("syscall" : : : "ax");
+               asm ("syscall" : : : "ax", "cx");

Why is this even split up into two asm instructions?
I guess there is nothing, that prevents the compiler from reordering the asm instructions, because it does not know, that they have side effects (they are not marked volatile).

Quoted 7 lines

asm volatile ("syscall" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP):
"rax", "rcx", "r11");

I am not sure if there is any good reason, to split this up into two asm instructions and if there is a good reason, to use the short names of the registers.

I was wondering too, I guess it's a direct copy of a reproducer from a kernel commit see:

https://lore.kernel.org/lkml/[email protected]/

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#16 lore raw
In reply to Cyril Hrubis
Quoted 23 lines

Hi!

Quoted 4 lines

Thanks! The bug was closed as 'adding "cx" worked'. Reading topperc's comment it looks like there no other way to fix the issue on clang than workaround with volatile. Does it mean that it's a syscall problem and clang can do nothing about it?

It's problem with the inline assembly in the body of the while loop, the call to the syscall changes the register value that is used for the D variable in the case of clang, so the loop exits prematurely. We have to add cx register to the clobber list for that asm statement so that compiler knows that it's changed by the assembly.

Interfacing assembly with C is a bit tricky since you have to explain to compiler which registers are changed from the assembly otherwise the results are undefined.

The patch should look like:

testcases/kernel/syscalls/signal/signal06.c +0 -0
index 64f886ee3..78efd0fb9 100644

testcases/kernel/syscalls/signal/signal06.c void test(void) · line 76

Quoted 5 lines
testcases/kernel/syscalls/signal/signal06.c +1 -1
@@ -73,7 +73,7 @@ void test(void)
                /* sys_tkill(pid, SIGHUP); asm to avoid save/reload
                 * fp regs around c call */
                asm ("" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP));
-               asm ("syscall" : : : "ax");
+               asm ("syscall" : : : "ax", "cx");

OK, I completely overlooked asm() in the test. Thanks!

Petr

Quoted 6 lines
testcases/kernel/syscalls/signal/signal06.c +0 -0
                loop++;
        }
 

Although it may not be a complete as the llwm issue suggests we should have a look at calling conventions for the syscall and check if we need to add any other registers to the clobber list.

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#17 lore raw
Quoted 8 lines · Joerg Vehlow

Why is this even split up into two asm instructions?
I guess there is nothing, that prevents the compiler from reordering the asm instructions, because it does not know, that they have side effects (they are not marked volatile).

asm volatile ("syscall" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP):
"rax", "rcx", "r11");

I tried this and clang complains that the "Asm-specifier for input or output variable conflicts with asm clobber list" for rax. Should it be

asm volatile ("syscall" : : "a"(__NR_tkill), "D"(pid), "S"(SIGHUP) :
"rcx", "r11");

instead? Is it implicit that __NR_tkill is going into rax so it will be clobbered?

Thanks,
Edward

Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp