might_sleep warnings in overwrite mode
Quoted 6 lines · Nathan Lynch
At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.
Is it possible that the lttng module code is messing up the current task's preempt count somehow?
Here's another manifestation of the issue -- when flood-pinging the board during tracing, I get this warning from __do_softirq:
huh, entered softirq 3 NET_RX 804ae528 preempt_count 00000101, exited with 00000102?
Quoted 3 lines · Nathan Lynch
At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.
Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.
----- Original Message -----
Quoted 10 lines
From: "Nathan Lynch" <[email protected]>
To: [email protected]
Sent: Thursday, November 14, 2013 1:16:53 PM
Subject: Re: [lttng-dev]might_sleepwarnings in overwrite modeQuoted 3 lines · Nathan Lynch
At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.
Quoted 6 lines
Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.
Allright! This info is really useful. Sorry for taking time to look into this, I was busy with preparation for 2.4-rc1.
Looking at:
fs/ext3/incode.c:
__ext3_get_inode_loc()
we can see that a use of the inlined sb_getblk() appears close to a tracepoint.
The tracepoint includes preempt disable/enable, exactly those:
include/linux/preempt.h:
#define preempt_disable_notrace() \
do { \
inc_preempt_count_notrace(); \
barrier(); \
} while (0)#define preempt_enable_notrace() \
do { \
preempt_enable_no_resched_notrace(); \
barrier(); \
preempt_check_resched_context(); \
} while (0)Can you try wrapping the _outside_ of those macros with barrier(), e.g.
#define preempt_disable_notrace() \
do { \
barrier(); \
inc_preempt_count_notrace(); \
barrier(); \
} while (0)#define preempt_enable_notrace() \
do { \
preempt_enable_no_resched_notrace(); \
barrier(); \
preempt_check_resched_context(); \
barrier(); \
} while (0)and try it out with the apparently buggy compiler to see if it helps ? It does look like the preempt inc or dec is slipping out and somehow triggers the might_sleep() warning. I don't see clearly how this could happen yet, since each of the inc/dec and the test are touching preempt_count(), but it's worth a try.
Maybe on ARM the current_thread_info() macro somehow hides important info from the compiler and it mistakenly reorders inc/dec vs the test. Another thing to try out (in addition to the first one) would be to try changing current_thread_info(), e.g., by turning asm ("sp") into a volatile inline assembly, and by adding "memory" clobbers to it.
Thoughts ?
Thanks,
Mathieu
Quoted 8 lines
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 55 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 10 lines
From: "Nathan Lynch" <[email protected]>
To: [email protected]
Sent: Thursday, November 14, 2013 1:16:53 PM
Subject: Re: [lttng-dev]might_sleepwarnings in overwrite modeQuoted 3 lines · Nathan Lynch
At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.
Quoted 6 lines
Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.Looking at:
fs/ext3/incode.c:
__ext3_get_inode_loc()we can see that a use of the inlined
sb_getblk()appears close to a tracepoint.The tracepoint includes preempt disable/enable, exactly those:
include/linux/preempt.h:#define
preempt_disable_notrace()\do { \ inc_preempt_count_notrace(); \ barrier(); \ } while (0)#define
preempt_enable_notrace()\do { \ preempt_enable_no_resched_notrace(); \ barrier(); \ preempt_check_resched_context(); \ } while (0)Can you try wrapping the _outside_ of those macros with
barrier(), e.g.#define
preempt_disable_notrace()\do { \ barrier(); \ inc_preempt_count_notrace(); \ barrier(); \ } while (0)#define
preempt_enable_notrace()\
Quoted 8 lines
do { \ preempt_enable_no_resched_notrace(); \ barrier(); \ preempt_check_resched_context(); \ barrier(); \ } while (0)and try it out with the apparently buggy compiler to see if it helps ?
I did this, and I think I'm pretty satisfied that it works around the issue. I haven't been able to provoke might_sleep or preempt count warnings with multiple runs. Complicating matters is that I keep hitting a null pointer dereference in the mmc code which I wasn't seeing before (and sorry about the line wrapping, I cannot make Thunderbird behave well here):
Unable to handle kernel NULL pointer dereference at virtual address 00000000 pgd = 80004000 [00000000] *pgd=00000000 Internal error: Oops: 17 [#1] PREEMPT SMP ARM Modules linked in: lttng_probe_writeback(O) lttng_probe_workqueue(O)
lttng_probe_vmscan(O) lttng_probe_udp(O) lttng_probe_timer(O) lttng_probe_sunrpc(O) lttng_probe_statedump(O) lttng_probe_sock(O) lttng_probe_skb(O) lttng_probe_signal(O) lttng_probe_scsi(O) lttng_probe_sched(O) lttng_probe_rpm(O) lttng_probe_regulator(O) lttng_probe_regmap(O) lttng_probe_rcu(O) lttng_probe_random(O) lttng_probe_printk(O) lttng_probe_power(O) lttng_probe_net(O) lttng_probe_napi(O) lttng_probe_module(O) lttng_probe_kmem(O) lttng_probe_jbd2(O) lttng_probe_jbd(O) lttng_probe_irq(O) lttng_probe_gpio(O) lttng_probe_ext4(O) lttng_probe_ext3(O) lttng_probe_compaction(O) lttng_probe_block(O) lttng_probe_asoc(O) lttng_types(O) lttng_ring_buffer_metadata_mmap_client(O) lttng_ring_buffer_client_mmap_overwrite(O) lttng_ring_buffer_client_mmap_discard(O) lttng_ring_buffer_metadata_client(O) lttng_ring_buffer_client_overwrite(O) lttng_ring_buffer_client_discard(O) lttng_tracer(O) lttng_statedump(O) lttng_ftrace(O) lttng_kprobes(O) lttng_lib_ring_buffer(O) lttng_kretprobes(O)
CPU: 0 PID: 143 Comm: mmcqd/0 Tainted: G O 3.11.6 #1 task: bf0d0900 ti: bf0be000 task.ti: bf0be000 PC is at sdhci_send_command+0x3fc/0xc58 LR is at sdhci_send_command+0x424/0xc58 pc : [<804371d0>] lr : [<804371f8>] psr: 80000093
sp : bf0bfd50 ip : bf0bfd50 fp : bf0bfdac
r10: 60000013 r9 : bfb7c440 r8 : 0000000f
r7 : 808da22c r6 : bb39c000 r5 : bb3ebe18 r4 : bb3ebe80 r3 : 87654321 r2 : bfa12c10 r1 : 81802b22 r0 : 00000000
Flags: Nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 4ecb404a DAC: 00000017
Process mmcqd/0 (pid: 143, stack limit = 0xbf0be238)
Stack: (0xbf0bfd50 to 0xbf0c0000)
fd40: 00000001 00000000 bf0bfd84
87654321
fd60: bfa12c10 7f0b700c bfb7c000 bb3ebda0 bfb7c440 00000001 bf0bfd94 bf0bfd88
fd80: 802d1af8 bfb7c000 bb3ebda0 bfb7c440 bfb7c518 40000013 00000001 bfb7c000
fda0: bf0bfdd4 bf0bfdb0 80438194 80436de0 bb3ebda0 bfb7c000 bb3ebd8c 00000000
fdc0: 00000001 00000001 bf0bfdec bf0bfdd8 804248d4 80437fd8 bb3ebec0 bf0bfe74
fde0: bf0bfe44 bf0bfdf0 804261b8 80424800 bf0be000 bf0be000 bf0be030 bb3ebc6c
fe00: bf0bfe44 00000000 bf0d0900 80047430 bf0bfe10 bf0bfe10 3aee2e82 bb368618
fe20: bb3ebc24 bb3c6000 bb317438 bb317438 bb3ebd9c bb3ebc00 bf0bfea4 bf0bfe48
fe40: 80433ba8 80425ef8 7f08d740 7f066a08 00000000 00000000 bf026000 00000000
fe60: bb3ebda0 00000000 bb3ebc00 00000000 00000038 cec07c38 cec07c00 bb317438
fe80: bb3c6000 bb3ebc24 bb368618 bfb7c000 bb3ebc00 bb3ebc00 bf0bfeec bf0bfea8
fea0: 804342a4 80433800 bb3ebd9c bb3ebc00 bb317438 bb3ebc2c bb368618 5e565e55
fec0: bb3ebc00 bb3ebc24 bb317438 bb3ebc2c bb368618 bf0be030 bf0be000 00000001
fee0: bf0bff24 bf0bfef0 80434cb4 804341a4 00000000 122c8001 80434bec bfa87bc4
ff00: 00000000 bb3ebc24 80434bec 00000000 00000000 00000000 bf0bffac bf0bff28
ff20: 80046520 80434bf8 8182be80 00000000 8182be80 bb3ebc24 00000000 00000000
ff40: dead4ead ffffffff ffffffff 80942bfc 00000000 00000000 807877a5 bf0bff5c
ff60: bf0bff5c 00000000 00000000 dead4ead ffffffff ffffffff 80942bfc 00000000
ff80: 00000000 807877a5 bf0bff88 bf0bff88 bfa87bc4 80046464 00000000 00000000
ffa0: 00000000 bf0bffb0 8000e438 80046470 00000000 00000000 00000000 00000000
ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
Backtrace: [<80436dd4>] (sdhci_send_command+0x0/0xc58) from [<80438194>] (sdhci_request+0x1c8/0x1e0) [<80437fcc>] (sdhci_request+0x0/0x1e0) from [<804248d4>] (mmc_start_request+0xe0/0xe8) r9:00000001 r8:00000001 r7:00000000 r6:bb3ebd8c r5:bfb7c000
[<804247f4>] (mmc_start_request+0x0/0xe8) from [<804261b8>] (mmc_start_req+0x2cc/0x360)
r5:bf0bfe74 r4:bb3ebec0
[<80425eec>] (mmc_start_req+0x0/0x360) from [<80433ba8>] (mmc_blk_issue_rw_rq+0x3b4/0x9a4)
[<804337f4>] (mmc_blk_issue_rw_rq+0x0/0x9a4) from [<804342a4>] (mmc_blk_issue_rq+0x10c/0x450)
[<80434198>] (mmc_blk_issue_rq+0x0/0x450) from [<80434cb4>] (mmc_queue_thread+0xc8/0x164)
[<80434bec>] (mmc_queue_thread+0x0/0x164) from [<80046520>] (kthread+0xbc/0xc8)
[<80046464>] (kthread+0x0/0xc8) from [<8000e438>] (ret_from_fork+0x14/0x20)
r7:00000000 r6:00000000 r5:80046464 r4:bfa87bc4
Code: e1a00006 e3a08000 e158000a aa00000c (e5901000)
I've narrowed this down to sdhci_adma_table_pre() which is inlined, but haven't gotten much further. Anyway, I think it's probably unrelated -- surely it wouldn't be _caused_ by adding the barriers -- and I'll have to figure out some way to work around it (perhaps it's fixed in 3.12).
Quoted 11 lines
It does look like the preempt inc or dec is slipping out and somehow triggers the
might_sleep()warning. I don't see clearly how this could happen yet, since each of the inc/dec and the test are touchingpreempt_count(), but it's worth a try.Maybe on ARM the
current_thread_info()macro somehow hides important info from the compiler and it mistakenly reorders inc/dec vs the test. Another thing to try out (in addition to the first one) would be to try changingcurrent_thread_info(), e.g., by turning asm ("sp") into a volatile inline assembly, and by adding "memory" clobbers to it.
Haven't tried this yet.
current_thread_info() not respecting program order with gcc 4.8.x
In reply to a message not in the archiveHi,
I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the current_thread_info() inline assembly. The only logical explanation for his issue I see so far is that read of the preempt_count within might_sleep() is reordered with preempt_enable() or preempt_disable(). AFAIU, this kind of issue also applies to other architectures.
First thing, preempt enable/disable only contains barrier() between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that the preempt_count() read in might_sleep() is not reordered with the preempt count inc/dec.
However, looking at ARM arch/arm/include/asm/thread_info.h:
static inline struct thread_info *current_thread_info(void)
{
register unsigned long sp asm ("sp");
return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
}The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)
As long as someone does:
struct thread_info *myinfo = current_thread_info();
load from myinfo->preempt_count;
store to myinfo->preempt_count;
The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of might_sleep() followed by preempt_enable() is:
load from current_thread_info()->preempt_count;
store to current_thread_info()->preempt_count;
Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.
One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).
Have any of you experienced this issue ? Any thoughts on the matter ?
I'm providing the original bug report email exchange below. I had confirmation that adding barrier() outside preempt_enable()/preempt_disable() does indeed work-around the issue, but I fear that this work-around would be leaving the current_thread_info() semantic gap in place: people expect program order to be respected.
Thanks,
Mathieu
----- Original Message -----
Quoted 110 lines
From: "Mathieu Desnoyers" <[email protected]>
To: "Nathan Lynch" <[email protected]>
Cc: [email protected]
Sent: Thursday, November 14, 2013 9:34:09 PM
Subject: Re: [lttng-dev]might_sleepwarnings in overwrite mode----- Original Message -----
Quoted 10 lines
From: "Nathan Lynch" <[email protected]>
To: [email protected]
Sent: Thursday, November 14, 2013 1:16:53 PM
Subject: Re: [lttng-dev]might_sleepwarnings in overwrite modeQuoted 3 lines · Nathan Lynch
At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.
Quoted 6 lines
Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.Allright! This info is really useful. Sorry for taking time to look into this, I was busy with preparation for 2.4-rc1.
Looking at:
fs/ext3/incode.c:
__ext3_get_inode_loc()we can see that a use of the inlined
sb_getblk()appears close to a tracepoint.The tracepoint includes preempt disable/enable, exactly those:
include/linux/preempt.h:#define
preempt_disable_notrace()\do { \ inc_preempt_count_notrace(); \ barrier(); \ } while (0)#define
preempt_enable_notrace()\do { \ preempt_enable_no_resched_notrace(); \ barrier(); \ preempt_check_resched_context(); \ } while (0)Can you try wrapping the _outside_ of those macros with
barrier(), e.g.#define
preempt_disable_notrace()\do { \ barrier(); \ inc_preempt_count_notrace(); \ barrier(); \ } while (0)#define
preempt_enable_notrace()\do { \ preempt_enable_no_resched_notrace(); \ barrier(); \ preempt_check_resched_context(); \ barrier(); \ } while (0)and try it out with the apparently buggy compiler to see if it helps ? It does look like the preempt inc or dec is slipping out and somehow triggers the
might_sleep()warning. I don't see clearly how this could happen yet, since each of the inc/dec and the test are touchingpreempt_count(), but it's worth a try.Maybe on ARM the
current_thread_info()macro somehow hides important info from the compiler and it mistakenly reorders inc/dec vs the test. Another thing to try out (in addition to the first one) would be to try changingcurrent_thread_info(), e.g., by turning asm ("sp") into a volatile inline assembly, and by adding "memory" clobbers to it.Thoughts ?
Thanks,
Mathieu
Quoted 8 lines
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
Quoted 47 lines · Mathieu Desnoyers
Hi,
I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the
current_thread_info()inline assembly. The only logical explanation for his issue I see so far is that read of thepreempt_countwithinmight_sleep()is reordered withpreempt_enable()orpreempt_disable(). AFAIU, this kind of issue also applies to other architectures.First thing, preempt enable/disable only contains
barrier()between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that thepreempt_count()read inmight_sleep()is not reordered with the preempt count inc/dec.However, looking at ARM
arch/arm/include/asm/thread_info.h:static inline struct
thread_info*current_thread_info(void){ register unsigned long sp asm ("sp"); return (structthread_info*)(sp & ~(THREAD_SIZE - 1)); }The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)
As long as someone does:
struct
thread_info*myinfo =current_thread_info();load from
myinfo->preempt_count; store tomyinfo->preempt_count;The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of
might_sleep()followed bypreempt_enable()is:load from current_thread_info()->preempt_count; store to current_thread_info()->preempt_count;
Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).
Quoted 8 lines
Have any of you experienced this issue ? Any thoughts on the matter ?
I'm providing the original bug report email exchange below. I had confirmation that adding
barrier()outside preempt_enable()/preempt_disable() does indeed work-around the issue, but I fear that this work-around would be leaving thecurrent_thread_info()semantic gap in place: people expect program order to be respected.
This all brings to mind commit 509eb76ebf977 ("ARM: 7747/1: pcpu: ensure __my_cpu_offset cannot be re-ordered across barrier()").
GCC seems overly happy with reordering asm() thingies.
Ideally we'd be able to tell gcc that:
register unsigned long *sp asm ("sp")
not only acts like a variable but should be treated like a variable and thus such reshuffeling should be limited to preserve program-order.
There doesn't seem to be any __attribute__() that can do this, and currently out only recourse seems to add either volatile, which seems overly strict as Mathieu said, or add fake input like Will did.
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Mathieu DesnoyersQuoted 1 line · Mathieu Desnoyers
Hi,
Hi Mathieu,
Quoted 3 lines
I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the
current_thread_info()inline assembly. The only logical explanation for his issue I see so far is that read of thepreempt_countwithinmight_sleep()is reordered withpreempt_enable()orpreempt_disable(). AFAIU, this kind of issue also applies to other architectures.First thing, preempt enable/disable only contains
barrier()between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that thepreempt_count()read inmight_sleep()is not reordered with the preempt count inc/dec.
This sounds almost identical to an issue I experienced with our optimised per-cpu code (more below).
Quoted 17 lines
However, looking at ARM
arch/arm/include/asm/thread_info.h:static inline struct
thread_info*current_thread_info(void){ register unsigned long sp asm ("sp"); return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); }The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)
As long as someone does:
struct
thread_info*myinfo =current_thread_info();load from
myinfo->preempt_count;
store tomyinfo->preempt_count;
Quoted 8 lines
The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of
might_sleep()followed bypreempt_enable()is:load from current_thread_info()->preempt_count;
store to current_thread_info()->preempt_count;Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).
Yup, that sucks, because you end up unable to cache the value when you know it hasn't changed.
Quoted 1 line
Have any of you experienced this issue ? Any thoughts on the matter ?
The way I got around this for the per-cpu code is to include a dummy memory constraint for the stack. This has a couple of advantages:
(1) It hazards against a "memory" clobber, so doesn't require use of `volatile'
(2) It doesn't require GCC to emit any address generation code, since dereferencing the sp is valid in ARM assembly
so adding something like:
asm("" :: "Q" (*sp));immediately after the declaration of sp in current_therad_info might do the trick. Do you have a way to test that?
Cheers,
Will
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Peter ZijlstraQuoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers
However, looking at ARM
arch/arm/include/asm/thread_info.h:static inline struct
thread_info*current_thread_info(void){ register unsigned long sp asm ("sp"); return (structthread_info*)(sp & ~(THREAD_SIZE - 1)); }The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)
The above is not inline assembly, it is a local register variable extension, see
http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars
Quoted 3 lines
Quoted 3 lines
Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.
Sure.
Quoted 3 lines
Quoted 3 lines
One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).
As it is not inline asm extension, you can't.
Of course you could add asm volatile ("" : "=r" (ret) : "0" (sp)); or similar and thus make it a barrier, but I think the current definition of current_thread_info meant to avoid all that extra overhead. Why does it matter when sp is different between the current_thread_info () calls? As long as sp & ~(THREAD_SIZE - 1) is the same, it shouldn't make a difference. Or is the call in between those changing sp to something else?
Jakub
Re: current_thread_info() not respecting program order with gcc 4.8.x
Quoted 26 lines · Jakub Jelinek
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers
However, looking at ARM
arch/arm/include/asm/thread_info.h:static inline struct
thread_info*current_thread_info(void){ register unsigned long sp asm ("sp"); return (structthread_info*)(sp & ~(THREAD_SIZE - 1)); }The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)
The above is not inline assembly, it is a local register variable extension, see
http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-VarsQuoted 3 lines
Quoted 3 lines
Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.Sure.
Quoted 3 lines
Quoted 3 lines
One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).
Quoted 8 lines
As it is not inline asm extension, you can't.
Of course you could add asm volatile ("" : "=r" (ret) : "0" (sp)); or similar and thus make it a barrier, but I think the current definition ofcurrent_thread_infomeant to avoid all that extra overhead. Why does it matter when sp is different between thecurrent_thread_info() calls? As long as sp & ~(THREAD_SIZE - 1) is the same, it shouldn't make a difference. Or is the call in between those changing sp to something else?
So what appears to be happening is that:
preempt_enable()
barrier();
current_thread_info()->preempt_count--;
(1)
might_sleep()
if (current_thread_info()->preempt_count)
/* raise all bloody hell */Gets re-ordered like:
barrier(); if (current_thread_info()->preempt_count) /* raise hell */ current_thread_info()->preempt_count--;
And by inserting a barrier() at (1) things revert to normal and work again.
The "sp" reg value itself doesn't change here.
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Will Deacon----- Original Message -----
Quoted 9 lines
From: "Will Deacon" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: [email protected], "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Linus Torvalds" <[email protected]>, "Andrew Morton" <[email protected]>
Sent: Tuesday, November 19, 2013 11:05:02 AM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.x
Quoted 4 lines
Quoted 1 line · Mathieu Desnoyers
Hi,
Hi Mathieu,
Hi Will,
Quoted 64 lines
Quoted 8 lines
I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the
current_thread_info()inline assembly. The only logical explanation for his issue I see so far is that read of thepreempt_countwithinmight_sleep()is reordered withpreempt_enable()orpreempt_disable(). AFAIU, this kind of issue also applies to other architectures.Quoted 5 lines
First thing, preempt enable/disable only contains
barrier()between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that thepreempt_count()read inmight_sleep()is not reordered with the preempt count inc/dec.This sounds almost identical to an issue I experienced with our optimised per-cpu code (more below).
Quoted 26 lines
However, looking at ARM
arch/arm/include/asm/thread_info.h:static inline struct
thread_info*current_thread_info(void){ register unsigned long sp asm ("sp"); return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); }The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)
As long as someone does:
struct
thread_info*myinfo =current_thread_info();load from
myinfo->preempt_count;
store tomyinfo->preempt_count;The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of
might_sleep()followed bypreempt_enable()is:load from current_thread_info()->preempt_count;
store to current_thread_info()->preempt_count;Quoted 6 lines
Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).
Yup, that sucks, because you end up unable to cache the value when you know it hasn't changed.
Quoted 1 line
Have any of you experienced this issue ? Any thoughts on the matter ?
The way I got around this for the per-cpu code is to include a dummy memory constraint for the stack. This has a couple of advantages:
(1) It hazards against a "memory" clobber, so doesn't require use of `volatile'
(2) It doesn't require GCC to emit any address generation code, since dereferencing the sp is valid in ARM assembly
Quoted 6 lines
so adding something like:
asm("" :: "Q" (*sp));immediately after the declaration of sp in
current_therad_infomight do the trick. Do you have a way to test that?
Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.
It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:
struct thread_info *ptra, *ptrb;
ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();
load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;
One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ? AFAIU, local reg vars appear to be architecture-specific, so maybe there is something fishy on ARM ?
Thanks,
Mathieu
Quoted 5 lines
Cheers,
Will
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
Quoted 24 lines · Mathieu Desnoyers
Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.
It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:
struct
thread_info*ptra, *ptrb;ptra =
current_thread_info();
/*
* eachcurrent_thread_info()would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb =current_thread_info();load from
ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store toptrb->preempt_count;
Quoted 5 lines
One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ? AFAIU, local reg vars appear to be architecture-specific, so maybe there is something fishy on ARM ?
Might help if you ask where the GCC people are on CC ;-)
Multiple local register variables w/ same register
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Mathieu Desnoyers
Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.
It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:
struct
thread_info*ptra, *ptrb;ptra =
current_thread_info();
/*
* eachcurrent_thread_info()would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb =current_thread_info();load from
ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store toptrb->preempt_count;
Quoted 5 lines
Quoted 5 lines
One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ? AFAIU, local reg vars appear to be architecture-specific, so maybe there is something fishy on ARM ?
It appears not:
int __attribute__((noinline)) f(void)
{
{
register int x __asm__("eax");
x = 1;
}
{
register int y __asm__("eax");
return ++y; }
}
extern void abort(void);
int main(void)
{
if (f() != 2)
abort();
return 0;
}
Anyone see anything wrong with the testcase? Do we thing this sort of thing ought to work, perhaps with scopes lengthened?
r~
Re: Multiple local register variables w/ same register
Quoted 19 lines · Richard Henderson
It appears not:
int __attribute__((noinline)) f(void)
{{ register int x __asm__("eax"); x = 1; } { register int y __asm__("eax"); return ++y;}
}extern void abort(void);
int main(void)
{
if (f()!= 2)
Quoted 6 lines
abort();return 0;
}Anyone see anything wrong with the testcase? Do we thing this sort of thing ought to work, perhaps with scopes lengthened?
I'd say this is undefined, when a local register var goes out of scope, it's value can change arbitrarily. If you insert some call in between the two scopes, it will surely have clobbered value, and even if there isn't any call in between those, any insn could in theory clobber those.
Jakub
Re: Multiple local register variables w/ same register
In reply to Richard HendersonQuoted 53 lines · Richard Henderson
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Mathieu Desnoyers
Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.
It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:
struct
thread_info*ptra, *ptrb;ptra =
current_thread_info();
/*
* eachcurrent_thread_info()would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb =current_thread_info();load from
ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store toptrb->preempt_count;Quoted 6 lines
Quoted 6 lines
One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ?
AFAIU, local reg vars appear to be architecture-specific, so maybe
there is something fishy on ARM ?It appears not:
int __attribute__((noinline)) f(void)
{{ register int x __asm__("eax"); x = 1; } { register int y __asm__("eax"); return ++y;}
}extern void abort(void);
int main(void)
{
if (f()!= 2)
Quoted 6 lines
abort();return 0;
}Anyone see anything wrong with the testcase? Do we thing this sort of thing ought to work, perhaps with scopes lengthened?
In general, I see no reason why the compiler should be forced to preserve a register between blocks like that. Relying on a value magically appearing in a general-purpose register seems to me fragile at best. Code like this adds inter-block dependencies that would not normally be there, and I'd rather not gamble on how the various optimisation passes deal with it, if it is even noticed at all.
What happens if you some code clobbering eax (e.g. a function call) between those two blocks? What do you think should happen?
Signature · 2 lines
Måns Rullgård [email protected]
Re: Multiple local register variables w/ same register
In reply to Richard Henderson----- Original Message -----
Quoted 62 lines
From: "Richard Henderson" <[email protected]>
To: "Peter Zijlstra" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Cc: "Will Deacon" <[email protected]>, [email protected], "Catalin Marinas" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Linus Torvalds" <[email protected]>, "Andrew Morton" <[email protected]>, "Jakub Jelinek" <[email protected]>, [email protected]
Sent: Tuesday, November 19, 2013 4:56:57 PM
Subject: Multiple local register variables w/ same registerQuoted 27 lines · Peter Zijlstra
Quoted 26 lines · Mathieu Desnoyers
Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test
it for us though.It might shuffle things around enough to work around the issue, but with the
approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:struct
thread_info*ptra, *ptrb;ptra =
current_thread_info();
/*
* eachcurrent_thread_info()would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb =current_thread_info();load from
ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store toptrb->preempt_count;Quoted 8 lines
Quoted 8 lines
One question that might be worth asking: with the local register variable extension
(http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables
bound to the same register as being aliased or not ? AFAIU, local reg vars appear
to be architecture-specific, so maybe there is something fishy on ARM ?It appears not:
int __attribute__((noinline)) f(void)
{{ register int x __asm__("eax"); x = 1; } { register int y __asm__("eax"); return ++y;}
}extern void abort(void);
Quoted 8 lines
int main(void)
{
if (f()!= 2)abort();return 0;
}Anyone see anything wrong with the testcase?
This testcase is targeting a general purpose register, whereas the issue I'm presenting gets the stack pointer as base address for many memory operations targeting the same offset from this base address. So strictly speaking, I think the two cases are slightly different.
Thanks,
Mathieu
Quoted 6 lines
Do we thing this sort of thing
ought to work, perhaps with scopes lengthened?r~
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [lttng-dev] Multiple local register variables w/ same register
----- Original Message -----
Quoted 88 lines
From: "Mathieu Desnoyers" <[email protected]>
To: "Richard Henderson" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, [email protected], "Peter Zijlstra" <[email protected]>, "Catalin Marinas" <[email protected]>, "Nathan Lynch" <[email protected]>, [email protected], "Will Deacon" <[email protected]>, [email protected], "Andrew Morton" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Linus Torvalds" <[email protected]>
Sent: Tuesday, November 19, 2013 5:25:18 PM
Subject: Re: [lttng-dev] Multiple local register variables w/ same register----- Original Message -----
Quoted 78 lines
From: "Richard Henderson" <[email protected]>
To: "Peter Zijlstra" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Cc: "Will Deacon" <[email protected]>, [email protected], "Catalin Marinas" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney"
<[email protected]>, "Linus Torvalds" <[email protected]>, "Andrew Morton" <[email protected]>, "Jakub Jelinek" <[email protected]>, [email protected]
Sent: Tuesday, November 19, 2013 4:56:57 PM
Subject: Multiple local register variables w/ same registerQuoted 29 lines · Peter Zijlstra
Quoted 28 lines · Mathieu Desnoyers
Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could
test
it for us though.It might shuffle things around enough to work around the issue, but with the
approach you propose, I would be concerned about the compiler being within
its rights to reorder the code into the following sequence:struct
thread_info*ptra, *ptrb;ptra =
current_thread_info();
/*
* eachcurrent_thread_info()would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb =current_thread_info();load from
ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store toptrb->preempt_count;Quoted 10 lines
Quoted 10 lines
One question that might be worth asking: with the local register variable
extension
(http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables
bound to the same register as being aliased or not ? AFAIU, local reg vars
appear
to be architecture-specific, so maybe there is something fishy on ARM ?It appears not:
int __attribute__((noinline)) f(void) {
{ register int x __asm__("eax"); x = 1; } { register int y __asm__("eax"); return ++y;}
}extern void abort(void);
int main(void)
{
if (f()!= 2)abort();return 0;
}
Quoted 6 lines
Quoted 1 line
Anyone see anything wrong with the testcase?
This testcase is targeting a general purpose register, whereas the issue I'm presenting gets the stack pointer as base address for many memory operations targeting the same offset from this base address. So strictly speaking, I think the two cases are slightly different.
By the way, I just had more info from Nathan, where a splat clearly shows a scheduling while atomic, which points rather to an unbalanced preempt enable/disable within LTTng. It's weird though, because so far it only triggers on ARM, and only with gcc 4.8.x. Don't waste your time on this one until we can figure out if the issue is within LTTng.
Thanks,
Mathieu
Quoted 23 lines
Thanks,
Mathieu
Quoted 6 lines
Do we thing this sort of thing
ought to work, perhaps with scopes lengthened?r~
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Mathieu DesnoyersQuoted 2 lines · Mathieu Desnoyers
Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.
I don't understand why you say that.
The ordering of the access to the asm("sp") register is totally irrelevant. You are "correct" in saying that the compiler is within its right to re-order them, but that is the worst kind of correct:
it's totally immaterial. In fact, we *want* the compiler to not just re-order the accesses to %sp, but to notice that it can combine them, and do CSE on that whole expression when it is used multiple times within the same function (like it often is used).
So the compiler can very much decide to re-read %sp all it wants, and re-order those reads all it wants, and that's not the bug at all. Putting a clobber or a volatile on it would disable the optimization we *want* to happen.
So don't bark up the wrong tree.
The bug seems to be that gcc re-orders the *memory* *accesses* through that point, which is not correct in any way, shape, or form. If we have a write to a memory location followed by a read of the same memory location, the compiler ABSOLUTELY MUST NOT RE-ORDER THEM. The write obviously changes the value of the read.
It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.
There may be some insane "two different objects on the stack cannot alias" logic, which is true for *objects* on the stack, but it sure as hell isn't true for random accesses through asm("sp").
If I read this thread correctly, you're all talking about something else than the actual bug, and are trying to say that there is something wrong with re-ordering the access to %sp itself. Missing the _real_ bug entirely. See above.
Linus
Re: current_thread_info() not respecting program order with gcc 4.8.x
----- Original Message -----
Quoted 46 lines
From: "Linus Torvalds" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Jakub Jelinek" <[email protected]>, "Richard Henderson" <[email protected]>
Cc: "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>
Sent: Tuesday, November 19, 2013 7:41:17 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 3 lines · Mathieu Desnoyers
Since each
current_thread_info()is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.I don't understand why you say that.
The ordering of the access to the
asm("sp")register is totally irrelevant. You are "correct" in saying that the compiler is within its right to re-order them, but that is the worst kind of correct:
it's totally immaterial. In fact, we *want* the compiler to not just re-order the accesses to %sp, but to notice that it can combine them, and do CSE on that whole expression when it is used multiple times within the same function (like it often is used).So the compiler can very much decide to re-read %sp all it wants, and re-order those reads all it wants, and that's not the bug at all. Putting a clobber or a volatile on it would disable the optimization we *want* to happen.
So don't bark up the wrong tree.
The bug seems to be that gcc re-orders the *memory* *accesses* through that point, which is not correct in any way, shape, or form. If we have a write to a memory location followed by a read of the same memory location, the compiler ABSOLUTELY MUST NOT RE-ORDER THEM. The write obviously changes the value of the read.
It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.
Quoted 8 lines
There may be some insane "two different objects on the stack cannot alias" logic, which is true for *objects* on the stack, but it sure as hell isn't true for random accesses through
asm("sp").If I read this thread correctly, you're all talking about something else than the actual bug, and are trying to say that there is something wrong with re-ordering the access to %sp itself. Missing the _real_ bug entirely. See above.
Yes, exactly, your explanation clarifies the underlying issue I'm trying to point at.
Thank you !
Mathieu
Quoted 3 lines
Linus
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Linus TorvaldsAm 20.11.2013 01:41, schrieb Linus Torvalds:
Quoted 6 lines
It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.
Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that current_thread_info() has the prototype
static inline struct thread_info *current_thread_info(void) __attribute_const__;
on arm (and arm64 and unicore32, something the paste from Mathieu missed so most people here might have missed that detail too). It's a very good finding from Luis.
I'm writing this message because his mail doesn't have an in-reply-to header, so it might be missed in this thread.
As Luis said, declaring current_thread_info() as a const function is wrong. The gcc manual says:
----
const
Many functions do not examine any values except their arguments, and have no effects except the return value. Basically this is just slightly more strict class than the pure attribute below, since function is not allowed to read global memory.
Note that a function that has pointer arguments and examines the data pointed to must not be declared const. Likewise, a function that calls a non-const function usually must not be const. It does not make sense for a const function to return void.
----
So current_thread_info() clearly violates the constrain to not read global memory. Or in other words, that __attribute_const__ tells gcc explicitly that the two reads are pointing to different locations because they are assumed to be local (through the const).
So This might be the reason why gcc misses that different calls to current_thread_info() might point to the same memory location.
As I've an arm gcc 4.8.1 ready too, I'm joining Luis question where the reordering can be found. If someone would point me to the source/object where this happens, I could have a look if removing the __attribute_const__ makes a difference.
Regards,
Alexander Holler
Re: current_thread_info() not respecting program order with gcc 4.8.x
Thanks for putting my e-mail in the right thread Alexander.
I also wanted to point out that we recently were hit by the following bug:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
which can mess up the handling of SP and cause strange failures in the kernel.
From the description of the bug in this thread, we don't think the bugs are related but whoever is going to try removing the "const attribute" please also make sure that the bug still reproduces after applying the fix for the GCC bug above.
Luis
Quoted 58 lines · Alexander Holler
Am 20.11.2013 01:41, schrieb Linus Torvalds:
Quoted 6 lines
It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.
Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that
current_thread_info()has the prototypestatic inline struct
thread_info*current_thread_info(void)__attribute_const__;on arm (and arm64 and unicore32, something the paste from Mathieu missed so most people here might have missed that detail too). It's a very good finding from Luis.
I'm writing this message because his mail doesn't have an in-reply-to header, so it might be missed in this thread.
As Luis said, declaring
current_thread_info()as a const function is wrong. The gcc manual says:----
constMany functions do not examine any values except their arguments, and have no effects except the return value. Basically this is just slightly more strict class than the pure attribute below, since function is not allowed to read global memory.
Note that a function that has pointer arguments and examines the data pointed to must not be declared const. Likewise, a function that calls a non-const function usually must not be const. It does not make sense for a const function to return void.
----So
current_thread_info()clearly violates the constrain to not read global memory. Or in other words, that__attribute_const__tells gcc explicitly that the two reads are pointing to different locations because they are assumed to be local (through the const).So This might be the reason why gcc misses that different calls to
current_thread_info()might point to the same memory location.As I've an arm gcc 4.8.1 ready too, I'm joining Luis question where the reordering can be found. If someone would point me to the source/object where this happens, I could have a look if removing the
__attribute_const__makes a difference.Regards,
Alexander Holler
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Alexander HollerQuoted 10 lines · Alexander Holler
Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that
current_thread_info()has the prototypestatic inline struct
thread_info*current_thread_info(void)__attribute_const__;on arm (and arm64 and unicore32, something the paste from Mathieu missed so most people here might have missed that detail too). It's a very good finding from Luis.
No, because it is immaterial.
We *want* gcc to optimize away multiple accesses to "sp". Because it doesn't *matter* whether "sp" changes or not, the *result* is always the same. That's what the "const" means.
The "& ~(THREAD_SIZE - 1)" part will remove all the bits that can change. Really. So the result *is* constant (within one thread). Marking it constant and telling gcc that it can combine these things is correct.
Guys, read my email again.
The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.
The bug is *outside* that "current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.
Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.
Linus
Re: current_thread_info() not respecting program order with gcc 4.8.x
Quoted 14 lines · Linus Torvalds
Quoted 8 lines · Alexander Holler
Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that
current_thread_info()has the prototypestatic inline struct
thread_info*current_thread_info(void)__attribute_const__;on arm (and arm64 and unicore32, something the paste from Mathieu missed
so
Quoted 2 lines
most people here might have missed that detail too). It's a very good finding from Luis.
Can someone provide a test case? We are unable to reproduce the problem in ChromeOS land.
Quoted 12 lines
No, because it is immaterial.
We *want* gcc to optimize away multiple accesses to "sp". Because it doesn't *matter* whether "sp" changes or not, the *result* is always the same. That's what the "const" means.
The "& ~(THREAD_SIZE - 1)" part will remove all the bits that can change. Really. So the result *is* constant (within one thread). Marking it constant and telling gcc that it can combine these things is correct.
Its unfortunate that the attribute '__const__ ' is very misleading - the semantic used by gcc is that this is an aliasing attribute that implies that the function does not depend in any way on global state. Meaning, if called with the same parameters, it expected to return the same value. It should have been called 'pure', but gcc has a different 'pure' attribute that imples something different :-(
That said, gcc should be smart enough to figure out that this function is
not doing anything complicated. Again, a test case would be much
appreciated.
Bhaskar
Quoted 5 lines
Guys, read my email again.
The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.
Quoted 10 lines
The bug is *outside* that "
current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.
Linus
Quoted 1 line
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Linus TorvaldsAm 21.11.2013 23:32, schrieb Linus Torvalds:
Quoted 4 lines · Alexander Holler
The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.
Sure, and I don't disagree on that.
Quoted 9 lines
The bug is *outside* that "
current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.
Sorry, that I still disagree.
I try to describe it more clearly why I still think that the problem might be because of that const declaration.
(...)
foobar1 = current_thread_info() __attribute_const__ {
return sp->somewhere_local;
}(...)
foobar2 = current_thread_info() __attribute_const__ {
return sp->somewhere_local;
}So, even if sp is the same in both cases, that const states that wherever sp points to is local to current_thread_info(), so it can't be the same for both cases.
Regards,
Alexander Holler
Re: current_thread_info() not respecting program order with gcc 4.8.x
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Luis
Quoted 49 lines · Alexander Holler
Am 21.11.2013 23:32, schrieb Linus Torvalds:
Quoted 6 lines
On Thu, Nov 21, 2013 at 8:02 AM, Alexander Holler <[email protected]> wrote:
The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.
Sure, and I don't disagree on that.
Quoted 9 lines
The bug is *outside* that "
current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.
Sorry, that I still disagree.
I try to describe it more clearly why I still think that the problem might be because of that const declaration.
(...)
foobar1 = current_thread_info() __attribute_const__ { return sp->somewhere_local; }(...)
foobar2 = current_thread_info() __attribute_const__ { return sp->somewhere_local; }So, even if sp is the same in both cases, that const states that wherever sp points to is local to
current_thread_info(), so it can't be the same for both cases.Regards,
Alexander Holler
Signature · 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Alexander HollerQuoted 2 lines · Alexander Holler
Sorry, that I still disagree.
Why? You're wrong.
I mean, anybody who disagrees with me is pretty much wrong just on pure principles, but I actually mean a deeper kind of wrong than that. I mean a very objective "you're clearly wrong".
Quoted 2 lines
I try to describe it more clearly why I still think that the problem might be because of that const declaration.
.. and then you use a totally bogus example to try to "prove" your point.
The example you use has nothing to do with what the function in case actually does.
The function we actually talk about RETURNS THE SAME VALUE EVERY TIME IT IS CALLED, regardless of arguments (which it doesn't happen to have). Ergo, it is "const".
Your example is pure and utter shit, since you still get confused about what is actually const and what isn't.
The fact is, "current_thread_info()" returns a constant value (within that execution context). Always has, always will. It fundamentally HAS to, since that is - by definition - what that "thread info" is all about.
The dereference (your "->somewhere_local") happens *outside* that const function scope. And no, that dereference is not const. But that's not what we tell gcc, and never has been.
So don't try to confuse things by trying to make
"current_thread_info()" something it isn't.
Basically, your whole argument boils down to "if the function did something else than what it does, then it wouldn't be const, so we shouldn't mark it const". But that argument is BULLSHIT, because the fact is, the function *doesn't* do what you try to claim it does.
Linus
Re: current_thread_info() not respecting program order with gcc 4.8.x
Am 22.11.2013 01:17, schrieb Linus Torvalds:
Quoted 6 lines · Alexander Holler
Basically, your whole argument boils down to "if the function did something else than what it does, then it wouldn't be const, so we shouldn't mark it const". But that argument is BULLSHIT, because the fact is, the function *doesn't* do what you try to claim it does.
Maybe gcc just makes the same false conclusion as I did in my description.
I read it as current_thread_info() returns "a pointer to something local" instead of returns "a pointer". Might be BULLSHIT but would explain the bug which seems to exist.
Alexander Holler
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Luis LozanoQuoted 5 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.
Jakub
Re: current_thread_info() not respecting program order with gcc 4.8.x
----- Original Message -----
Quoted 18 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 5 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Quoted 7 lines
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.
The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules compiled against that kernel.
First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.
Then I looked at gcc 4.8 changelog for ARM, new feature: -fno-sched-pressure (sched pressure is there by default). Nathan tried compiling lttng-modules with -fno-sched-pressure. The problem still reproduces.
Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:
Has the following function. We identified that adding a barrier() as shown below works around the issue:
static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,
uint32_t event_id)
{
struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);
int ret, cpu;
cpu = lib_ring_buffer_get_cpu(&client_config);
if (cpu < 0)
return -EPERM;
ctx->cpu = cpu;
switch (lttng_chan->header_type) {
case 1: /* compact */
if (event_id > 30)
ctx->rflags |= LTTNG_RFLAG_EXTENDED;
break;
case 2: /* large */
if (event_id > 65534)
ctx->rflags |= LTTNG_RFLAG_EXTENDED;
break;
default:
WARN_ON_ONCE(1);
}
ret = lib_ring_buffer_reserve(&client_config, ctx);
if (ret)
goto put;
lttng_write_event_header(&client_config, ctx, event_id);
return 0;
put:
lib_ring_buffer_put_cpu(&client_config);
---------------> barrier() added here; <----------------------------
return ret;
}Where barrier() is the usual asm volatile with "memory" clobber, nothing else.
Nathan gave me the binary diff for the assembly generated for this function without the barrier and with the barrier:
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]
@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4 <lttng_event_reserve+0xbd8> fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8> fe0: e3500000 cmp r0, #0
We tried disabling the ftrace function tracing to get mcount out of the way, and the problem still reproduces.
I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.
I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.
Thoughts ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
Hi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening. In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?
Thanks
Luis
Quoted 154 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 18 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 5 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Quoted 7 lines
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.
The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules compiled against that kernel.
First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.
Then I looked at gcc 4.8 changelog for ARM, new feature: -fno-sched-pressure (sched pressure is there by default). Nathan tried compiling lttng-modules with -fno-sched-pressure. The problem still reproduces.
Knowing that adding
barrier()outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:Has the following function. We identified that adding a
barrier()as shown below works around the issue:static
int lttng_event_reserve(structlib_ring_buffer_ctx*ctx,uint32_t event_id) { struct lttng_channel *lttng_chan = channel_get_private(ctx->chan); int ret, cpu; cpu = lib_ring_buffer_get_cpu(&client_config); if (cpu < 0) return -EPERM; ctx->cpu = cpu; switch (lttng_chan->header_type) { case 1: /* compact */ if (event_id > 30) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; case 2: /* large */ if (event_id > 65534) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; default: WARN_ON_ONCE(1); } ret = lib_ring_buffer_reserve(&client_config, ctx); if (ret) goto put; lttng_write_event_header(&client_config, ctx, event_id); return 0; put: lib_ring_buffer_put_cpu(&client_config); ---------------> barrier() added here; <---------------------------- return ret; }Where
barrier()is the usual asm volatile with "memory" clobber, nothing else.Nathan gave me the binary diff for the assembly generated for this function without the barrier and with the barrier:
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5
-5@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4 <lttng_event_reserve+0xbd8> fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8> fe0: e3500000 cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the way, and the problem still reproduces.
I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.
I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of
kmem_cache_freetracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.Thoughts ?
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
Re: current_thread_info() not respecting program order with gcc 4.8.x
----- Original Message -----
Quoted 11 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.x
Quoted 8 lines
Hi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.
The following instruction
f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8
appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3, which makes me wonder if it's good or not. With slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:
00000efc <lttng_event_reserve>:
efc: e1a0c00d mov ip, sp
f00: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
f04: e24cb004 sub fp, ip, #4
f08: e24dd03c sub sp, sp, #60 ; 0x3c
f0c: e52de004 push {lr} ; (str lr, [sp, #-4]!)
f10: ebfffffe bl 0 <__gnu_mcount_nc>
f14: e5903000 ldr r3, [r0]
f18: e1a04000 mov r4, r0
f1c: e1a0000d mov r0, sp
f20: e5936030 ldr r6, [r3, #48] ; 0x30
f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0
f28: e3c3303f bic r3, r3, #63 ; 0x3f
f2c: e50b104c str r1, [fp, #-76] ; 0xffffffb4
f30: e5932004 ldr r2, [r3, #4]
f34: e2822001 add r2, r2, #1
f38: e5832004 str r2, [r3, #4]
f3c: ebfffffe bl 0 <debug_smp_processor_id>
f40: e59f2e44 ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90>
f44: e59f3e44 ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94>
f48: e7921100 ldr r1, [r2, r0, lsl #2]
f4c: e1a05000 mov r5, r0
f50: e7932001 ldr r2, [r3, r1]My rusty ARM assembler knowledge analyzes this like:
ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76
So yes, it does look like the -76 from fp is within the stack.
Quoted 4 lines
In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?
My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning, did not manage to get one working even after following 2 howtos, trying with Debian packages, etc. Nathan told me this diff was the full comparison between the two functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.
But after a long day of debugging, it's time for some sleep. I will get back to this tomorrow,
Thanks,
Mathieu
Quoted 189 lines
Thanks
Luis
Quoted 165 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 24 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 5 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Quoted 7 lines
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.
The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules with
-fno-sched-pressure. The problem still reproduces.Knowing that adding
barrier()outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:Has the following function. We identified that adding a
barrier()as shown below
works around the issue:static
int lttng_event_reserve(structlib_ring_buffer_ctx*ctx,uint32_t event_id) { struct lttng_channel *lttng_chan = channel_get_private(ctx->chan); int ret, cpu; cpu = lib_ring_buffer_get_cpu(&client_config); if (cpu < 0) return -EPERM; ctx->cpu = cpu; switch (lttng_chan->header_type) { case 1: /* compact */ if (event_id > 30) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; case 2: /* large */ if (event_id > 65534) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; default: WARN_ON_ONCE(1); } ret = lib_ring_buffer_reserve(&client_config, ctx); if (ret) goto put; lttng_write_event_header(&client_config, ctx, event_id); return 0; put: lib_ring_buffer_put_cpu(&client_config); ---------------> barrier() added here; <---------------------------- return ret; }Where
barrier()is the usual asm volatile with "memory" clobber, nothing else.Nathan gave me the binary diff for the assembly generated for this function without
the barrier and with the barrier:tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0
-014:12:52.997355907 -0600
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5
-5@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4 <lttng_event_reserve+0xbd8> fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8> fe0: e3500000 cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of
kmem_cache_freetracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.Thoughts ?
Thanks,
Quoted 7 lines
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com--
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
Quoted 27 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 3 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds"
Quoted 1 line
<[email protected]>, "Richard Henderson" <[email protected]>,
"Linux Kernel Mailing List"
Quoted 1 line
<[email protected]>, "Will Deacon" <[email protected]>,
"Catalin Marinas" <[email protected]>,
Quoted 1 line
"Peter Zijlstra" <[email protected]>, [email protected],
"Nathan Lynch" <[email protected]>, "Paul
Quoted 1 line
E. McKenney" <[email protected]>, "Andrew Morton" <
[email protected]>, "Bhaskar Janakiraman"
Quoted 3 lines
<[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re:current_thread_info()not respecting program order with gcc4.8.x
Quoted 9 lines
Hi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.
Quoted 6 lines
The following instruction
f1c:
e50b1048str r1, [fp, #-72] ;0xffffffb8appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3, which makes me wonder if it's good or not.
It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just instructions ordered a little different?
Luis
Quoted 280 lines
With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:
00000efc<lttng_event_reserve>:efc: e1a0c00d mov ip, sp f00: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc} f04: e24cb004 sub fp, ip, #4 f08: e24dd03c sub sp, sp, #60 ; 0x3c f0c: e52de004 push {lr} ; (str lr, [sp, #-4]!) f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 f28: e3c3303f bic r3, r3, #63 ; 0x3f f2c: e50b104c str r1, [fp, #-76] ; 0xffffffb4 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4] f3c: ebfffffe bl 0 <debug_smp_processor_id> f40: e59f2e44 ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90> f44: e59f3e44 ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94> f48: e7921100 ldr r1, [r2, r0, lsl #2] f4c: e1a05000 mov r5, r0 f50: e7932001 ldr r2, [r3, r1]My rusty ARM assembler knowledge analyzes this like:
ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76So yes, it does look like the -76 from fp is within the stack.
Quoted 4 lines
In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning,
did not manage to get one working even after following 2 howtos, trying with Debian
packages, etc. Nathan told me this diff was the full comparison between the two
functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.But after a long day of debugging, it's time for some sleep. I will get back to this
tomorrow,Thanks,
Mathieu
Quoted 16 lines
Thanks
Luis
Quoted 6 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 5 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
"Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order withgcc
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
4.8.x
Quoted 1 line · Luis Lozano
I think we need a reproducer. Without this we may all be going on
the
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed
source,
Quoted 1 line
Quoted 1 line
Quoted 1 line
compiler version, flags and how it was configured and some hint in
which
Quoted 1 line
Quoted 1 line
Quoted 1 line
function to look for what exactly? We don't necessarily need a
runtime
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned
input
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full
day
Quoted 1 line
Quoted 1 line
of bug hunting. We're not there yet, but many options have been
eliminated.
Quoted 2 lines
Quoted 2 lines
The issue shows up in stress test, when tracing with lttng-modules
2.4-rc1,
Quoted 5 lines
Quoted 5 lines
on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.First, I asked Nathan to compile his kernel with gcc 4.7, and
lttng-modules
Quoted 3 lines
Quoted 3 lines
with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile therest
Quoted 5 lines
Quoted 5 lines
of the kernel does not matter.
Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compilinglttng-modules
Quoted 5 lines
Quoted 5 lines
with
-fno-sched-pressure. The problem still reproduces.Knowing that adding
barrier()outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location wasresponsible
Quoted 1 line
Quoted 1 line
for working around the issue. Skipping a long investigation, here is
the
Quoted 3 lines
Quoted 3 lines
executive summary:
Quoted 2 lines
Quoted 2 lines
Has the following function. We identified that adding a
barrier()asshown
Quoted 8 lines
Quoted 8 lines
below
works around the issue:static
int lttng_event_reserve(structlib_ring_buffer_ctx*ctx,uint32_t event_id){
structlttng_channel*lttng_chan=
channel_get_private(ctx->chan);Quoted 32 lines
Quoted 32 lines
int ret, cpu; cpu = lib_ring_buffer_get_cpu(&client_config); if (cpu < 0) return -EPERM; ctx->cpu = cpu; switch (lttng_chan->header_type) { case 1: /* compact */ if (event_id > 30) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; case 2: /* large */ if (event_id > 65534) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; default: WARN_ON_ONCE(1); } ret = lib_ring_buffer_reserve(&client_config, ctx); if (ret) goto put; lttng_write_event_header(&client_config, ctx, event_id); return 0; put: lib_ring_buffer_put_cpu(&client_config); ---------------> barrier() added here; <---------------------------- return ret; }Quoted 1 line
Quoted 1 line
Where
barrier()is the usual asm volatile with "memory" clobber,nothing
Quoted 3 lines
Quoted 3 lines
else.
Nathan gave me the binary diff for the assembly generated for this
function
Quoted 4 lines
Quoted 4 lines
without
the barrier and with the barrier:---
/tmp/lttng_event_reserve-4.8.2.dump2013-11-21 11:14:14.536495079-0600
Quoted 16 lines
Quoted 16 lines
+++
/tmp/lttng_event_reserve-with-barrier-4.8.2.dump2013-11-21
14:12:52.997355907 -0600tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +4
-4@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4]Quoted 8 lines
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 line 56
Quoted 8 lines
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +1
-1f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4<
lttng_event_reserve+0xbd8>Quoted 2 lines
Review line 58
Quoted 2 lines
(quoted diff) +0
-0fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4<
lttng_event_reserve+0xbd8>Quoted 8 lines
Quoted 8 lines
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0
-0fe0: e3500000 cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for
the
Quoted 8 lines
Quoted 8 lines
right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
Quoted 8 lines
Quoted 8 lines
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of
kmem_cache_freetracepoint with hackbench running. A race between a short window of stack use below sp and interrupthandlers
Quoted 18 lines
Quoted 6 lines
would trigger with this kind of stress-test.
Thoughts ?
Thanks,
Quoted 7 lines
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com--
Quoted 9 lines
Quoted 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1
(408)431-5164
Quoted 1 line
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Mathieu DesnoyersQuoted 22 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 11 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 8 lines
Hi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.
Quoted 6 lines
The following instruction
f1c:
e50b1048str r1, [fp, #-72] ;0xffffffb8appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3,
It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just the instructions ordered a little different?
Luis
Quoted 255 lines
which makes me wonder if it's good or not. With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:
00000efc<lttng_event_reserve>:efc: e1a0c00d mov ip, sp f00: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc} f04: e24cb004 sub fp, ip, #4 f08: e24dd03c sub sp, sp, #60 ; 0x3c f0c: e52de004 push {lr} ; (str lr, [sp, #-4]!) f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 f28: e3c3303f bic r3, r3, #63 ; 0x3f f2c: e50b104c str r1, [fp, #-76] ; 0xffffffb4 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4] f3c: ebfffffe bl 0 <debug_smp_processor_id> f40: e59f2e44 ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90> f44: e59f3e44 ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94> f48: e7921100 ldr r1, [r2, r0, lsl #2] f4c: e1a05000 mov r5, r0 f50: e7932001 ldr r2, [r3, r1]My rusty ARM assembler knowledge analyzes this like:
ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76So yes, it does look like the -76 from fp is within the stack.
Quoted 4 lines
In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning, did not manage to get one working even after following 2 howtos, trying with Debian packages, etc. Nathan told me this diff was the full comparison between the two functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.
But after a long day of debugging, it's time for some sleep. I will get back to this tomorrow,
Thanks,
Mathieu
Quoted 187 lines
Thanks
Luis
Quoted 165 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 24 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 5 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Quoted 7 lines
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.
The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules with
-fno-sched-pressure. The problem still reproduces.Knowing that adding
barrier()outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:Has the following function. We identified that adding a
barrier()as shown below
works around the issue:static
int lttng_event_reserve(structlib_ring_buffer_ctx*ctx,uint32_t event_id) { struct lttng_channel *lttng_chan = channel_get_private(ctx->chan); int ret, cpu; cpu = lib_ring_buffer_get_cpu(&client_config); if (cpu < 0) return -EPERM; ctx->cpu = cpu; switch (lttng_chan->header_type) { case 1: /* compact */ if (event_id > 30) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; case 2: /* large */ if (event_id > 65534) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; default: WARN_ON_ONCE(1); } ret = lib_ring_buffer_reserve(&client_config, ctx); if (ret) goto put; lttng_write_event_header(&client_config, ctx, event_id); return 0; put: lib_ring_buffer_put_cpu(&client_config); ---------------> barrier() added here; <---------------------------- return ret; }Where
barrier()is the usual asm volatile with "memory" clobber, nothing else.Nathan gave me the binary diff for the assembly generated for this function without
the barrier and with the barrier:tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0
-014:12:52.997355907 -0600
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5
-5@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4 <lttng_event_reserve+0xbd8> fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8> fe0: e3500000 cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of
kmem_cache_freetracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.Thoughts ?
Thanks,
Quoted 7 lines
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com--
Quoted 2 lines
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Re: current_thread_info() not respecting program order with gcc 4.8.x
Quoted 28 lines · Luis Lozano
Quoted 26 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 11 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 8 lines
Hi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.The following instruction
f1c:
e50b1048str r1, [fp, #-72] ;0xffffffb8
Quoted 9 lines
Quoted 2 lines
appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3,
It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just the instructions ordered a little different?
unless there is a dependency between the following 2 instructions?
- f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8
...
f20: e5936030 ldr r6, [r3, #48] ; 0x30
and moving the str across the ldr makes a difference...
Quoted 257 lines
Luis
Quoted 255 lines
which makes me wonder if it's good or not. With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:
00000efc<lttng_event_reserve>:efc: e1a0c00d mov ip, sp f00: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc} f04: e24cb004 sub fp, ip, #4 f08: e24dd03c sub sp, sp, #60 ; 0x3c f0c: e52de004 push {lr} ; (str lr, [sp, #-4]!) f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 f28: e3c3303f bic r3, r3, #63 ; 0x3f f2c: e50b104c str r1, [fp, #-76] ; 0xffffffb4 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4] f3c: ebfffffe bl 0 <debug_smp_processor_id> f40: e59f2e44 ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90> f44: e59f3e44 ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94> f48: e7921100 ldr r1, [r2, r0, lsl #2] f4c: e1a05000 mov r5, r0 f50: e7932001 ldr r2, [r3, r1]My rusty ARM assembler knowledge analyzes this like:
ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76So yes, it does look like the -76 from fp is within the stack.
Quoted 4 lines
In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning, did not manage to get one working even after following 2 howtos, trying with Debian packages, etc. Nathan told me this diff was the full comparison between the two functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.
But after a long day of debugging, it's time for some sleep. I will get back to this tomorrow,
Thanks,
Mathieu
Quoted 187 lines
Thanks
Luis
Quoted 165 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 24 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 5 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.
evidence of the reorder or reproducer please?
Quoted 7 lines
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.
The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules with
-fno-sched-pressure. The problem still reproduces.Knowing that adding
barrier()outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:Has the following function. We identified that adding a
barrier()as shown below
works around the issue:static
int lttng_event_reserve(structlib_ring_buffer_ctx*ctx,uint32_t event_id) { struct lttng_channel *lttng_chan = channel_get_private(ctx->chan); int ret, cpu; cpu = lib_ring_buffer_get_cpu(&client_config); if (cpu < 0) return -EPERM; ctx->cpu = cpu; switch (lttng_chan->header_type) { case 1: /* compact */ if (event_id > 30) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; case 2: /* large */ if (event_id > 65534) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; default: WARN_ON_ONCE(1); } ret = lib_ring_buffer_reserve(&client_config, ctx); if (ret) goto put; lttng_write_event_header(&client_config, ctx, event_id); return 0; put: lib_ring_buffer_put_cpu(&client_config); ---------------> barrier() added here; <---------------------------- return ret; }Where
barrier()is the usual asm volatile with "memory" clobber, nothing else.Nathan gave me the binary diff for the assembly generated for this function without
the barrier and with the barrier:tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0
-014:12:52.997355907 -0600
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5
-5@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4 <lttng_event_reserve+0xbd8> fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8> fe0: e3500000 cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of
kmem_cache_freetracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.Thoughts ?
Thanks,
Quoted 7 lines
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com--
Quoted 2 lines
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to a message not in the archive----- Original Message -----
Quoted 72 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Friday, November 22, 2013 3:16:27 AM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 52 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 37 lines
From: "Luis Lozano" < [email protected] >
To: "Mathieu Desnoyers" < [email protected] >
Cc: "Jakub Jelinek" < [email protected] >, "Alexander Holler" < [email protected] >, "Linus Torvalds"
< [email protected] >, "Richard Henderson" < [email protected]
Quoted 1 line
,
"Linux Kernel Mailing List"
< [email protected] >, "Will Deacon" < [email protected] >, "Catalin Marinas" < [email protected] >,
"Peter Zijlstra" < [email protected] >, [email protected] , "Nathan Lynch" < [email protected] >, "Paul
E. McKenney" < [email protected] >, "Andrew Morton" < [email protected] >, "Bhaskar Janakiraman"
< [email protected] >, "Han Shen" < [email protected] >
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re:
current_thread_info()not respecting program order with gcc 4.8.xHi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt
handler was corrupting the stack of a routine which had an invalid
Quoted 7 lines
value of SP (not really on "top" of the stack) for part of the
routine.
But in the assembly code you sent, I don't see where sp is being
modified... or where the access to "below" sp is happening.
The following instruction
f1c:
e50b1048str r1, [fp, #-72] ;0xffffffb8
Quoted 7 lines
Quoted 3 lines
appears in the assembly generated by gcc 4.8.2, but not in the one
generated by 4.7.3, which makes me wonder if it's good or not.
It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just instructions ordered a little different?
Assembly generated by 4.7.3 vs 4.8.2 are much more different.
The diff I've presented yesterday is between 4.8.2 and 4.8.2+barrier() at the end of the function error path. When I can get a cross compiler to work, I will send out more the complete assembly listings of the function.
Thanks,
Mathieu
Quoted 525 lines
Luis
Quoted 522 lines
With
slightly different build output from the diff (probably a different config
from Nathan), the first function instructions look like:
00000efc<lttng_event_reserve>:efc:
e1a0c00dmov ip, spf00:
e92ddff0push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}f04:
e24cb004sub fp, ip, #4f08:
e24dd03csub sp, sp, #60 ;0x3cf0c:
e52de004push {lr} ; (str lr, [sp, #-4]!)f10: ebfffffe bl 0 <
__gnu_mcount_nc>f14:
e5903000ldr r3, [r0]f18:
e1a04000mov r4, r0f1c:
e1a0000dmov r0, spf20:
e5936030ldr r6, [r3, #48] ;0x30f24:
e3c03d7fbic r3, r0, #8128 ;0x1fc0f28:
e3c3303fbic r3, r3, #63 ;0x3ff2c:
e50b104cstr r1, [fp, #-76] ;0xffffffb4f30:
e5932004ldr r2, [r3, #4]f34:
e2822001add r2, r2, #1f38:
e5832004str r2, [r3, #4]f3c: ebfffffe bl 0 <
debug_smp_processor_id>f40:
e59f2e44ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90>f44:
e59f3e44ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94>f48:
e7921100ldr r1, [r2, r0, lsl #2]f4c:
e1a05000mov r5, r0f50:
e7932001ldr r2, [r3, r1]My rusty ARM assembler knowledge analyzes this like:
ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack
fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <
__gnu_mcount_nc> (dynamically patched to a pop {lr})[...]
store r1 into mem location fp - 76
So yes, it does look like the -76 from fp is within the stack.
Quoted 7 lines
In GCC bug 58854 it was pretty clear that the restore of the sp
register in the epilogue is moved to somewhere close to the prologue
of the routine.
Are we missing some diffs from the assembly comparison?
My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning,
did not manage to get one working even after following 2 howtos, trying with
Debianpackages, etc. Nathan told me this diff was the full comparison between the two
functions, but you'll understand that at this point I want to reproduce
everything myself, because this is a _weird_ issue.
But after a long day of debugging, it's time for some sleep. I will get back
to thistomorrow,
Thanks,
Mathieu
Quoted 391 lines
Thanks
Luis
On Thu, Nov 21, 2013 at 5:57 PM, Mathieu Desnoyers
Quoted 357 lines · < [email protected] > wrote:
----- Original Message -----
Quoted 60 lines
From: "Jakub Jelinek" < [email protected] >
To: "Luis Lozano" < [email protected] >
Cc "Alexander Holler" < [email protected] >, "Linus Torvalds"< [email protected] >, "Mathieu Desnoyers"
< [email protected] >, "Richard Henderson" < [email protected] >,
"Linux Kernel Mailing List"
< [email protected] >, "Will Deacon" < [email protected]
Quoted 1 line
,
"Catalin Marinas" < [email protected] >,
"Peter Zijlstra" < [email protected] >, [email protected] ,
"Nathan Lynch" < [email protected] >, "Paul
E. McKenney" < [email protected] >, "Andrew Morton"
< [email protected] >, "Bhaskar Janakiraman"
< [email protected] >, "Han Shen" < [email protected] >
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:
current_thread_info()not respecting program order with gcc4.8.x
Quoted 10 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the
wrong path. This whole conversation started on an *assumption* that
some accesses were being reordered.
evidence of the reorder or reproducer please?
Yeah, if a compiler bug is suspected, can anybody please open
a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source,
compiler version, flags and how it was configured and some hint in which
Quoted 8 lines
function to look for what exactly? We don't necessarily need a runtime
small reproducer, but if it can be shown in the assembly what has been
reordered and why you think it shouldn't, with the above mentioned input
that ought to be sufficient. Thanks.
OK OK, let me reply on list first so I can share the result of a full day
of bug hunting. We're not there yet, but many options have been eliminated.
The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1,
on ARM. It's been reproduced with a Linux kernel 3.12 so far, with
lttng-modules
compiled against that kernel.
First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules
with gcc 4.8.x (and vice-versa). The problem only appears when
lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest
of the kernel does not matter.
Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules
with
-fno-sched-pressure. The problem still reproduces.
Knowing that adding
barrier()outside of preempt_disable()/enable() wasfixing the issue, we tried identifying which code location was responsible
for working around the issue. Skipping a long investigation, here is the
executive summary:
Has the following function. We identified that adding a
barrier()as shownbelow
works around the issue:
static
int lttng_event_reserve(struct
lib_ring_buffer_ctx*ctx,
uint32_tevent_id){
struct
lttng_channel*lttng_chan=channel_get_private(ctx->chan);int ret, cpu;
cpu =
lib_ring_buffer_get_cpu(&client_config);if (cpu < 0)
return
-EPERM;
ctx->cpu= cpu;switch (
lttng_chan->header_type) {case 1: /* compact */
if (
event_id> 30)
ctx->rflags|=LTTNG_RFLAG_EXTENDED;break;
case 2: /* large */
if (
event_id> 65534)
ctx->rflags|=LTTNG_RFLAG_EXTENDED;break;
default:
WARN_ON_ONCE(1);}
ret =
lib_ring_buffer_reserve(&client_config, ctx);if (ret)
goto put;
lttng_write_event_header(&client_config, ctx, event_id);return 0;
put:
lib_ring_buffer_put_cpu(&client_config);--------------->
barrier()added here;<----------------------------
return ret;
}
Where
barrier()is the usual asm volatile with "memory" clobber, nothingelse.
Nathan gave me the binary diff for the assembly generated for this function
without
the barrier and with the barrier:
---
/tmp/lttng_event_reserve-4.8.2.dump2013-11-21 11:14:14.536495079 -0600+++
/tmp/lttng_event_reserve-with-barrier-4.8.2.dump2013-11-2114:12:52.997355907 -0600
(quoted diff) +0
-0@@ -7,11 +7,11 @@f10: ebfffffe bl 0 <
__gnu_mcount_nc>f14:
e5903000ldr r3, [r0]f18:
e1a04000mov r4, r0- f1c:
e50b1048str r1, [fp, #-72] ;0xffffffb8+ f1c:
e1a0000dmov r0, spf20:
e5936030ldr r6, [r3, #48] ;0x30- f24:
e1a0000dmov r0, sp- f28:
e3c03d7fbic r3, r0, #8128 ;0x1fc0- f2c:
e3c3303fbic r3, r3, #63 ;0x3f+ f24:
e3c03d7fbic r3, r0, #8128 ;0x1fc0+ f28:
e3c3303fbic r3, r3, #63 ;0x3f+ f2c:
e50b1048str r1, [fp, #-72] ;0xffffffb8f30:
e5932004ldr r2, [r3, #4]f34:
e2822001add r2, r2, #1f38:
e5832004str r2, [r3, #4](quoted diff) +0
-0@@ -53,7 +53,7 @@fc8:
e3c3303fbic r3, r3, #63 ;0x3ffcc:
e5933000ldr r3, [r3]fd0:
e3130002tst r3, #2- fd4:
0a000000beq fdc <lttng_event_reserve+0xe0>+ fd4:
0a0002bebeq 1ad4 <lttng_event_reserve+0xbd8>fd8: ebfffffe bl 0 <
preempt_schedule>fdc:
ea0002bcb 1ad4 <lttng_event_reserve+0xbd8>fe0:
e3500000cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the
way,
and the problem still reproduces.
I'm stopping here in terms of details about the disassembly, because I
need to double check with Nathan that I get the right disassembly for the
right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my
machine.
I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run
on top of the thread stack (?). If it's the case, then anything stored
on the stack below "sp" could be overwritten by an interrupt handler.
This would fit well the reproduction scenario for this bug: Nathan runs
LTTng tracing of
kmem_cache_freetracepoint with hackbench running.A race between a short window of stack use below sp and interrupt handlers
would trigger with this kind of stress-test.
Thoughts ?
Thanks,
Mathieu
Quoted 7 lines
--
Mathieu Desnoyers
EfficiOS Inc.
Quoted 8 lines
--
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
--
Mathieu Desnoyers
EfficiOS Inc.
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: current_thread_info() not respecting program order with gcc 4.8.x
In reply to Luis Lozano----- Original Message -----
Quoted 50 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Friday, November 22, 2013 3:18:14 AM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 28 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 17 lines
From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re:current_thread_info()not respecting program order with gcc 4.8.xQuoted 8 lines
Hi Mathieu,
Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.Quoted 6 lines
The following instruction
f1c:
e50b1048str r1, [fp, #-72] ;0xffffffb8appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3,
Quoted 5 lines
It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just the instructions ordered a little different?Luis
(resending reply in plain text)
Assembly generated by 4.7.3 vs 4.8.2 are much more different.
The diff I've presented yesterday is between 4.8.2 and 4.8.2+barrier() at the end of the function error path. When I can get a cross compiler to work, I will send out more the complete assembly listings of the function.
Thanks,
Mathieu
Quoted 290 lines
Quoted 289 lines
which makes me wonder if it's good or not. With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:
00000efc<lttng_event_reserve>:efc: e1a0c00d mov ip, sp f00: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc} f04: e24cb004 sub fp, ip, #4 f08: e24dd03c sub sp, sp, #60 ; 0x3c f0c: e52de004 push {lr} ; (str lr, [sp, #-4]!) f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 f28: e3c3303f bic r3, r3, #63 ; 0x3f f2c: e50b104c str r1, [fp, #-76] ; 0xffffffb4 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4] f3c: ebfffffe bl 0 <debug_smp_processor_id> f40: e59f2e44 ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90> f44: e59f3e44 ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94> f48: e7921100 ldr r1, [r2, r0, lsl #2] f4c: e1a05000 mov r5, r0 f50: e7932001 ldr r2, [r3, r1]My rusty ARM assembler knowledge analyzes this like:
ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76So yes, it does look like the -76 from fp is within the stack.
Quoted 4 lines
In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning,
did not manage to get one working even after following 2 howtos, trying with Debian
packages, etc. Nathan told me this diff was the full comparison between the two
functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.But after a long day of debugging, it's time for some sleep. I will get back to this
tomorrow,Thanks,
Mathieu
Quoted 211 lines
Thanks
Luis
Quoted 189 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 27 lines
From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>,
"Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re:current_thread_info()not respecting program order with gcc
4.8.xQuoted 6 lines · Luis Lozano
I think we need a reproducer. Without this we may all be going on the
wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.evidence of the reorder or reproducer please?
Quoted 11 lines
Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source,
compiler version, flags and how it was configured and some hint in which
function to look for what exactly? We don't necessarily need a runtime
small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input
that ought to be sufficient. Thanks.OK OK, let me reply on list first so I can share the result of a full day
of bug hunting. We're not there yet, but many options have been eliminated.The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1,
on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules
with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest
of the kernel does not matter.Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules
with
-fno-sched-pressure. The problem still reproduces.Knowing that adding
barrier()outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible
for working around the issue. Skipping a long investigation, here is the
executive summary:Has the following function. We identified that adding a
barrier()as shown
below
works around the issue:static
int lttng_event_reserve(structlib_ring_buffer_ctx*ctx,uint32_t event_id) { struct lttng_channel *lttng_chan = channel_get_private(ctx->chan); int ret, cpu; cpu = lib_ring_buffer_get_cpu(&client_config); if (cpu < 0) return -EPERM; ctx->cpu = cpu; switch (lttng_chan->header_type) { case 1: /* compact */ if (event_id > 30) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; case 2: /* large */ if (event_id > 65534) ctx->rflags |= LTTNG_RFLAG_EXTENDED; break; default: WARN_ON_ONCE(1); } ret = lib_ring_buffer_reserve(&client_config, ctx); if (ret) goto put; lttng_write_event_header(&client_config, ctx, event_id); return 0; put: lib_ring_buffer_put_cpu(&client_config); ---------------> barrier() added here; <---------------------------- return ret; }Where
barrier()is the usual asm volatile with "memory" clobber, nothing
else.Nathan gave me the binary diff for the assembly generated for this function
without
the barrier and with the barrier:---
/tmp/lttng_event_reserve-4.8.2.dump2013-11-21 11:14:14.536495079 -0600
+++/tmp/lttng_event_reserve-with-barrier-4.8.2.dump2013-11-21
14:12:52.997355907 -0600tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5
-5@@ -7,11 +7,11 @@ f10: ebfffffe bl 0 <__gnu_mcount_nc> f14: e5903000 ldr r3, [r0] f18: e1a04000 mov r4, r0 - f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 + f1c: e1a0000d mov r0, sp f20: e5936030 ldr r6, [r3, #48] ; 0x30 - f24: e1a0000d mov r0, sp - f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 - f2c: e3c3303f bic r3, r3, #63 ; 0x3f + f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0 + f28: e3c3303f bic r3, r3, #63 ; 0x3f + f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8 f30: e5932004 ldr r2, [r3, #4] f34: e2822001 add r2, r2, #1 f38: e5832004 str r2, [r3, #4]@@ -53,7 +53,7 @@ fc8: e3c3303f bic r3, r3, #63 ; 0x3f fcc: e5933000 ldr r3, [r3] fd0: e3130002 tst r3, #2 - fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0> + fd4: 0a0002be beq 1ad4<
lttng_event_reserve+0xbd8>fd8: ebfffffe bl 0 <preempt_schedule> fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8> fe0: e3500000 cmp r0, #0We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the
right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.I'm attaching Nathan's ARM configuration.
It does look behave a bit like this bug pointed out by Luis:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of
kmem_cache_freetracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers
would trigger with this kind of stress-test.Thoughts ?
Thanks,
Quoted 7 lines
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com--
Quoted 3 lines
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [lttng-dev] current_thread_info() not respecting program order with gcc 4.8.x
Very interesting result:
Here is the asm diff between the problematic function compiled with gcc 4.8.2 vs that same function compiled with gcc 4.8.2 with the "lightly tested patch" in bug http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
lttng-ring-buffer-client-overwrite.ko-4.8.2-fix.objdump renamed from lttng-ring-buffer-client-overwrite.ko-4.8.2.objdump +4 -4
@@ -363,9 +363,9 @@ Disassembly of section .text: 504: e0647008 rsb r7, r4, r8 508: e0874004 add r4, r7, r4 50c: e0650004 rsb r0, r5, r4 - 510: e24bd028 sub sp, fp, #40 ; 0x28 + 510: e58a6000 str r6, [sl] 514: e6ef0070 uxtb r0, r0 - 518: e58a6000 str r6, [sl] + 518: e24bd028 sub sp, fp, #40 ; 0x28 51c: e89daff0 ldm sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc} ...
@@ -1938,8 +1938,8 @@ Disassembly of section .text: 1d74: ebfffffe bl 0 <warn_slowpath_null> 1d78: e51b205c ldr r2, [fp, #-92] ; 0x5c 1d7c: eafffef5 b 1958 <lttng_event_reserve+0xa5c> - 1d80: e24bd028 sub sp, fp, #40 ; 0x28 - 1d84: e51b0048 ldr r0, [fp, #-72] ; 0x48 + 1d80: e51b0048 ldr r0, [fp, #-72] ; 0x48 + 1d84: e24bd028 sub sp, fp, #40 ; 0x28 1d88: e89daff0 ldm sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc} ... 1d98: 0000017e .word 0x0000017e
So far, Nathan has not reproduced the issue with the fixed gcc. He's running those stress tests a couple more hours to get more confidence in the result.
Not sure about the first two stores (they use the stack limit pointer "sl", which I'm clueless about), but the last snippet clearly fixes a one instruction stack usage below sp race window. Before the fix:
- 1d80: e24bd028 sub sp, fp, #40 ; 0x28
- 1d84: e51b0048 ldr r0, [fp, #-72] ; 0x48
sp = fp - 40
load from memory location fp - 72 .... wrong !
The full objdumps (before and after gcc fix) are attached.
Thoughts ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com