Skip to contentSkip to conversation

might_sleep warnings in overwrite mode

DISCUSSION Nathan Lynch 12 participants 36 messages Nov 11, 2013 – Nov 22, 2013 4 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 6 lines · Nathan Lynch

At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.

Is it possible that the lttng module code is messing up the current task's preempt count somehow?

Here's another manifestation of the issue -- when flood-pinging the board during tracing, I get this warning from __do_softirq:

huh, entered softirq 3 NET_RX 804ae528 preempt_count 00000101, exited with 00000102?

#2 lore raw
In reply to a message not in the archive
Quoted 3 lines · Nathan Lynch

At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.

Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

----- Original Message -----

Quoted 10 lines

From: "Nathan Lynch" <[email protected]>
To: [email protected]
Sent: Thursday, November 14, 2013 1:16:53 PM
Subject: Re: [lttng-dev] might_sleep warnings in overwrite mode

Quoted 3 lines · Nathan Lynch

At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.

Quoted 6 lines

Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.

Allright! This info is really useful. Sorry for taking time to look into this, I was busy with preparation for 2.4-rc1.

Looking at:

fs/ext3/incode.c:

__ext3_get_inode_loc()

we can see that a use of the inlined sb_getblk() appears close to a tracepoint.

The tracepoint includes preempt disable/enable, exactly those:

include/linux/preempt.h:

#define preempt_disable_notrace() \

do { \
        inc_preempt_count_notrace(); \
        barrier(); \
} while (0)

#define preempt_enable_notrace() \

do { \
        preempt_enable_no_resched_notrace(); \
        barrier(); \
        preempt_check_resched_context(); \
} while (0)

Can you try wrapping the _outside_ of those macros with barrier(), e.g.

#define preempt_disable_notrace() \

do { \
        barrier(); \
        inc_preempt_count_notrace(); \
        barrier(); \
} while (0)

#define preempt_enable_notrace() \

do { \
        preempt_enable_no_resched_notrace(); \
        barrier(); \
        preempt_check_resched_context(); \
        barrier(); \
} while (0)

and try it out with the apparently buggy compiler to see if it helps ? It does look like the preempt inc or dec is slipping out and somehow triggers the might_sleep() warning. I don't see clearly how this could happen yet, since each of the inc/dec and the test are touching preempt_count(), but it's worth a try.

Maybe on ARM the current_thread_info() macro somehow hides important info from the compiler and it mistakenly reorders inc/dec vs the test. Another thing to try out (in addition to the first one) would be to try changing current_thread_info(), e.g., by turning asm ("sp") into a volatile inline assembly, and by adding "memory" clobbers to it.

Thoughts ?

Thanks,

Mathieu

Quoted 8 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 55 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 10 lines

From: "Nathan Lynch" <[email protected]>
To: [email protected]
Sent: Thursday, November 14, 2013 1:16:53 PM
Subject: Re: [lttng-dev] might_sleep warnings in overwrite mode

Quoted 3 lines · Nathan Lynch

At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.

Quoted 6 lines

Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.

Looking at:

fs/ext3/incode.c:

__ext3_get_inode_loc()

we can see that a use of the inlined sb_getblk() appears close to a tracepoint.

The tracepoint includes preempt disable/enable, exactly those:

include/linux/preempt.h:

#define preempt_disable_notrace() \

do { \
        inc_preempt_count_notrace(); \
        barrier(); \
} while (0)

#define preempt_enable_notrace() \

do { \
        preempt_enable_no_resched_notrace(); \
        barrier(); \
        preempt_check_resched_context(); \
} while (0)

Can you try wrapping the _outside_ of those macros with barrier(), e.g.

#define preempt_disable_notrace() \

do { \
        barrier(); \
        inc_preempt_count_notrace(); \
        barrier(); \
} while (0)

#define preempt_enable_notrace() \

Quoted 8 lines
do { \
        preempt_enable_no_resched_notrace(); \
        barrier(); \
        preempt_check_resched_context(); \
        barrier(); \
} while (0)

and try it out with the apparently buggy compiler to see if it helps ?

I did this, and I think I'm pretty satisfied that it works around the issue. I haven't been able to provoke might_sleep or preempt count warnings with multiple runs. Complicating matters is that I keep hitting a null pointer dereference in the mmc code which I wasn't seeing before (and sorry about the line wrapping, I cannot make Thunderbird behave well here):

Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = 80004000
[00000000] *pgd=00000000
Internal error: Oops: 17 [#1] PREEMPT SMP ARM
Modules linked in: lttng_probe_writeback(O) lttng_probe_workqueue(O)

lttng_probe_vmscan(O) lttng_probe_udp(O) lttng_probe_timer(O) lttng_probe_sunrpc(O) lttng_probe_statedump(O) lttng_probe_sock(O) lttng_probe_skb(O) lttng_probe_signal(O) lttng_probe_scsi(O) lttng_probe_sched(O) lttng_probe_rpm(O) lttng_probe_regulator(O) lttng_probe_regmap(O) lttng_probe_rcu(O) lttng_probe_random(O) lttng_probe_printk(O) lttng_probe_power(O) lttng_probe_net(O) lttng_probe_napi(O) lttng_probe_module(O) lttng_probe_kmem(O) lttng_probe_jbd2(O) lttng_probe_jbd(O) lttng_probe_irq(O) lttng_probe_gpio(O) lttng_probe_ext4(O) lttng_probe_ext3(O) lttng_probe_compaction(O) lttng_probe_block(O) lttng_probe_asoc(O) lttng_types(O) lttng_ring_buffer_metadata_mmap_client(O) lttng_ring_buffer_client_mmap_overwrite(O) lttng_ring_buffer_client_mmap_discard(O) lttng_ring_buffer_metadata_client(O) lttng_ring_buffer_client_overwrite(O) lttng_ring_buffer_client_discard(O) lttng_tracer(O) lttng_statedump(O) lttng_ftrace(O) lttng_kprobes(O) lttng_lib_ring_buffer(O) lttng_kretprobes(O)

CPU: 0 PID: 143 Comm: mmcqd/0 Tainted: G           O 3.11.6 #1
task: bf0d0900 ti: bf0be000 task.ti: bf0be000
PC is at sdhci_send_command+0x3fc/0xc58
LR is at sdhci_send_command+0x424/0xc58
pc : [<804371d0>]    lr : [<804371f8>]    psr: 80000093

sp : bf0bfd50 ip : bf0bfd50 fp : bf0bfdac
r10: 60000013 r9 : bfb7c440 r8 : 0000000f
r7 : 808da22c r6 : bb39c000 r5 : bb3ebe18 r4 : bb3ebe80 r3 : 87654321 r2 : bfa12c10 r1 : 81802b22 r0 : 00000000
Flags: Nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 4ecb404a DAC: 00000017
Process mmcqd/0 (pid: 143, stack limit = 0xbf0be238)

Stack: (0xbf0bfd50 to 0xbf0c0000)

fd40: 00000001 00000000 bf0bfd84
87654321
fd60: bfa12c10 7f0b700c bfb7c000 bb3ebda0 bfb7c440 00000001 bf0bfd94 bf0bfd88
fd80: 802d1af8 bfb7c000 bb3ebda0 bfb7c440 bfb7c518 40000013 00000001 bfb7c000
fda0: bf0bfdd4 bf0bfdb0 80438194 80436de0 bb3ebda0 bfb7c000 bb3ebd8c 00000000
fdc0: 00000001 00000001 bf0bfdec bf0bfdd8 804248d4 80437fd8 bb3ebec0 bf0bfe74
fde0: bf0bfe44 bf0bfdf0 804261b8 80424800 bf0be000 bf0be000 bf0be030 bb3ebc6c
fe00: bf0bfe44 00000000 bf0d0900 80047430 bf0bfe10 bf0bfe10 3aee2e82 bb368618
fe20: bb3ebc24 bb3c6000 bb317438 bb317438 bb3ebd9c bb3ebc00 bf0bfea4 bf0bfe48
fe40: 80433ba8 80425ef8 7f08d740 7f066a08 00000000 00000000 bf026000 00000000
fe60: bb3ebda0 00000000 bb3ebc00 00000000 00000038 cec07c38 cec07c00 bb317438
fe80: bb3c6000 bb3ebc24 bb368618 bfb7c000 bb3ebc00 bb3ebc00 bf0bfeec bf0bfea8
fea0: 804342a4 80433800 bb3ebd9c bb3ebc00 bb317438 bb3ebc2c bb368618 5e565e55
fec0: bb3ebc00 bb3ebc24 bb317438 bb3ebc2c bb368618 bf0be030 bf0be000 00000001
fee0: bf0bff24 bf0bfef0 80434cb4 804341a4 00000000 122c8001 80434bec bfa87bc4
ff00: 00000000 bb3ebc24 80434bec 00000000 00000000 00000000 bf0bffac bf0bff28
ff20: 80046520 80434bf8 8182be80 00000000 8182be80 bb3ebc24 00000000 00000000
ff40: dead4ead ffffffff ffffffff 80942bfc 00000000 00000000 807877a5 bf0bff5c
ff60: bf0bff5c 00000000 00000000 dead4ead ffffffff ffffffff 80942bfc 00000000
ff80: 00000000 807877a5 bf0bff88 bf0bff88 bfa87bc4 80046464 00000000 00000000
ffa0: 00000000 bf0bffb0 8000e438 80046470 00000000 00000000 00000000 00000000
ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000

Backtrace:
[<80436dd4>] (sdhci_send_command+0x0/0xc58) from [<80438194>]
(sdhci_request+0x1c8/0x1e0)
[<80437fcc>] (sdhci_request+0x0/0x1e0) from [<804248d4>]
(mmc_start_request+0xe0/0xe8)
 r9:00000001 r8:00000001 r7:00000000 r6:bb3ebd8c r5:bfb7c000

[<804247f4>] (mmc_start_request+0x0/0xe8) from [<804261b8>] (mmc_start_req+0x2cc/0x360)
r5:bf0bfe74 r4:bb3ebec0
[<80425eec>] (mmc_start_req+0x0/0x360) from [<80433ba8>] (mmc_blk_issue_rw_rq+0x3b4/0x9a4)
[<804337f4>] (mmc_blk_issue_rw_rq+0x0/0x9a4) from [<804342a4>] (mmc_blk_issue_rq+0x10c/0x450)
[<80434198>] (mmc_blk_issue_rq+0x0/0x450) from [<80434cb4>] (mmc_queue_thread+0xc8/0x164)
[<80434bec>] (mmc_queue_thread+0x0/0x164) from [<80046520>] (kthread+0xbc/0xc8)
[<80046464>] (kthread+0x0/0xc8) from [<8000e438>] (ret_from_fork+0x14/0x20)
r7:00000000 r6:00000000 r5:80046464 r4:bfa87bc4

Code: e1a00006 e3a08000 e158000a aa00000c (e5901000)

I've narrowed this down to sdhci_adma_table_pre() which is inlined, but haven't gotten much further. Anyway, I think it's probably unrelated -- surely it wouldn't be _caused_ by adding the barriers -- and I'll have to figure out some way to work around it (perhaps it's fixed in 3.12).

Quoted 11 lines

It does look like the preempt inc or dec is slipping out and somehow triggers the might_sleep() warning. I don't see clearly how this could happen yet, since each of the inc/dec and the test are touching preempt_count(), but it's worth a try.

Maybe on ARM the current_thread_info() macro somehow hides important info from the compiler and it mistakenly reorders inc/dec vs the test. Another thing to try out (in addition to the first one) would be to try changing current_thread_info(), e.g., by turning asm ("sp") into a volatile inline assembly, and by adding "memory" clobbers to it.

Haven't tried this yet.

#7 lore raw

current_thread_info() not respecting program order with gcc 4.8.x

In reply to a message not in the archive

Hi,

I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the current_thread_info() inline assembly. The only logical explanation for his issue I see so far is that read of the preempt_count within might_sleep() is reordered with preempt_enable() or preempt_disable(). AFAIU, this kind of issue also applies to other architectures.

First thing, preempt enable/disable only contains barrier() between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that the preempt_count() read in might_sleep() is not reordered with the preempt count inc/dec.

However, looking at ARM arch/arm/include/asm/thread_info.h:

static inline struct thread_info *current_thread_info(void)

{
        register unsigned long sp asm ("sp");
        return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
}

The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)

As long as someone does:

struct thread_info *myinfo = current_thread_info();

load from myinfo->preempt_count;
store to myinfo->preempt_count;

The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of might_sleep() followed by preempt_enable() is:

load from current_thread_info()->preempt_count;
store to current_thread_info()->preempt_count;

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).

Have any of you experienced this issue ? Any thoughts on the matter ?

I'm providing the original bug report email exchange below. I had confirmation that adding barrier() outside preempt_enable()/preempt_disable() does indeed work-around the issue, but I fear that this work-around would be leaving the current_thread_info() semantic gap in place: people expect program order to be respected.

Thanks,

Mathieu

----- Original Message -----

Quoted 110 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "Nathan Lynch" <[email protected]>
Cc: [email protected]
Sent: Thursday, November 14, 2013 9:34:09 PM
Subject: Re: [lttng-dev] might_sleep warnings in overwrite mode

----- Original Message -----

Quoted 10 lines

From: "Nathan Lynch" <[email protected]>
To: [email protected]
Sent: Thursday, November 14, 2013 1:16:53 PM
Subject: Re: [lttng-dev] might_sleep warnings in overwrite mode

Quoted 3 lines · Nathan Lynch

At this point I cannot trigger the issue without overwrite mode on 3.11.6, but on a 3.8-based vendor kernel I can recreate it regardless of the mode.

Quoted 6 lines

Some updates on this:
- It's not specific to overwrite mode; I was able to provoke it on 3.11.6 without --overwrite. Overwrite mode does seem to recreate the issue more readily.
- It seems to be GCC version-dependent. 4.8.1 and 4.8.2 produce code which warns/bugs; 4.7.3 does not.

Allright! This info is really useful. Sorry for taking time to look into this, I was busy with preparation for 2.4-rc1.

Looking at:

fs/ext3/incode.c:

__ext3_get_inode_loc()

we can see that a use of the inlined sb_getblk() appears close to a tracepoint.

The tracepoint includes preempt disable/enable, exactly those:

include/linux/preempt.h:

#define preempt_disable_notrace() \

do { \
        inc_preempt_count_notrace(); \
        barrier(); \
} while (0)

#define preempt_enable_notrace() \

do { \
        preempt_enable_no_resched_notrace(); \
        barrier(); \
        preempt_check_resched_context(); \
} while (0)

Can you try wrapping the _outside_ of those macros with barrier(), e.g.

#define preempt_disable_notrace() \

do { \
        barrier(); \
        inc_preempt_count_notrace(); \
        barrier(); \
} while (0)

#define preempt_enable_notrace() \

do { \
        preempt_enable_no_resched_notrace(); \
        barrier(); \
        preempt_check_resched_context(); \
        barrier(); \
} while (0)

and try it out with the apparently buggy compiler to see if it helps ? It does look like the preempt inc or dec is slipping out and somehow triggers the might_sleep() warning. I don't see clearly how this could happen yet, since each of the inc/dec and the test are touching preempt_count(), but it's worth a try.

Maybe on ARM the current_thread_info() macro somehow hides important info from the compiler and it mistakenly reorders inc/dec vs the test. Another thing to try out (in addition to the first one) would be to try changing current_thread_info(), e.g., by turning asm ("sp") into a volatile inline assembly, and by adding "memory" clobbers to it.

Thoughts ?

Thanks,

Mathieu

Quoted 8 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#8 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 47 lines · Mathieu Desnoyers

Hi,

I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the current_thread_info() inline assembly. The only logical explanation for his issue I see so far is that read of the preempt_count within might_sleep() is reordered with preempt_enable() or preempt_disable(). AFAIU, this kind of issue also applies to other architectures.

First thing, preempt enable/disable only contains barrier() between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that the preempt_count() read in might_sleep() is not reordered with the preempt count inc/dec.

However, looking at ARM arch/arm/include/asm/thread_info.h:

static inline struct thread_info *current_thread_info(void) { register unsigned long sp asm ("sp"); return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); }

The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)

As long as someone does:

struct thread_info *myinfo = current_thread_info();

load from myinfo->preempt_count; store to myinfo->preempt_count;

The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of might_sleep() followed by preempt_enable() is:

load from current_thread_info()->preempt_count; store to current_thread_info()->preempt_count;

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).

Quoted 8 lines

Have any of you experienced this issue ? Any thoughts on the matter ?

I'm providing the original bug report email exchange below. I had confirmation that adding barrier() outside preempt_enable()/preempt_disable() does indeed work-around the issue, but I fear that this work-around would be leaving the current_thread_info() semantic gap in place: people expect program order to be respected.

This all brings to mind commit 509eb76ebf977 ("ARM: 7747/1: pcpu: ensure __my_cpu_offset cannot be re-ordered across barrier()").

GCC seems overly happy with reordering asm() thingies.

Ideally we'd be able to tell gcc that:

register unsigned long *sp asm ("sp")

not only acts like a variable but should be treated like a variable and thus such reshuffeling should be limited to preserve program-order.

There doesn't seem to be any __attribute__() that can do this, and currently out only recourse seems to add either volatile, which seems overly strict as Mathieu said, or add fake input like Will did.

#9 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers

However, looking at ARM arch/arm/include/asm/thread_info.h:

static inline struct thread_info *current_thread_info(void) { register unsigned long sp asm ("sp"); return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); }

The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)

The above is not inline assembly, it is a local register variable extension, see
http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars

Quoted 3 lines
Quoted 3 lines

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

Sure.

Quoted 3 lines
Quoted 3 lines

One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).

As it is not inline asm extension, you can't.
Of course you could add asm volatile ("" : "=r" (ret) : "0" (sp)); or similar and thus make it a barrier, but I think the current definition of current_thread_info meant to avoid all that extra overhead. Why does it matter when sp is different between the current_thread_info () calls? As long as sp & ~(THREAD_SIZE - 1) is the same, it shouldn't make a difference. Or is the call in between those changing sp to something else?

Jakub

#10 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 26 lines · Jakub Jelinek
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers

However, looking at ARM arch/arm/include/asm/thread_info.h:

static inline struct thread_info *current_thread_info(void) { register unsigned long sp asm ("sp"); return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); }

The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)

The above is not inline assembly, it is a local register variable extension, see
http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars

Quoted 3 lines
Quoted 3 lines

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

Sure.

Quoted 3 lines
Quoted 3 lines

One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).

Quoted 8 lines

As it is not inline asm extension, you can't.
Of course you could add asm volatile ("" : "=r" (ret) : "0" (sp)); or similar and thus make it a barrier, but I think the current definition of current_thread_info meant to avoid all that extra overhead. Why does it matter when sp is different between the current_thread_info () calls? As long as sp & ~(THREAD_SIZE - 1) is the same, it shouldn't make a difference. Or is the call in between those changing sp to something else?

So what appears to be happening is that:

preempt_enable()
barrier();
current_thread_info()->preempt_count--;

    (1)

    might_sleep()
 	if (current_thread_info()->preempt_count)
		/* raise all bloody hell */

Gets re-ordered like:

	barrier();
	if (current_thread_info()->preempt_count)
		/* raise hell */
	current_thread_info()->preempt_count--;

And by inserting a barrier() at (1) things revert to normal and work again.

The "sp" reg value itself doesn't change here.

#11 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Mathieu Desnoyers
Quoted 1 line · Mathieu Desnoyers

Hi,

Hi Mathieu,

Quoted 3 lines

I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the current_thread_info() inline assembly. The only logical explanation for his issue I see so far is that read of the preempt_count within might_sleep() is reordered with preempt_enable() or preempt_disable(). AFAIU, this kind of issue also applies to other architectures.

First thing, preempt enable/disable only contains barrier() between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that the preempt_count() read in might_sleep() is not reordered with the preempt count inc/dec.

This sounds almost identical to an issue I experienced with our optimised per-cpu code (more below).

Quoted 17 lines

However, looking at ARM arch/arm/include/asm/thread_info.h:

static inline struct thread_info *current_thread_info(void)

{
        register unsigned long sp asm ("sp");
        return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
}

The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)

As long as someone does:

struct thread_info *myinfo = current_thread_info();

load from myinfo->preempt_count;
store to myinfo->preempt_count;

Quoted 8 lines

The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of might_sleep() followed by preempt_enable() is:

load from current_thread_info()->preempt_count;
store to current_thread_info()->preempt_count;

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).

Yup, that sucks, because you end up unable to cache the value when you know it hasn't changed.

Quoted 1 line

Have any of you experienced this issue ? Any thoughts on the matter ?

The way I got around this for the per-cpu code is to include a dummy memory constraint for the stack. This has a couple of advantages:

(1) It hazards against a "memory" clobber, so doesn't require use of `volatile'

(2) It doesn't require GCC to emit any address generation code, since dereferencing the sp is valid in ARM assembly

so adding something like:

	asm("" :: "Q" (*sp));

immediately after the declaration of sp in current_therad_info might do the trick. Do you have a way to test that?

Cheers,

Will

#12 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

----- Original Message -----

Quoted 9 lines

From: "Will Deacon" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: [email protected], "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Linus Torvalds" <[email protected]>, "Andrew Morton" <[email protected]>
Sent: Tuesday, November 19, 2013 11:05:02 AM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 4 lines
Quoted 1 line · Mathieu Desnoyers

Hi,

Hi Mathieu,

Hi Will,

Quoted 64 lines
Quoted 8 lines

I got a bug report on ARM which appears to be caused by an aggressive gcc optimisation starting from gcc 4.8.x due to lack of constraints on the current_thread_info() inline assembly. The only logical explanation for his issue I see so far is that read of the preempt_count within might_sleep() is reordered with preempt_enable() or preempt_disable(). AFAIU, this kind of issue also applies to other architectures.

Quoted 5 lines

First thing, preempt enable/disable only contains barrier() between the inc/dec and the inside of the critical section, not the outside. Therefore, we are relying on program order to ensure that the preempt_count() read in might_sleep() is not reordered with the preempt count inc/dec.

This sounds almost identical to an issue I experienced with our optimised per-cpu code (more below).

Quoted 26 lines

However, looking at ARM arch/arm/include/asm/thread_info.h:

static inline struct thread_info *current_thread_info(void)

{
        register unsigned long sp asm ("sp");
        return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
}

The inline assembly has no clobber and is not volatile. (this is also true for all other architectures I've looked at so far, which includes x86 and powerpc)

As long as someone does:

struct thread_info *myinfo = current_thread_info();

load from myinfo->preempt_count;
store to myinfo->preempt_count;

The program order should be preserved, because the read and the write are done wrt same base. However, what we have here in the case of might_sleep() followed by preempt_enable() is:

load from current_thread_info()->preempt_count;
store to current_thread_info()->preempt_count;

Quoted 6 lines

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

One possible solution to this might be to add "memory" clobber and volatile to this inline asm, but I fear it would put way too much constraints on the compiler optimizations (too heavyweight).

Yup, that sucks, because you end up unable to cache the value when you know it hasn't changed.

Quoted 1 line

Have any of you experienced this issue ? Any thoughts on the matter ?

The way I got around this for the per-cpu code is to include a dummy memory constraint for the stack. This has a couple of advantages:

(1) It hazards against a "memory" clobber, so doesn't require use of `volatile'

(2) It doesn't require GCC to emit any address generation code, since dereferencing the sp is valid in ARM assembly

Quoted 6 lines

so adding something like:

	asm("" :: "Q" (*sp));

immediately after the declaration of sp in current_therad_info might do the trick. Do you have a way to test that?

Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.

It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:

struct thread_info *ptra, *ptrb;

ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();

load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;

One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ? AFAIU, local reg vars appear to be architecture-specific, so maybe there is something fishy on ARM ?

Thanks,

Mathieu

Quoted 5 lines

Cheers,

Will

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#13 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 24 lines · Mathieu Desnoyers

Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.

It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:

struct thread_info *ptra, *ptrb;

ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();

load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;

Quoted 5 lines

One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ? AFAIU, local reg vars appear to be architecture-specific, so maybe there is something fishy on ARM ?

Might help if you ask where the GCC people are on CC ;-)

#14 lore raw

Multiple local register variables w/ same register

Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Mathieu Desnoyers

Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.

It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:

struct thread_info *ptra, *ptrb;

ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();

load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;

Quoted 5 lines
Quoted 5 lines

One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ? AFAIU, local reg vars appear to be architecture-specific, so maybe there is something fishy on ARM ?

It appears not:

int __attribute__((noinline)) f(void)
{

  {
    register int x __asm__("eax");
    x = 1;
  }
  {
    register int y __asm__("eax");
    return ++y;

}
}

extern void abort(void);

int main(void)
{
if (f() != 2)

    abort();

return 0;
}

Anyone see anything wrong with the testcase? Do we thing this sort of thing ought to work, perhaps with scopes lengthened?

r~

#15 lore raw

Re: Multiple local register variables w/ same register

Quoted 19 lines · Richard Henderson

It appears not:

int __attribute__((noinline)) f(void)
{

  {
    register int x __asm__("eax");
    x = 1;
  }
  {
    register int y __asm__("eax");
    return ++y;

}
}

extern void abort(void);

int main(void)
{
if (f() != 2)

Quoted 6 lines
    abort();

return 0;
}

Anyone see anything wrong with the testcase? Do we thing this sort of thing ought to work, perhaps with scopes lengthened?

I'd say this is undefined, when a local register var goes out of scope, it's value can change arbitrarily. If you insert some call in between the two scopes, it will surely have clobbered value, and even if there isn't any call in between those, any insn could in theory clobber those.

Jakub

#16 lore raw

Re: Multiple local register variables w/ same register

In reply to Richard Henderson
Quoted 53 lines · Richard Henderson
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Mathieu Desnoyers

Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test it for us though.

It might shuffle things around enough to work around the issue, but with the approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:

struct thread_info *ptra, *ptrb;

ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();

load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;

Quoted 6 lines
Quoted 6 lines

One question that might be worth asking: with the local register variable extension (http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables bound to the same register as being aliased or not ?
AFAIU, local reg vars appear to be architecture-specific, so maybe
there is something fishy on ARM ?

It appears not:

int __attribute__((noinline)) f(void)
{

  {
    register int x __asm__("eax");
    x = 1;
  }
  {
    register int y __asm__("eax");
    return ++y;

}
}

extern void abort(void);

int main(void)
{
if (f() != 2)

Quoted 6 lines
    abort();

return 0;
}

Anyone see anything wrong with the testcase? Do we thing this sort of thing ought to work, perhaps with scopes lengthened?

In general, I see no reason why the compiler should be forced to preserve a register between blocks like that. Relying on a value magically appearing in a general-purpose register seems to me fragile at best. Code like this adds inter-block dependencies that would not normally be there, and I'd rather not gamble on how the various optimisation passes deal with it, if it is even noticed at all.

What happens if you some code clobbering eax (e.g. a function call) between those two blocks? What do you think should happen?

Signature · 2 lines
Måns Rullgård
[email protected]
#17 lore raw

Re: Multiple local register variables w/ same register

In reply to Richard Henderson

----- Original Message -----

Quoted 62 lines

From: "Richard Henderson" <[email protected]>
To: "Peter Zijlstra" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Cc: "Will Deacon" <[email protected]>, [email protected], "Catalin Marinas" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Linus Torvalds" <[email protected]>, "Andrew Morton" <[email protected]>, "Jakub Jelinek" <[email protected]>, [email protected]
Sent: Tuesday, November 19, 2013 4:56:57 PM
Subject: Multiple local register variables w/ same register

Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Mathieu Desnoyers

Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could test
it for us though.

It might shuffle things around enough to work around the issue, but with the
approach you propose, I would be concerned about the compiler being within its rights to reorder the code into the following sequence:

struct thread_info *ptra, *ptrb;

ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();

load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;

Quoted 8 lines
Quoted 8 lines

One question that might be worth asking: with the local register variable extension
(http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables
bound to the same register as being aliased or not ? AFAIU, local reg vars appear
to be architecture-specific, so maybe there is something fishy on ARM ?

It appears not:

int __attribute__((noinline)) f(void)
{

  {
    register int x __asm__("eax");
    x = 1;
  }
  {
    register int y __asm__("eax");
    return ++y;

}
}

extern void abort(void);

Quoted 8 lines

int main(void)
{
if (f() != 2)

    abort();

return 0;
}

Anyone see anything wrong with the testcase?

This testcase is targeting a general purpose register, whereas the issue I'm presenting gets the stack pointer as base address for many memory operations targeting the same offset from this base address. So strictly speaking, I think the two cases are slightly different.

Thanks,

Mathieu

Quoted 6 lines

Do we thing this sort of thing
ought to work, perhaps with scopes lengthened?

r~

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#18 lore raw

Re: [lttng-dev] Multiple local register variables w/ same register

----- Original Message -----

Quoted 88 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "Richard Henderson" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, [email protected], "Peter Zijlstra" <[email protected]>, "Catalin Marinas" <[email protected]>, "Nathan Lynch" <[email protected]>, [email protected], "Will Deacon" <[email protected]>, [email protected], "Andrew Morton" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Linus Torvalds" <[email protected]>
Sent: Tuesday, November 19, 2013 5:25:18 PM
Subject: Re: [lttng-dev] Multiple local register variables w/ same register

----- Original Message -----

Quoted 78 lines

From: "Richard Henderson" <[email protected]>
To: "Peter Zijlstra" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Cc: "Will Deacon" <[email protected]>, [email protected], "Catalin Marinas" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney"
<[email protected]>, "Linus Torvalds" <[email protected]>, "Andrew Morton" <[email protected]>, "Jakub Jelinek" <[email protected]>, [email protected]
Sent: Tuesday, November 19, 2013 4:56:57 PM
Subject: Multiple local register variables w/ same register

Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Mathieu Desnoyers

Unfortunately I don't have a ARM cross-compiler setup ready. Nathan could
test
it for us though.

It might shuffle things around enough to work around the issue, but with the
approach you propose, I would be concerned about the compiler being within
its rights to reorder the code into the following sequence:

struct thread_info *ptra, *ptrb;

ptra = current_thread_info();
/*
* each current_thread_info() would have a clobber on *sp, which orders
* those two wrt each other.
*/
ptrb = current_thread_info();

load from ptra->preempt_count;
/*
* however, the following accesses that depend on ptra and ptrb could be
* reordered if the compiler has no way to know that ptra and ptrb are
* aliased.
*/
store to ptrb->preempt_count;

Quoted 10 lines
Quoted 10 lines

One question that might be worth asking: with the local register variable
extension
(http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/Local-Reg-Vars.html#Local-Reg-Vars)
(thanks to Jakub for the pointer), should the compiler consider two variables
bound to the same register as being aliased or not ? AFAIU, local reg vars
appear
to be architecture-specific, so maybe there is something fishy on ARM ?

It appears not:

int __attribute__((noinline)) f(void) {

  {
    register int x __asm__("eax");
    x = 1;
  }
  {
    register int y __asm__("eax");
    return ++y;

}
}

extern void abort(void);

int main(void)
{
if (f() != 2)

    abort();

return 0;
}

Quoted 6 lines
Quoted 1 line

Anyone see anything wrong with the testcase?

This testcase is targeting a general purpose register, whereas the issue I'm presenting gets the stack pointer as base address for many memory operations targeting the same offset from this base address. So strictly speaking, I think the two cases are slightly different.

By the way, I just had more info from Nathan, where a splat clearly shows a scheduling while atomic, which points rather to an unbalanced preempt enable/disable within LTTng. It's weird though, because so far it only triggers on ARM, and only with gcc 4.8.x. Don't waste your time on this one until we can figure out if the issue is within LTTng.

Thanks,

Mathieu

Quoted 23 lines

Thanks,

Mathieu

Quoted 6 lines

Do we thing this sort of thing
ought to work, perhaps with scopes lengthened?

r~

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#19 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Mathieu Desnoyers
Quoted 2 lines · Mathieu Desnoyers

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

I don't understand why you say that.

The ordering of the access to the asm("sp") register is totally irrelevant. You are "correct" in saying that the compiler is within its right to re-order them, but that is the worst kind of correct:
it's totally immaterial. In fact, we *want* the compiler to not just re-order the accesses to %sp, but to notice that it can combine them, and do CSE on that whole expression when it is used multiple times within the same function (like it often is used).

So the compiler can very much decide to re-read %sp all it wants, and re-order those reads all it wants, and that's not the bug at all. Putting a clobber or a volatile on it would disable the optimization we *want* to happen.

So don't bark up the wrong tree.

The bug seems to be that gcc re-orders the *memory* *accesses* through that point, which is not correct in any way, shape, or form. If we have a write to a memory location followed by a read of the same memory location, the compiler ABSOLUTELY MUST NOT RE-ORDER THEM. The write obviously changes the value of the read.

It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.

There may be some insane "two different objects on the stack cannot alias" logic, which is true for *objects* on the stack, but it sure as hell isn't true for random accesses through asm("sp").

If I read this thread correctly, you're all talking about something else than the actual bug, and are trying to say that there is something wrong with re-ordering the access to %sp itself. Missing the _real_ bug entirely. See above.

Linus

#20 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

----- Original Message -----

Quoted 46 lines

From: "Linus Torvalds" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, "Jakub Jelinek" <[email protected]>, "Richard Henderson" <[email protected]>
Cc: "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>
Sent: Tuesday, November 19, 2013 7:41:17 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 3 lines · Mathieu Desnoyers

Since each current_thread_info() is a different asm ("sp") without clobber nor volatile, AFAIU, the compiler is within its right to reorder them.

I don't understand why you say that.

The ordering of the access to the asm("sp") register is totally irrelevant. You are "correct" in saying that the compiler is within its right to re-order them, but that is the worst kind of correct:
it's totally immaterial. In fact, we *want* the compiler to not just re-order the accesses to %sp, but to notice that it can combine them, and do CSE on that whole expression when it is used multiple times within the same function (like it often is used).

So the compiler can very much decide to re-read %sp all it wants, and re-order those reads all it wants, and that's not the bug at all. Putting a clobber or a volatile on it would disable the optimization we *want* to happen.

So don't bark up the wrong tree.

The bug seems to be that gcc re-orders the *memory* *accesses* through that point, which is not correct in any way, shape, or form. If we have a write to a memory location followed by a read of the same memory location, the compiler ABSOLUTELY MUST NOT RE-ORDER THEM. The write obviously changes the value of the read.

It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.

Quoted 8 lines

There may be some insane "two different objects on the stack cannot alias" logic, which is true for *objects* on the stack, but it sure as hell isn't true for random accesses through asm("sp").

If I read this thread correctly, you're all talking about something else than the actual bug, and are trying to say that there is something wrong with re-ordering the access to %sp itself. Missing the _real_ bug entirely. See above.

Yes, exactly, your explanation clarifies the underlying issue I'm trying to point at.

Thank you !

Mathieu

Quoted 3 lines

Linus

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#21 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Linus Torvalds

Am 20.11.2013 01:41, schrieb Linus Torvalds:

Quoted 6 lines

It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.

Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that current_thread_info() has the prototype

static inline struct thread_info *current_thread_info(void) __attribute_const__;

on arm (and arm64 and unicore32, something the paste from Mathieu missed so most people here might have missed that detail too). It's a very good finding from Luis.

I'm writing this message because his mail doesn't have an in-reply-to header, so it might be missed in this thread.

As Luis said, declaring current_thread_info() as a const function is wrong. The gcc manual says:

----
const

Many functions do not examine any values except their arguments, and have no effects except the return value. Basically this is just slightly more strict class than the pure attribute below, since function is not allowed to read global memory.

Note that a function that has pointer arguments and examines the data pointed to must not be declared const. Likewise, a function that calls a non-const function usually must not be const. It does not make sense for a const function to return void.
----

So current_thread_info() clearly violates the constrain to not read global memory. Or in other words, that __attribute_const__ tells gcc explicitly that the two reads are pointing to different locations because they are assumed to be local (through the const).

So This might be the reason why gcc misses that different calls to current_thread_info() might point to the same memory location.

As I've an arm gcc 4.8.1 ready too, I'm joining Luis question where the reordering can be found. If someone would point me to the source/object where this happens, I could have a look if removing the __attribute_const__ makes a difference.

Regards,

Alexander Holler

#22 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Thanks for putting my e-mail in the right thread Alexander.

I also wanted to point out that we recently were hit by the following bug:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

which can mess up the handling of SP and cause strange failures in the kernel.

From the description of the bug in this thread, we don't think the bugs are related but whoever is going to try removing the "const attribute" please also make sure that the bug still reproduces after applying the fix for the GCC bug above.

Luis

Quoted 58 lines · Alexander Holler

Am 20.11.2013 01:41, schrieb Linus Torvalds:

Quoted 6 lines

It seems that some gcc alias analysis completely incorrectly thinks that they are not the same memory location, and do not alias. My guess would be that gcc sees that that they are based on the stack pointer with "different" offsets, and decides that the memory locations must be different - without noticing that the "& ~(THREAD_SIZE - 1)" will end up generating the same address for both of them.

Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that current_thread_info() has the prototype

static inline struct thread_info *current_thread_info(void) __attribute_const__;

on arm (and arm64 and unicore32, something the paste from Mathieu missed so most people here might have missed that detail too). It's a very good finding from Luis.

I'm writing this message because his mail doesn't have an in-reply-to header, so it might be missed in this thread.

As Luis said, declaring current_thread_info() as a const function is wrong. The gcc manual says:

----
const

Many functions do not examine any values except their arguments, and have no effects except the return value. Basically this is just slightly more strict class than the pure attribute below, since function is not allowed to read global memory.

Note that a function that has pointer arguments and examines the data pointed to must not be declared const. Likewise, a function that calls a non-const function usually must not be const. It does not make sense for a const function to return void.
----

So current_thread_info() clearly violates the constrain to not read global memory. Or in other words, that __attribute_const__ tells gcc explicitly that the two reads are pointing to different locations because they are assumed to be local (through the const).

So This might be the reason why gcc misses that different calls to current_thread_info() might point to the same memory location.

As I've an arm gcc 4.8.1 ready too, I'm joining Luis question where the reordering can be found. If someone would point me to the source/object where this happens, I could have a look if removing the __attribute_const__ makes a difference.

Regards,

Alexander Holler

#23 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Alexander Holler
Quoted 10 lines · Alexander Holler

Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that current_thread_info() has the prototype

static inline struct thread_info *current_thread_info(void) __attribute_const__;

on arm (and arm64 and unicore32, something the paste from Mathieu missed so most people here might have missed that detail too). It's a very good finding from Luis.

No, because it is immaterial.

We *want* gcc to optimize away multiple accesses to "sp". Because it doesn't *matter* whether "sp" changes or not, the *result* is always the same. That's what the "const" means.

The "& ~(THREAD_SIZE - 1)" part will remove all the bits that can change. Really. So the result *is* constant (within one thread). Marking it constant and telling gcc that it can combine these things is correct.

Guys, read my email again.

The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.

The bug is *outside* that "current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.

Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.

Linus

#24 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 14 lines · Linus Torvalds
Quoted 8 lines · Alexander Holler

Luis Lozano just noted (see https://lkml.org/lkml/2013/11/20/625) that current_thread_info() has the prototype

static inline struct thread_info *current_thread_info(void) __attribute_const__;

on arm (and arm64 and unicore32, something the paste from Mathieu missed

so

Quoted 2 lines

most people here might have missed that detail too). It's a very good finding from Luis.

Can someone provide a test case? We are unable to reproduce the problem in ChromeOS land.

Quoted 12 lines

No, because it is immaterial.

We *want* gcc to optimize away multiple accesses to "sp". Because it doesn't *matter* whether "sp" changes or not, the *result* is always the same. That's what the "const" means.

The "& ~(THREAD_SIZE - 1)" part will remove all the bits that can change. Really. So the result *is* constant (within one thread). Marking it constant and telling gcc that it can combine these things is correct.

Its unfortunate that the attribute '__const__ ' is very misleading - the semantic used by gcc is that this is an aliasing attribute that implies that the function does not depend in any way on global state. Meaning, if called with the same parameters, it expected to return the same value. It should have been called 'pure', but gcc has a different 'pure' attribute that imples something different :-(

That said, gcc should be smart enough to figure out that this function is
not doing anything complicated. Again, a test case would be much
appreciated.

Bhaskar

Quoted 5 lines

Guys, read my email again.

The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.

Quoted 10 lines

The bug is *outside* that "current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.

Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.

Linus

Quoted 1 line

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#25 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Linus Torvalds

Am 21.11.2013 23:32, schrieb Linus Torvalds:

Quoted 4 lines · Alexander Holler

The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.

Sure, and I don't disagree on that.

Quoted 9 lines

The bug is *outside* that "current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.

Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.

Sorry, that I still disagree.

I try to describe it more clearly why I still think that the problem might be because of that const declaration.

(...)

foobar1 = current_thread_info() __attribute_const__ {
	return sp->somewhere_local;
}

(...)

foobar2 = current_thread_info() __attribute_const__ {
	return sp->somewhere_local;
}

So, even if sp is the same in both cases, that const states that wherever sp points to is local to current_thread_info(), so it can't be the same for both cases.

Regards,

Alexander Holler

#26 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Luis

Quoted 49 lines · Alexander Holler

Am 21.11.2013 23:32, schrieb Linus Torvalds:

Quoted 6 lines

On Thu, Nov 21, 2013 at 8:02 AM, Alexander Holler <[email protected]> wrote:

The bug is not that gcc can re-order or combine the accesses to "sp". WE WANT THAT TO HAPPEN.

Sure, and I don't disagree on that.

Quoted 9 lines

The bug is *outside* that "current_thread_info()" macro/inline function. It's the *dereference* of the pointer that gcc re-orders. AND THAT IS WRONG.

Gcc seems to mess up the alias analysis, and decide that the deferences cannot alias. Which is wrong. They clearly *can* alias, exactly because the value of "sp & ~(THREAD_SIZE - 1)" ends up having the same value all the time.

Sorry, that I still disagree.

I try to describe it more clearly why I still think that the problem might be because of that const declaration.

(...)

foobar1 = current_thread_info() __attribute_const__ {
        return sp->somewhere_local;
}

(...)

foobar2 = current_thread_info() __attribute_const__ {
        return sp->somewhere_local;
}

So, even if sp is the same in both cases, that const states that wherever sp points to is local to current_thread_info(), so it can't be the same for both cases.

Regards,

Alexander Holler

Signature · 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
#27 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 5 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.

Jakub

#28 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

----- Original Message -----

Quoted 18 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 5 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Quoted 7 lines

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature: -fno-sched-pressure (sched pressure is there by default). Nathan tried compiling lttng-modules with -fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown below works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)
{
        struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here; <----------------------------
        return ret;
}

Where barrier() is the usual asm volatile with "memory" clobber, nothing else.

Nathan gave me the binary diff for the assembly generated for this function without the barrier and with the barrier:

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4 <lttng_event_reserve+0xbd8>
      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4 <lttng_event_reserve+0xbd8>
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way, and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#29 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening. In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?

Thanks

Luis

Quoted 154 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 18 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 5 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Quoted 7 lines

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature: -fno-sched-pressure (sched pressure is there by default). Nathan tried compiling lttng-modules with -fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown below works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)
{
        struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here; <----------------------------
        return ret;
}

Where barrier() is the usual asm volatile with "memory" clobber, nothing else.

Nathan gave me the binary diff for the assembly generated for this function without the barrier and with the barrier:

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4 <lttng_event_reserve+0xbd8>
      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4 <lttng_event_reserve+0xbd8>
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way, and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
#30 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

----- Original Message -----

Quoted 11 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 8 lines

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.

The following instruction

f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3, which makes me wonder if it's good or not. With slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:

00000efc <lttng_event_reserve>:

     efc:	e1a0c00d 	mov	ip, sp
     f00:	e92ddff0 	push	{r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
     f04:	e24cb004 	sub	fp, ip, #4
     f08:	e24dd03c 	sub	sp, sp, #60	; 0x3c
     f0c:	e52de004 	push	{lr}		; (str lr, [sp, #-4]!)
     f10:	ebfffffe 	bl	0 <__gnu_mcount_nc>
     f14:	e5903000 	ldr	r3, [r0]
     f18:	e1a04000 	mov	r4, r0
     f1c:	e1a0000d 	mov	r0, sp
     f20:	e5936030 	ldr	r6, [r3, #48]	; 0x30
     f24:	e3c03d7f 	bic	r3, r0, #8128	; 0x1fc0
     f28:	e3c3303f 	bic	r3, r3, #63	; 0x3f
     f2c:	e50b104c 	str	r1, [fp, #-76]	; 0xffffffb4
     f30:	e5932004 	ldr	r2, [r3, #4]
     f34:	e2822001 	add	r2, r2, #1
     f38:	e5832004 	str	r2, [r3, #4]
     f3c:	ebfffffe 	bl	0 <debug_smp_processor_id>
     f40:	e59f2e44 	ldr	r2, [pc, #3652]	; 1d8c <lttng_event_reserve+0xe90>
     f44:	e59f3e44 	ldr	r3, [pc, #3652]	; 1d90 <lttng_event_reserve+0xe94>
     f48:	e7921100 	ldr	r1, [r2, r0, lsl #2]
     f4c:	e1a05000 	mov	r5, r0
     f50:	e7932001 	ldr	r2, [r3, r1]

My rusty ARM assembler knowledge analyzes this like:

ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76

So yes, it does look like the -76 from fp is within the stack.

Quoted 4 lines

In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?

My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning, did not manage to get one working even after following 2 howtos, trying with Debian packages, etc. Nathan told me this diff was the full comparison between the two functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.

But after a long day of debugging, it's time for some sleep. I will get back to this tomorrow,

Thanks,

Mathieu

Quoted 189 lines

Thanks

Luis

Quoted 165 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 24 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 5 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Quoted 7 lines

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules with
-fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown below
works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)
{
        struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here;
        <----------------------------
        return ret;
}

Where barrier() is the usual asm volatile with "memory" clobber, nothing else.

Nathan gave me the binary diff for the assembly generated for this function without
the barrier and with the barrier:

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0 -0

14:12:52.997355907 -0600

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4 <lttng_event_reserve+0xbd8>
      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4 <lttng_event_reserve+0xbd8>
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Quoted 7 lines

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--

Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#31 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 27 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 3 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <

[email protected]>, "Linus Torvalds"

Quoted 1 line

<[email protected]>, "Richard Henderson" <[email protected]>,

"Linux Kernel Mailing List"

Quoted 1 line

<[email protected]>, "Will Deacon" <[email protected]>,

"Catalin Marinas" <[email protected]>,

Quoted 1 line

"Peter Zijlstra" <[email protected]>, [email protected],

"Nathan Lynch" <[email protected]>, "Paul

Quoted 1 line

E. McKenney" <[email protected]>, "Andrew Morton" <

[email protected]>, "Bhaskar Janakiraman"

Quoted 3 lines

<[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re: current_thread_info() not respecting program order with gcc

4.8.x

Quoted 9 lines

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.

Quoted 6 lines

The following instruction

f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3, which makes me wonder if it's good or not.

It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just instructions ordered a little different?

Luis

Quoted 280 lines

With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:

00000efc <lttng_event_reserve>:

     efc:       e1a0c00d        mov     ip, sp
     f00:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl, fp,
ip, lr, pc}
     f04:       e24cb004        sub     fp, ip, #4
     f08:       e24dd03c        sub     sp, sp, #60     ; 0x3c
     f0c:       e52de004        push    {lr}            ; (str lr, [sp,
#-4]!)
     f10:       ebfffffe        bl      0 <__gnu_mcount_nc>
     f14:       e5903000        ldr     r3, [r0]
     f18:       e1a04000        mov     r4, r0
     f1c:       e1a0000d        mov     r0, sp
     f20:       e5936030        ldr     r6, [r3, #48]   ; 0x30
     f24:       e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
     f28:       e3c3303f        bic     r3, r3, #63     ; 0x3f
     f2c:       e50b104c        str     r1, [fp, #-76]  ; 0xffffffb4
     f30:       e5932004        ldr     r2, [r3, #4]
     f34:       e2822001        add     r2, r2, #1
     f38:       e5832004        str     r2, [r3, #4]
     f3c:       ebfffffe        bl      0 <debug_smp_processor_id>
     f40:       e59f2e44        ldr     r2, [pc, #3652] ; 1d8c
<lttng_event_reserve+0xe90>
     f44:       e59f3e44        ldr     r3, [pc, #3652] ; 1d90
<lttng_event_reserve+0xe94>
     f48:       e7921100        ldr     r1, [r2, r0, lsl #2]
     f4c:       e1a05000        mov     r5, r0
     f50:       e7932001        ldr     r2, [r3, r1]

My rusty ARM assembler knowledge analyzes this like:

ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76

So yes, it does look like the -76 from fp is within the stack.

Quoted 4 lines

In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?

My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning,
did not manage to get one working even after following 2 howtos, trying with Debian
packages, etc. Nathan told me this diff was the full comparison between the two
functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.

But after a long day of debugging, it's time for some sleep. I will get back to this
tomorrow,

Thanks,

Mathieu

Quoted 16 lines

Thanks

Luis

Quoted 6 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 5 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <

[email protected]>,

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

"Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with

gcc

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

4.8.x

Quoted 1 line · Luis Lozano

I think we need a reproducer. Without this we may all be going on

the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed

source,

Quoted 1 line
Quoted 1 line
Quoted 1 line

compiler version, flags and how it was configured and some hint in

which

Quoted 1 line
Quoted 1 line
Quoted 1 line

function to look for what exactly? We don't necessarily need a

runtime

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned

input

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full

day

Quoted 1 line
Quoted 1 line

of bug hunting. We're not there yet, but many options have been

eliminated.

Quoted 2 lines
Quoted 2 lines

The issue shows up in stress test, when tracing with lttng-modules

2.4-rc1,

Quoted 5 lines
Quoted 5 lines

on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and

lttng-modules

Quoted 3 lines
Quoted 3 lines

with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the

rest

Quoted 5 lines
Quoted 5 lines

of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling

lttng-modules

Quoted 5 lines
Quoted 5 lines

with
-fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was

responsible

Quoted 1 line
Quoted 1 line

for working around the issue. Skipping a long investigation, here is

the

Quoted 3 lines
Quoted 3 lines

executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Quoted 2 lines
Quoted 2 lines

Has the following function. We identified that adding a barrier() as

shown

Quoted 8 lines
Quoted 8 lines

below
works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)

{
struct lttng_channel *lttng_chan =

channel_get_private(ctx->chan);

Quoted 32 lines
Quoted 32 lines
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here;
        <----------------------------
        return ret;
}
Quoted 1 line
Quoted 1 line

Where barrier() is the usual asm volatile with "memory" clobber,

nothing

Quoted 3 lines
Quoted 3 lines

else.

Nathan gave me the binary diff for the assembly generated for this

function

Quoted 4 lines
Quoted 4 lines

without
the barrier and with the barrier:

--- /tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079

-0600

Quoted 16 lines
Quoted 16 lines

+++ /tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21
14:12:52.997355907 -0600

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +4 -4
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
Quoted 8 lines

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 line 56

Quoted 8 lines
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +1 -1
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4

<lttng_event_reserve+0xbd8>

Quoted 2 lines

Review line 58

Quoted 2 lines
(quoted diff) +0 -0
      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4

<lttng_event_reserve+0xbd8>

Quoted 8 lines
Quoted 8 lines
tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0 -0
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for

the

Quoted 8 lines
Quoted 8 lines

right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

Quoted 8 lines
Quoted 8 lines

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt

handlers

Quoted 18 lines
Quoted 6 lines

would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Quoted 7 lines

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--

Quoted 9 lines
Quoted 1 line

Luis A. Lozano | Software Engineer | [email protected] | +1

(408)431-5164

Quoted 1 line

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#32 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Mathieu Desnoyers
Quoted 22 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 11 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 8 lines

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.

Quoted 6 lines

The following instruction

f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3,

It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just the instructions ordered a little different?

Luis

Quoted 255 lines

which makes me wonder if it's good or not. With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:

00000efc <lttng_event_reserve>:

     efc:       e1a0c00d        mov     ip, sp
     f00:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
     f04:       e24cb004        sub     fp, ip, #4
     f08:       e24dd03c        sub     sp, sp, #60     ; 0x3c
     f0c:       e52de004        push    {lr}            ; (str lr, [sp, #-4]!)
     f10:       ebfffffe        bl      0 <__gnu_mcount_nc>
     f14:       e5903000        ldr     r3, [r0]
     f18:       e1a04000        mov     r4, r0
     f1c:       e1a0000d        mov     r0, sp
     f20:       e5936030        ldr     r6, [r3, #48]   ; 0x30
     f24:       e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
     f28:       e3c3303f        bic     r3, r3, #63     ; 0x3f
     f2c:       e50b104c        str     r1, [fp, #-76]  ; 0xffffffb4
     f30:       e5932004        ldr     r2, [r3, #4]
     f34:       e2822001        add     r2, r2, #1
     f38:       e5832004        str     r2, [r3, #4]
     f3c:       ebfffffe        bl      0 <debug_smp_processor_id>
     f40:       e59f2e44        ldr     r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90>
     f44:       e59f3e44        ldr     r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94>
     f48:       e7921100        ldr     r1, [r2, r0, lsl #2]
     f4c:       e1a05000        mov     r5, r0
     f50:       e7932001        ldr     r2, [r3, r1]

My rusty ARM assembler knowledge analyzes this like:

ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76

So yes, it does look like the -76 from fp is within the stack.

Quoted 4 lines

In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?

My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning, did not manage to get one working even after following 2 howtos, trying with Debian packages, etc. Nathan told me this diff was the full comparison between the two functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.

But after a long day of debugging, it's time for some sleep. I will get back to this tomorrow,

Thanks,

Mathieu

Quoted 187 lines

Thanks

Luis

Quoted 165 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 24 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 5 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Quoted 7 lines

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules with
-fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown below
works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)
{
        struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here;
        <----------------------------
        return ret;
}

Where barrier() is the usual asm volatile with "memory" clobber, nothing else.

Nathan gave me the binary diff for the assembly generated for this function without
the barrier and with the barrier:

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0 -0

14:12:52.997355907 -0600

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4 <lttng_event_reserve+0xbd8>
      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4 <lttng_event_reserve+0xbd8>
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Quoted 7 lines

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--

Quoted 2 lines

Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

#33 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 28 lines · Luis Lozano
Quoted 26 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 11 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 8 lines

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.

The following instruction

f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

Quoted 9 lines
Quoted 2 lines

appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3,

It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just the instructions ordered a little different?

unless there is a dependency between the following 2 instructions?
- f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8
...

      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30

and moving the str across the ldr makes a difference...

Quoted 257 lines

Luis

Quoted 255 lines

which makes me wonder if it's good or not. With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:

00000efc <lttng_event_reserve>:

     efc:       e1a0c00d        mov     ip, sp
     f00:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
     f04:       e24cb004        sub     fp, ip, #4
     f08:       e24dd03c        sub     sp, sp, #60     ; 0x3c
     f0c:       e52de004        push    {lr}            ; (str lr, [sp, #-4]!)
     f10:       ebfffffe        bl      0 <__gnu_mcount_nc>
     f14:       e5903000        ldr     r3, [r0]
     f18:       e1a04000        mov     r4, r0
     f1c:       e1a0000d        mov     r0, sp
     f20:       e5936030        ldr     r6, [r3, #48]   ; 0x30
     f24:       e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
     f28:       e3c3303f        bic     r3, r3, #63     ; 0x3f
     f2c:       e50b104c        str     r1, [fp, #-76]  ; 0xffffffb4
     f30:       e5932004        ldr     r2, [r3, #4]
     f34:       e2822001        add     r2, r2, #1
     f38:       e5832004        str     r2, [r3, #4]
     f3c:       ebfffffe        bl      0 <debug_smp_processor_id>
     f40:       e59f2e44        ldr     r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90>
     f44:       e59f3e44        ldr     r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94>
     f48:       e7921100        ldr     r1, [r2, r0, lsl #2]
     f4c:       e1a05000        mov     r5, r0
     f50:       e7932001        ldr     r2, [r3, r1]

My rusty ARM assembler knowledge analyzes this like:

ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76

So yes, it does look like the -76 from fp is within the stack.

Quoted 4 lines

In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?

My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning, did not manage to get one working even after following 2 howtos, trying with Debian packages, etc. Nathan told me this diff was the full comparison between the two functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.

But after a long day of debugging, it's time for some sleep. I will get back to this tomorrow,

Thanks,

Mathieu

Quoted 187 lines

Thanks

Luis

Quoted 165 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 24 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 5 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Quoted 7 lines

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source, compiler version, flags and how it was configured and some hint in which function to look for what exactly? We don't necessarily need a runtime small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1, on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules with
-fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible for working around the issue. Skipping a long investigation, here is the executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown below
works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)
{
        struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here;
        <----------------------------
        return ret;
}

Where barrier() is the usual asm volatile with "memory" clobber, nothing else.

Nathan gave me the binary diff for the assembly generated for this function without
the barrier and with the barrier:

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +0 -0

14:12:52.997355907 -0600

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4 <lttng_event_reserve+0xbd8>
      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4 <lttng_event_reserve+0xbd8>
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Quoted 7 lines

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--

Quoted 2 lines

Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 1 line
Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164
#34 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Luis Lozano

----- Original Message -----

Quoted 50 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Friday, November 22, 2013 3:18:14 AM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 28 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 17 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List"
<[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 9:36:27 PM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 8 lines

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt handler was corrupting the stack of a routine which had an invalid value of SP (not really on "top" of the stack) for part of the routine.
But in the assembly code you sent, I don't see where sp is being modified... or where the access to "below" sp is happening.

Quoted 6 lines

The following instruction

f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

appears in the assembly generated by gcc 4.8.2, but not in the one generated by 4.7.3,

Quoted 5 lines

It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just the instructions ordered a little different?

Luis

(resending reply in plain text)

Assembly generated by 4.7.3 vs 4.8.2 are much more different.

The diff I've presented yesterday is between 4.8.2 and 4.8.2+barrier() at the end of the function error path. When I can get a cross compiler to work, I will send out more the complete assembly listings of the function.

Thanks,

Mathieu

Quoted 290 lines
Quoted 289 lines

which makes me wonder if it's good or not. With
slightly different build output from the diff (probably a different config from Nathan), the first function instructions look like:

00000efc <lttng_event_reserve>:

     efc:       e1a0c00d        mov     ip, sp
     f00:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl, fp,
     ip, lr, pc}
     f04:       e24cb004        sub     fp, ip, #4
     f08:       e24dd03c        sub     sp, sp, #60     ; 0x3c
     f0c:       e52de004        push    {lr}            ; (str lr, [sp,
     #-4]!)
     f10:       ebfffffe        bl      0 <__gnu_mcount_nc>
     f14:       e5903000        ldr     r3, [r0]
     f18:       e1a04000        mov     r4, r0
     f1c:       e1a0000d        mov     r0, sp
     f20:       e5936030        ldr     r6, [r3, #48]   ; 0x30
     f24:       e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
     f28:       e3c3303f        bic     r3, r3, #63     ; 0x3f
     f2c:       e50b104c        str     r1, [fp, #-76]  ; 0xffffffb4
     f30:       e5932004        ldr     r2, [r3, #4]
     f34:       e2822001        add     r2, r2, #1
     f38:       e5832004        str     r2, [r3, #4]
     f3c:       ebfffffe        bl      0 <debug_smp_processor_id>
     f40:       e59f2e44        ldr     r2, [pc, #3652] ; 1d8c
     <lttng_event_reserve+0xe90>
     f44:       e59f3e44        ldr     r3, [pc, #3652] ; 1d90
     <lttng_event_reserve+0xe94>
     f48:       e7921100        ldr     r1, [r2, r0, lsl #2]
     f4c:       e1a05000        mov     r5, r0
     f50:       e7932001        ldr     r2, [r3, r1]

My rusty ARM assembler knowledge analyzes this like:

ip = sp (ip being the scratch register)
push 11 registers * 4 bytes = 44 bytes onto the stack fp = ip - 4
sp = sp - 60 (sp = ip - 104)
push {lr} bl
0 <__gnu_mcount_nc> (dynamically patched to a pop {lr}) [...]
store r1 into mem location fp - 76

So yes, it does look like the -76 from fp is within the stack.

Quoted 4 lines

In GCC bug 58854 it was pretty clear that the restore of the sp register in the epilogue is moved to somewhere close to the prologue of the routine.
Are we missing some diffs from the assembly comparison?

My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning,
did not manage to get one working even after following 2 howtos, trying with Debian
packages, etc. Nathan told me this diff was the full comparison between the two
functions, but you'll understand that at this point I want to reproduce everything myself, because this is a _weird_ issue.

But after a long day of debugging, it's time for some sleep. I will get back to this
tomorrow,

Thanks,

Mathieu

Quoted 211 lines

Thanks

Luis

Quoted 189 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 27 lines

From: "Jakub Jelinek" <[email protected]>
To: "Luis Lozano" <[email protected]>
Cc: "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Mathieu Desnoyers" <[email protected]>, "Richard Henderson" <[email protected]>,
"Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>,
"Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul
E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Thursday, November 21, 2013 7:39:04 PM
Subject: Re: current_thread_info() not respecting program order with gcc
4.8.x

Quoted 6 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the
wrong path. This whole conversation started on an *assumption* that some accesses were being reordered.

evidence of the reorder or reproducer please?

Quoted 11 lines

Yeah, if a compiler bug is suspected, can anybody please open a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source,
compiler version, flags and how it was configured and some hint in which
function to look for what exactly? We don't necessarily need a runtime
small reproducer, but if it can be shown in the assembly what has been reordered and why you think it shouldn't, with the above mentioned input
that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day
of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1,
on ARM. It's been reproduced with a Linux kernel 3.12 so far, with lttng-modules
compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules
with gcc 4.8.x (and vice-versa). The problem only appears when lttng-modules
are compiled with gcc 4.8.x. The compiler version used to compile the rest
of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature:
-fno-sched-pressure
(sched pressure is there by default). Nathan tried compiling lttng-modules
with
-fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was fixing the issue, we tried identifying which code location was responsible
for working around the issue. Skipping a long investigation, here is the
executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown
below
works around the issue:

static
int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

                      uint32_t event_id)
{
        struct lttng_channel *lttng_chan =
        channel_get_private(ctx->chan);
        int ret, cpu;

        cpu = lib_ring_buffer_get_cpu(&client_config);
        if (cpu < 0)
                return -EPERM;
        ctx->cpu = cpu;

        switch (lttng_chan->header_type) {
        case 1: /* compact */
                if (event_id > 30)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        case 2: /* large */
                if (event_id > 65534)
                        ctx->rflags |= LTTNG_RFLAG_EXTENDED;
                break;
        default:
                WARN_ON_ONCE(1);
        }

        ret = lib_ring_buffer_reserve(&client_config, ctx);
        if (ret)
                goto put;
        lttng_write_event_header(&client_config, ctx, event_id);
        return 0;
put:
        lib_ring_buffer_put_cpu(&client_config);
        ---------------> barrier() added here;
        <----------------------------
        return ret;
}

Where barrier() is the usual asm volatile with "memory" clobber, nothing
else.

Nathan gave me the binary diff for the assembly generated for this function
without
the barrier and with the barrier:

--- /tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600
+++ /tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21
14:12:52.997355907 -0600

tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21 14:12:52.997355907 -0600 renamed from tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600 +5 -5
@@ -7,11 +7,11 @@
      f10:      ebfffffe        bl      0 <__gnu_mcount_nc>
      f14:      e5903000        ldr     r3, [r0]
      f18:      e1a04000        mov     r4, r0
-     f1c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
+     f1c:      e1a0000d        mov     r0, sp
      f20:      e5936030        ldr     r6, [r3, #48]   ; 0x30
-     f24:      e1a0000d        mov     r0, sp
-     f28:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
-     f2c:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f24:      e3c03d7f        bic     r3, r0, #8128   ; 0x1fc0
+     f28:      e3c3303f        bic     r3, r3, #63     ; 0x3f
+     f2c:      e50b1048        str     r1, [fp, #-72]  ; 0xffffffb8
      f30:      e5932004        ldr     r2, [r3, #4]
      f34:      e2822001        add     r2, r2, #1
      f38:      e5832004        str     r2, [r3, #4]
@@ -53,7 +53,7 @@
      fc8:      e3c3303f        bic     r3, r3, #63     ; 0x3f
      fcc:      e5933000        ldr     r3, [r3]
      fd0:      e3130002        tst     r3, #2
-     fd4:      0a000000        beq     fdc <lttng_event_reserve+0xe0>
+     fd4:      0a0002be        beq     1ad4

<lttng_event_reserve+0xbd8>

      fd8:      ebfffffe        bl      0 <preempt_schedule>
      fdc:      ea0002bc        b       1ad4
      <lttng_event_reserve+0xbd8>
      fe0:      e3500000        cmp     r0, #0

We tried disabling the ftrace function tracing to get mcount out of the way,
and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I need to double check with Nathan that I get the right disassembly for the
right
cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run on top of the thread stack (?). If it's the case, then anything stored on the stack below "sp" could be overwritten by an interrupt handler. This would fit well the reproduction scenario for this bug: Nathan runs LTTng tracing of kmem_cache_free tracepoint with hackbench running. A race between a short window of stack use below sp and interrupt handlers
would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Quoted 7 lines

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--

Quoted 3 lines

Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#35 lore raw

Re: [lttng-dev] current_thread_info() not respecting program order with gcc 4.8.x

Very interesting result:

Here is the asm diff between the problematic function compiled with gcc 4.8.2 vs that same function compiled with gcc 4.8.2 with the "lightly tested patch" in bug http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

lttng-ring-buffer-client-overwrite.ko-4.8.2-fix.objdump renamed from lttng-ring-buffer-client-overwrite.ko-4.8.2.objdump +4 -4
@@ -363,9 +363,9 @@ Disassembly of section .text:
      504:	e0647008 	rsb	r7, r4, r8
      508:	e0874004 	add	r4, r7, r4
      50c:	e0650004 	rsb	r0, r5, r4
-     510:	e24bd028 	sub	sp, fp, #40	; 0x28
+     510:	e58a6000 	str	r6, [sl]
      514:	e6ef0070 	uxtb	r0, r0
-     518:	e58a6000 	str	r6, [sl]
+     518:	e24bd028 	sub	sp, fp, #40	; 0x28
      51c:	e89daff0 	ldm	sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc}
 	...
 
@@ -1938,8 +1938,8 @@ Disassembly of section .text:
     1d74:	ebfffffe 	bl	0 <warn_slowpath_null>
     1d78:	e51b205c 	ldr	r2, [fp, #-92]	; 0x5c
     1d7c:	eafffef5 	b	1958 <lttng_event_reserve+0xa5c>
-    1d80:	e24bd028 	sub	sp, fp, #40	; 0x28
-    1d84:	e51b0048 	ldr	r0, [fp, #-72]	; 0x48
+    1d80:	e51b0048 	ldr	r0, [fp, #-72]	; 0x48
+    1d84:	e24bd028 	sub	sp, fp, #40	; 0x28
     1d88:	e89daff0 	ldm	sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc}
 	...
     1d98:	0000017e 	.word	0x0000017e

So far, Nathan has not reproduced the issue with the fixed gcc. He's running those stress tests a couple more hours to get more confidence in the result.

Not sure about the first two stores (they use the stack limit pointer "sl", which I'm clueless about), but the last snippet clearly fixes a one instruction stack usage below sp race window. Before the fix:

- 1d80: e24bd028 sub sp, fp, #40 ; 0x28
- 1d84: e51b0048 ldr r0, [fp, #-72] ; 0x48

sp = fp - 40
load from memory location fp - 72 .... wrong !

The full objdumps (before and after gcc fix) are attached.

Thoughts ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): CANxoKduwK=9__0WFXFcTWjQn3Rbn+HgSWZyL0FN_VuJ2Q_2TPQ@mail.gmail.com

#37 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

----- Original Message -----

Quoted 72 lines

From: "Luis Lozano" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Jakub Jelinek" <[email protected]>, "Alexander Holler" <[email protected]>, "Linus Torvalds" <[email protected]>, "Richard Henderson" <[email protected]>, "Linux Kernel Mailing List" <[email protected]>, "Will Deacon" <[email protected]>, "Catalin Marinas" <[email protected]>, "Peter Zijlstra" <[email protected]>, [email protected], "Nathan Lynch" <[email protected]>, "Paul E. McKenney" <[email protected]>, "Andrew Morton" <[email protected]>, "Bhaskar Janakiraman" <[email protected]>, "Han Shen" <[email protected]>
Sent: Friday, November 22, 2013 3:16:27 AM
Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Quoted 52 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 37 lines

From: "Luis Lozano" < [email protected] >

To: "Mathieu Desnoyers" < [email protected] >

Cc: "Jakub Jelinek" < [email protected] >, "Alexander Holler" < [email protected] >, "Linus Torvalds"

< [email protected] >, "Richard Henderson" < [email protected]

Quoted 1 line

,

"Linux Kernel Mailing List"

< [email protected] >, "Will Deacon" < [email protected] >, "Catalin Marinas" < [email protected] >,

"Peter Zijlstra" < [email protected] >, [email protected] , "Nathan Lynch" < [email protected] >, "Paul

E. McKenney" < [email protected] >, "Andrew Morton" < [email protected] >, "Bhaskar Janakiraman"

< [email protected] >, "Han Shen" < [email protected] >

Sent: Thursday, November 21, 2013 9:36:27 PM

Subject: Re: current_thread_info() not respecting program order with gcc 4.8.x

Hi Mathieu,

Yes the problem we were seeing with GCC bug 58854 is that an interrupt

handler was corrupting the stack of a routine which had an invalid

Quoted 7 lines

value of SP (not really on "top" of the stack) for part of the

routine.

But in the assembly code you sent, I don't see where sp is being

modified... or where the access to "below" sp is happening.

The following instruction

f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

Quoted 7 lines
Quoted 3 lines

appears in the assembly generated by gcc 4.8.2, but not in the one

generated by 4.7.3, which makes me wonder if it's good or not.

It is there in 4.7.3 just a few lines difference.
I don't see any meaningful difference between the 2 assembly snippets... Just instructions ordered a little different?

Assembly generated by 4.7.3 vs 4.8.2 are much more different.

The diff I've presented yesterday is between 4.8.2 and 4.8.2+barrier() at the end of the function error path. When I can get a cross compiler to work, I will send out more the complete assembly listings of the function.

Thanks,

Mathieu

Quoted 525 lines

Luis

Quoted 522 lines

With

slightly different build output from the diff (probably a different config

from Nathan), the first function instructions look like:

00000efc <lttng_event_reserve>:

efc: e1a0c00d mov ip, sp

f00: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}

f04: e24cb004 sub fp, ip, #4

f08: e24dd03c sub sp, sp, #60 ; 0x3c

f0c: e52de004 push {lr} ; (str lr, [sp, #-4]!)

f10: ebfffffe bl 0 <__gnu_mcount_nc>

f14: e5903000 ldr r3, [r0]

f18: e1a04000 mov r4, r0

f1c: e1a0000d mov r0, sp

f20: e5936030 ldr r6, [r3, #48] ; 0x30

f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0

f28: e3c3303f bic r3, r3, #63 ; 0x3f

f2c: e50b104c str r1, [fp, #-76] ; 0xffffffb4

f30: e5932004 ldr r2, [r3, #4]

f34: e2822001 add r2, r2, #1

f38: e5832004 str r2, [r3, #4]

f3c: ebfffffe bl 0 <debug_smp_processor_id>

f40: e59f2e44 ldr r2, [pc, #3652] ; 1d8c <lttng_event_reserve+0xe90>

f44: e59f3e44 ldr r3, [pc, #3652] ; 1d90 <lttng_event_reserve+0xe94>

f48: e7921100 ldr r1, [r2, r0, lsl #2]

f4c: e1a05000 mov r5, r0

f50: e7932001 ldr r2, [r3, r1]

My rusty ARM assembler knowledge analyzes this like:

ip = sp (ip being the scratch register)

push 11 registers * 4 bytes = 44 bytes onto the stack

fp = ip - 4

sp = sp - 60 (sp = ip - 104)

push {lr} bl

0 <__gnu_mcount_nc> (dynamically patched to a pop {lr})

[...]

store r1 into mem location fp - 76

So yes, it does look like the -76 from fp is within the stack.

Quoted 7 lines

In GCC bug 58854 it was pretty clear that the restore of the sp

register in the epilogue is moved to somewhere close to the prologue

of the routine.

Are we missing some diffs from the assembly comparison?

My next step is to setup my own 4.8.2 ARM cross-compiler. I've tried this morning,

did not manage to get one working even after following 2 howtos, trying with
Debian

packages, etc. Nathan told me this diff was the full comparison between the two

functions, but you'll understand that at this point I want to reproduce

everything myself, because this is a _weird_ issue.

But after a long day of debugging, it's time for some sleep. I will get back
to this

tomorrow,

Thanks,

Mathieu

Quoted 391 lines

Thanks

Luis

On Thu, Nov 21, 2013 at 5:57 PM, Mathieu Desnoyers

Quoted 357 lines · < [email protected] > wrote:

----- Original Message -----

Quoted 60 lines

From: "Jakub Jelinek" < [email protected] >

To: "Luis Lozano" < [email protected] >

Cc "Alexander Holler" < [email protected] >, "Linus Torvalds"

< [email protected] >, "Mathieu Desnoyers"

< [email protected] >, "Richard Henderson" < [email protected] >,

"Linux Kernel Mailing List"

< [email protected] >, "Will Deacon" < [email protected]

Quoted 1 line

,

"Catalin Marinas" < [email protected] >,

"Peter Zijlstra" < [email protected] >, [email protected] ,

"Nathan Lynch" < [email protected] >, "Paul

E. McKenney" < [email protected] >, "Andrew Morton"

< [email protected] >, "Bhaskar Janakiraman"

< [email protected] >, "Han Shen" < [email protected] >

Sent: Thursday, November 21, 2013 7:39:04 PM

Subject: Re: current_thread_info() not respecting program order with gcc

4.8.x

Quoted 10 lines · Luis Lozano

I think we need a reproducer. Without this we may all be going on the

wrong path. This whole conversation started on an *assumption* that

some accesses were being reordered.

evidence of the reorder or reproducer please?

Yeah, if a compiler bug is suspected, can anybody please open

a bugreport in http://gcc.gnu.org/bugzilla/ with the preprocessed source,

compiler version, flags and how it was configured and some hint in which

Quoted 8 lines

function to look for what exactly? We don't necessarily need a runtime

small reproducer, but if it can be shown in the assembly what has been

reordered and why you think it shouldn't, with the above mentioned input

that ought to be sufficient. Thanks.

OK OK, let me reply on list first so I can share the result of a full day

of bug hunting. We're not there yet, but many options have been eliminated.

The issue shows up in stress test, when tracing with lttng-modules 2.4-rc1,

on ARM. It's been reproduced with a Linux kernel 3.12 so far, with

lttng-modules

compiled against that kernel.

First, I asked Nathan to compile his kernel with gcc 4.7, and lttng-modules

with gcc 4.8.x (and vice-versa). The problem only appears when

lttng-modules

are compiled with gcc 4.8.x. The compiler version used to compile the rest

of the kernel does not matter.

Then I looked at gcc 4.8 changelog for ARM, new feature:

-fno-sched-pressure

(sched pressure is there by default). Nathan tried compiling lttng-modules

with

-fno-sched-pressure. The problem still reproduces.

Knowing that adding barrier() outside of preempt_disable()/enable() was

fixing the issue, we tried identifying which code location was responsible

for working around the issue. Skipping a long investigation, here is the

executive summary:

http://git.lttng.org/?p=lttng-modules.git;a=blob;f=lttng-ring-buffer-client.h;h=50c47b3bf49f6c2dd24e250cf1a9b97808cd8e27;hb=HEAD

Has the following function. We identified that adding a barrier() as shown

below

works around the issue:

static

int lttng_event_reserve(struct lib_ring_buffer_ctx *ctx,

uint32_t event_id)

{

struct lttng_channel *lttng_chan = channel_get_private(ctx->chan);

int ret, cpu;

cpu = lib_ring_buffer_get_cpu(&client_config);

if (cpu < 0)

return -EPERM;

ctx->cpu = cpu;

switch (lttng_chan->header_type) {

case 1: /* compact */

if (event_id > 30)

ctx->rflags |= LTTNG_RFLAG_EXTENDED;

break;

case 2: /* large */

if (event_id > 65534)

ctx->rflags |= LTTNG_RFLAG_EXTENDED;

break;

default:

WARN_ON_ONCE(1);

}

ret = lib_ring_buffer_reserve(&client_config, ctx);

if (ret)

goto put;

lttng_write_event_header(&client_config, ctx, event_id);

return 0;

put:

lib_ring_buffer_put_cpu(&client_config);

---------------> barrier() added here;

<----------------------------

return ret;

}

Where barrier() is the usual asm volatile with "memory" clobber, nothing

else.

Nathan gave me the binary diff for the assembly generated for this function

without

the barrier and with the barrier:

--- /tmp/lttng_event_reserve-4.8.2.dump 2013-11-21 11:14:14.536495079 -0600

+++ /tmp/lttng_event_reserve-with-barrier-4.8.2.dump 2013-11-21

14:12:52.997355907 -0600

(quoted diff) +0 -0
@@ -7,11 +7,11 @@
 

f10: ebfffffe bl 0 <__gnu_mcount_nc>

f14: e5903000 ldr r3, [r0]

f18: e1a04000 mov r4, r0

- f1c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

+ f1c: e1a0000d mov r0, sp

f20: e5936030 ldr r6, [r3, #48] ; 0x30

- f24: e1a0000d mov r0, sp

- f28: e3c03d7f bic r3, r0, #8128 ; 0x1fc0

- f2c: e3c3303f bic r3, r3, #63 ; 0x3f

+ f24: e3c03d7f bic r3, r0, #8128 ; 0x1fc0

+ f28: e3c3303f bic r3, r3, #63 ; 0x3f

+ f2c: e50b1048 str r1, [fp, #-72] ; 0xffffffb8

f30: e5932004 ldr r2, [r3, #4]

f34: e2822001 add r2, r2, #1

f38: e5832004 str r2, [r3, #4]

(quoted diff) +0 -0
@@ -53,7 +53,7 @@
 

fc8: e3c3303f bic r3, r3, #63 ; 0x3f

fcc: e5933000 ldr r3, [r3]

fd0: e3130002 tst r3, #2

- fd4: 0a000000 beq fdc <lttng_event_reserve+0xe0>

+ fd4: 0a0002be beq 1ad4 <lttng_event_reserve+0xbd8>

fd8: ebfffffe bl 0 <preempt_schedule>

fdc: ea0002bc b 1ad4 <lttng_event_reserve+0xbd8>

fe0: e3500000 cmp r0, #0

We tried disabling the ftrace function tracing to get mcount out of the

way,

and the problem still reproduces.

I'm stopping here in terms of details about the disassembly, because I

need to double check with Nathan that I get the right disassembly for the

right

cases. I also terribly need to setup a 4.8.2 ARM cross-compiler on my

machine.

I'm attaching Nathan's ARM configuration.

It does look behave a bit like this bug pointed out by Luis:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854

AFAIU (please correct me if I am wrong), ARM's interrupt handler run

on top of the thread stack (?). If it's the case, then anything stored

on the stack below "sp" could be overwritten by an interrupt handler.

This would fit well the reproduction scenario for this bug: Nathan runs

LTTng tracing of kmem_cache_free tracepoint with hackbench running.

A race between a short window of stack use below sp and interrupt handlers

would trigger with this kind of stress-test.

Thoughts ?

Thanks,

Mathieu

Quoted 7 lines

--

Mathieu Desnoyers

EfficiOS Inc.

http://www.efficios.com

Quoted 8 lines

--

Luis A. Lozano | Software Engineer | [email protected] | +1 (408)431-5164

--

Mathieu Desnoyers

EfficiOS Inc.

http://www.efficios.com

Signature · 3 lines
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#38 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

In reply to Alexander Holler
Quoted 2 lines · Alexander Holler

Sorry, that I still disagree.

Why? You're wrong.

I mean, anybody who disagrees with me is pretty much wrong just on pure principles, but I actually mean a deeper kind of wrong than that. I mean a very objective "you're clearly wrong".

Quoted 2 lines

I try to describe it more clearly why I still think that the problem might be because of that const declaration.

.. and then you use a totally bogus example to try to "prove" your point.

The example you use has nothing to do with what the function in case actually does.

The function we actually talk about RETURNS THE SAME VALUE EVERY TIME IT IS CALLED, regardless of arguments (which it doesn't happen to have). Ergo, it is "const".

Your example is pure and utter shit, since you still get confused about what is actually const and what isn't.

The fact is, "current_thread_info()" returns a constant value (within that execution context). Always has, always will. It fundamentally HAS to, since that is - by definition - what that "thread info" is all about.

The dereference (your "->somewhere_local") happens *outside* that const function scope. And no, that dereference is not const. But that's not what we tell gcc, and never has been.

So don't try to confuse things by trying to make
"current_thread_info()" something it isn't.

Basically, your whole argument boils down to "if the function did something else than what it does, then it wouldn't be const, so we shouldn't mark it const". But that argument is BULLSHIT, because the fact is, the function *doesn't* do what you try to claim it does.

Linus

#39 lore raw

Re: current_thread_info() not respecting program order with gcc 4.8.x

Am 22.11.2013 01:17, schrieb Linus Torvalds:

Quoted 6 lines · Alexander Holler

Basically, your whole argument boils down to "if the function did something else than what it does, then it wouldn't be const, so we shouldn't mark it const". But that argument is BULLSHIT, because the fact is, the function *doesn't* do what you try to claim it does.

Maybe gcc just makes the same false conclusion as I did in my description.

I read it as current_thread_info() returns "a pointer to something local" instead of returns "a pointer". Might be BULLSHIT but would explain the bug which seems to exist.

Alexander Holler