Skip to contentSkip to conversation

system gets stuck in a lock during boot

DISCUSSION Justin P. Mattock 8 participants 62 messages Aug 13, 2009 – Oct 12, 2009

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249
[   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1
[   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be
[   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40
[   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40
[   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd
[   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b
[   0.659276] [<ffffffff81696f0a>] ? event_trace_init+0x0/0x1a2
[   0.659519] [<ffffffff81697061>] event_trace_init+0x157/0x1a2
[   0.659762] [<ffffffff81696f0a>] ? event_trace_init+0x0/0x1a2
[   0.660008] [<ffffffff810091db>] do_one_initcall+0x65/0x14e
[   0.660251] [<ffffffff81140000>] ? proc_set_super+0x49/0x5d
[   0.660488] [<ffffffff8167ef53>] kernel_init+0x160/0x1cc
[   0.660731] [<ffffffff8102801a>] child_rip+0xa/0x20
[   0.660973] [<ffffffff8167edf3>] ? kernel_init+0x0/0x1cc
[   0.661220] [<ffffffff81028010>] ? child_rip+0x0/0x20

seems my other machine boots just fine, maybe this is something with x86_64.

Justin P. Mattock

#1 lore raw
Quoted 5 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

Quoted 15 lines
[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249
[   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1
[   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be
[   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40
[   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40
[   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd
[   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Thanks.

Quoted 21 lines
[   0.659276] [<ffffffff81696f0a>] ? event_trace_init+0x0/0x1a2
[   0.659519] [<ffffffff81697061>] event_trace_init+0x157/0x1a2
[   0.659762] [<ffffffff81696f0a>] ? event_trace_init+0x0/0x1a2
[   0.660008] [<ffffffff810091db>] do_one_initcall+0x65/0x14e
[   0.660251] [<ffffffff81140000>] ? proc_set_super+0x49/0x5d
[   0.660488] [<ffffffff8167ef53>] kernel_init+0x160/0x1cc
[   0.660731] [<ffffffff8102801a>] child_rip+0xa/0x20
[   0.660973] [<ffffffff8167edf3>] ? kernel_init+0x0/0x1cc
[   0.661220] [<ffffffff81028010>] ? child_rip+0x0/0x20

seems my other machine boots just fine, maybe this is something with x86_64.

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

maintainer
#2 lore raw
Quoted 24 lines · Frederic Weisbecker
Quoted 5 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

Quoted 15 lines
[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249
[   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1
[   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be
[   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40
[   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40
[   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd
[   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b
Quoted 4 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

#3 lore raw
Quoted 25 lines · Steven Rostedt
Quoted 24 lines · Frederic Weisbecker
Quoted 20 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249 [   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1 [   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be [   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40 [   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40 [   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd [   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b

Quoted 10 lines
Quoted 4 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

yeah this happens with rc6 but not with rc5.

The system is a linux from scratch build x86_64 I used these flags for building everything:

CFLAGS=" -m64 -mtune=core2 -march=core2 -mfpmath=both -O2 -pipe -fomit-frame-pointer -fstack-protector" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"

also: I have a macbook(not x86_64) that is running rc6 that seems to be fine.(I can double check).

attached is .config, dmesg as well.

Signature · 1 line
Justin P. Mattock
maintainer
#4 lore raw
Quoted 42 lines · Justin Mattock
Quoted 25 lines · Steven Rostedt
Quoted 24 lines · Frederic Weisbecker
Quoted 20 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249 [   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1 [   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be [   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40 [   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40 [   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd [   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b

Quoted 10 lines
Quoted 4 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

yeah this happens with rc6 but not with rc5.

The system is a linux from scratch build x86_64 I used these flags for building everything:

Quoted 7 lines

CFLAGS=" -m64 -mtune=core2 -march=core2 -mfpmath=both -O2 -pipe -fomit-frame-pointer -fstack-protector" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"

also: I have a macbook(not x86_64) that is running rc6 that seems to be fine.(I can double check).

attached is .config, dmesg as well.

I used your config and I can not reproduce the bug you have. Was this a constant issue? That is, was it reproducible at every boot?

-- Steve

#5 lore raw
Quoted 53 lines · Steven Rostedt
Quoted 50 lines · Justin Mattock
Quoted 26 lines · Steven Rostedt
Quoted 24 lines · Frederic Weisbecker
Quoted 21 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249
[   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1
[   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be
[   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40
[   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40
[   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd
[   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b
Quoted 11 lines
Quoted 5 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

yeah this happens with rc6 but not with rc5.

The system is a linux from scratch build x86_64 I used these flags for building everything:

  CFLAGS=" -m64 -mtune=core2 -march=core2 -mfpmath=both  -O2 -pipe
  -fomit-frame-pointer -fstack-protector" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"

also: I have a macbook(not x86_64) that is running rc6 that seems to be fine.(I can double check).

Quoted 7 lines
Quoted 2 lines

attached is .config, dmesg as well.

I used your config and I can not reproduce the bug you have. Was this a constant issue? That is, was it reproducible at every boot?

-- Steve

yes.
every boot this shows up.

Justin P. Mattock

maintainer
#6 lore raw
In reply to Justin Mattock
Quoted 7 lines · Justin Mattock
Quoted 6 lines · Steven Rostedt
Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

BTW, it does not look like it got stuck in a lock. The backtrace shows that it paniced inside event_create_dir due to a bad pointer.

Quoted 35 lines
Quoted 19 lines
Quoted 19 lines
Quoted 16 lines

[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249 [   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1 [   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be [   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40 [   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40 [   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd [   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b

Quoted 10 lines
Quoted 4 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

yeah this happens with rc6 but not with rc5.

The system is a linux from scratch build x86_64 I used these flags for building everything:

Quoted 7 lines

CFLAGS=" -m64 -mtune=core2 -march=core2 -mfpmath=both -O2 -pipe -fomit-frame-pointer -fstack-protector" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"

also: I have a macbook(not x86_64) that is running rc6 that seems to be fine.(I can double check).

attached is .config, dmesg as well.

BTW, how did you get the dmesg? Is it from a clean boot?

Since it works fine in -rc5 but breaks in -rc6, could you do a git bisect on it? It shouldn't take too long.

Thanks,

-- Steve

#7 lore raw
Quoted 61 lines · Steven Rostedt
Quoted 11 lines · Justin Mattock
Quoted 9 lines · Steven Rostedt
Quoted 7 lines · Frederic Weisbecker
Quoted 5 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

BTW, it does not look like it got stuck in a lock. The backtrace shows that it paniced inside event_create_dir due to a bad pointer.

Quoted 33 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249
[   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1
[   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be
[   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40
[   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40
[   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd
[   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b
Quoted 11 lines
Quoted 5 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

yeah this happens with rc6 but not with rc5.

The system is a linux from scratch build x86_64 I used these flags for building everything:

Quoted 8 lines
  CFLAGS=" -m64 -mtune=core2 -march=core2 -mfpmath=both  -O2 -pipe
  -fomit-frame-pointer -fstack-protector" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"

also: I have a macbook(not x86_64) that is running rc6 that seems to be fine.(I can double check).

attached is .config, dmesg as well.

Quoted 10 lines

BTW, how did you get the dmesg? Is it from a clean boot?

Since it works fine in -rc5 but breaks in -rc6, could you do a git bisect on it? It shouldn't take too long.

Thanks,

-- Steve

dmesg is from rc5 figured it would be good to
send that.

as for whatever is happening with this I don't really know.

The problem with doing a git bisect is these kernels are tar.balls although I do have a fresh pulled tree which might work.
if not Ill just start reverting
commits from the web interface until she boots properly. (luckily its from rc5 to rc6)

At the moment I'm trying to get the grub2 properly configured so I can get more of this message on the screen. and then manually write it down.
(have to learn the new gfxload thing).

Justin P. Mattock

#8 lore raw
In reply to Steven Rostedt
Quoted 61 lines · Steven Rostedt
Quoted 11 lines · Justin Mattock
Quoted 9 lines · Steven Rostedt
Quoted 7 lines · Frederic Weisbecker
Quoted 5 lines · Justin P. Mattock

I posted already and had no reply,
I cannot boot my system with the latest git

seems to be stuck in some lock:

BTW, it does not look like it got stuck in a lock. The backtrace shows that it paniced inside event_create_dir due to a bad pointer.

Quoted 33 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
[   0.655640] [<ffffffff81440c33>] panic+0x84/0x129
[   0.655883] [<ffffffff81063f96>]  do_exit+0x84/0x67e
[   0.656125] [<ffffffff81445044>] oops_end+0x9c/0xb7
[   0.656368] [<ffffffff81044a6c>] no_context+0x200/0x223
[   0.656611] [<ffffffff81442d3d>] ? __mutex_lock_slowpath+0x226/0x249
[   0.656848] [<ffffffff81044c36>] __bad_area_nosemaphore+0x1a7/0x1e1
[   0.657097] [<ffffffff81442b01>]? mutex_unlock+0x1c/0x32
[   0.657097] [<ffffffff811ce094>] ? inode_doinit_with_dentry+0x533/0x5be
[   0.657579] [<ffffffff81443ee0>] ? _spin_unlock+0x1c/0x32
[   0.657822] [<ffffffff81044c91>] bad_area_nosemaphore+0x2a/0x40
[   0.658066] [<ffffffff811ce149>] ? selinux_d_instantiate+0x2a/0x40
[   0.658309] [<ffffffff8144687d>] do_page_fault+0x192/0x2dd
[   0.658549] [<ffffffff814444cf>] page_fault+0x1f/0x30
[   0.658791] [<ffffffff8120e856>] ? strcmp+0x17/0x46
[   0.659040] [<ffffffff810b9ca0>] event_create_dir+0x49/0x37b
Quoted 11 lines
Quoted 5 lines

Ouch...

I've never seen such panic. Could you send me your config, hopefully I could reproduce it?

Send me the config too please.

-- Steve

yeah this happens with rc6 but not with rc5.

The system is a linux from scratch build x86_64 I used these flags for building everything:

Quoted 8 lines
  CFLAGS=" -m64 -mtune=core2 -march=core2 -mfpmath=both  -O2 -pipe
  -fomit-frame-pointer -fstack-protector" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"

also: I have a macbook(not x86_64) that is running rc6 that seems to be fine.(I can double check).

attached is .config, dmesg as well.

Quoted 10 lines

BTW, how did you get the dmesg? Is it from a clean boot?

Since it works fine in -rc5 but breaks in -rc6, could you do a git bisect on it? It shouldn't take too long.

Thanks,

-- Steve

well grub2 sure works o.k.
the new gfxload is pretty good.
problem is the screen just says
[Linux-bzImage etc.....]
and that's it. (was hoping to make the
font smaller to reveal the bug).

at this point the only choice I have is to undo
commits until I hit the right one.

Justin P. Mattock

maintainer
#9 lore raw
Quoted 9 lines · Justin P. Mattock

well grub2 sure works o.k.
the new gfxload is pretty good.
problem is the screen just says
[Linux-bzImage etc.....]
and that's it. (was hoping to make the
font smaller to reveal the bug).

at this point the only choice I have is to undo
commits until I hit the right one.

How come you can not do a git bisect?

It would be faster.

-- Steve

#10 lore raw
Quoted 13 lines · Steven Rostedt
Quoted 10 lines · Justin P. Mattock

well grub2 sure works o.k.
the new gfxload is pretty good.
problem is the screen just says
[Linux-bzImage etc.....]
and that's it. (was hoping to make the
font smaller to reveal the bug).

at this point the only choice I have is to undo
commits until I hit the right one.

Quoted 8 lines

How come you can not do a git bisect?

It would be faster.

-- Steve

I'll give it a try

Justin P. Mattock

#11 lore raw
In reply to Justin P. Mattock
Quoted 17 lines
Quoted 10 lines

BTW, how did you get the dmesg? Is it from a clean boot?

Since it works fine in -rc5 but breaks in -rc6, could you do a git bisect on it? It shouldn't take too long.

Thanks,

-- Steve

well grub2 sure works o.k.
the new gfxload is pretty good.
problem is the screen just says
[Linux-bzImage etc.....]
and that's it. (was hoping to make the
font smaller to reveal the bug).

Quoted 3 lines

at this point the only choice I have is to undo
commits until I hit the right one.

You still can do bisect manually. or use quilt?

#12 lore raw
Quoted 18 lines · Li Zefan
Quoted 18 lines
Quoted 11 lines

BTW, how did you get the dmesg? Is it from a clean boot?

Since it works fine in -rc5 but breaks in -rc6, could you do a git bisect on it? It shouldn't take too long.

Thanks,

-- Steve

well grub2 sure works o.k.
the new gfxload is pretty good.
problem is the screen just says
[Linux-bzImage etc.....]
and that's it. (was hoping to make the
font smaller to reveal the bug).

Quoted 9 lines
Quoted 4 lines

at this point the only choice I have is to undo
commits until I hit the right one.

You still can do bisect manually. or use quilt?

after going schizo with reverting patches, I've come down to thinking that this freeze might
have to do with the tracer merge that happened
a few days ago. (been avoiding that one) I'll have to revert (If I can) the whole thing to see if this machine boot's with rc6.
As for perf I reverted everything
up to rc5, and the machine still sticks telling me that might not be the issue.

Justin P. Mattock

#13 lore raw
In reply to Li Zefan
Quoted 18 lines · Li Zefan
Quoted 18 lines
Quoted 11 lines

BTW, how did you get the dmesg? Is it from a clean boot?

Since it works fine in -rc5 but breaks in -rc6, could you do a git bisect on it? It shouldn't take too long.

Thanks,

-- Steve

well grub2 sure works o.k.
the new gfxload is pretty good.
problem is the screen just says
[Linux-bzImage etc.....]
and that's it. (was hoping to make the
font smaller to reveal the bug).

Quoted 9 lines
Quoted 4 lines

at this point the only choice I have is to undo
commits until I hit the right one.

You still can do bisect manually. or use quilt?

Hey alright 2.6.31-rc6 booted.

I must admit, I've been afraid of doing
a git bisect for some time now. but after learning to do a bisect, I cant see a better solution to finding a bug(biggest issue I see is having a huge .config).

Anyways the culprit for my issue is this commit:

af6af30c0fcd77e621638e53ef8b176bca8bd3b4

reverting this on rc6 booted the machine up.

Justin P. Mattock

#14 lore raw
CC Peter, Ingo
Quoted 7 lines

Hey alright 2.6.31-rc6 booted.

I must admit, I've been afraid of doing
a git bisect for some time now. but after learning to do a bisect, I cant see a better solution to finding a bug(biggest issue I see is having a huge .config).

Quoted 6 lines

Anyways the culprit for my issue is this commit:

af6af30c0fcd77e621638e53ef8b176bca8bd3b4

reverting this on rc6 booted the machine up.

Thanks!

Hmm...I don't see how this commit can cause oops:

Review struct ftrace_event_call { · line 124

Quoted 12 lines
(quoted diff) +3 -5
@@ -119,11 +119,9 @@ struct ftrace_event_call {
        void                    *filter;
        void                    *mod;
 
-#ifdef CONFIG_EVENT_PROFILE
-       atomic_t        profile_count;
-       int             (*profile_enable)(struct ftrace_event_call *);
-       void            (*profile_disable)(struct ftrace_event_call *);
-#endif
+       atomic_t                profile_count;
+       int                     (*profile_enable)(struct ftrace_event_call *);
+       void                    (*profile_disable)(struct ftrace_event_call *);

In your .config, CONFIG_EVENT_PROFILE=y, so this change makes no difference.

kernel/trace/trace_event_profile.c int ftrace_profile_enable(int event_id) · line 17

Quoted 13 lines
(quoted diff) +0 -0
 };
 
 #define MAX_FILTER_PRED                32
kernel/trace/trace_event_profile.c +1 -1
index 5b5895a..11ba5bb 100644
@@ -14,7 +14,7 @@ int ftrace_profile_enable(int event_id)
 
        mutex_lock(&event_mutex);
        list_for_each_entry(event, &ftrace_events, list) {
-               if (event->id == event_id) {
+               if (event->id == event_id && event->profile_enable) {

You will run into this hunk only when you run the perf tool, so should have nothing to do with the boot failure.

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 943

Quoted 13 lines
kernel/trace/trace_event_profile.c +0 -0
                        ret = event->profile_enable(event);
                        break;
                }
kernel/trace/trace_events.c +1 -1
index 23d2972..e75276a 100644
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                entry = trace_create_file("enable", 0644, call->dir, call,
                                          enable);
 
-       if (call->id)
+       if (call->id && call->profile_enable)

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 945

Quoted 2 lines
kernel/trace/trace_events.c +0 -0
                entry = trace_create_file("id", 0444, call->dir, call,
                                          id);

Any way, I don't think this commit does the right thing:

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

can you try this patch:

---
 include/linux/ftrace_event.h   |    2 ++
 kernel/trace/trace_events.c    |    2 +-
 tools/perf/util/parse-events.c |    8 +++++++-
 3 files changed, 10 insertions(+), 2 deletions(-)
include/linux/ftrace_event.h +2 -0
index a81170d..398c925 100644
@@ -121,9 +121,11 @@ struct ftrace_event_call {
 	void			*filter;
 	void			*mod;
 
+#ifdef CONFIG_EVENT_PROFILE
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
+#endif
 };
 
 #define MAX_FILTER_PRED		32
kernel/trace/trace_events.c +1 -1
index e75276a..23d2972 100644
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentry *d_events,
 		entry = trace_create_file("enable", 0644, call->dir, call,
 					  enable);
 
-	if (call->id && call->profile_enable)
+	if (call->id)
 		entry = trace_create_file("id", 0444, call->dir, call,
 					  id);
 
tools/perf/util/parse-events.c +7 -1
index 0441784..1077ebd 100644
@@ -113,13 +113,19 @@ static unsigned long hw_cache_stat[C(MAX)] = {
  [C(BPU)]	= (CACHE_READ),
 };
 
+static int is_tp_subsystem(struct dirent *sys_dir)
+{
+	return !!strcmp(sys_dir->d_name, "ftrace");
+}
+
 #define for_each_subsystem(sys_dir, sys_dirent, sys_next, file, st)	       \
 	while (!readdir_r(sys_dir, &sys_dirent, &sys_next) && sys_next)	       \
 	if (snprintf(file, MAXPATHLEN, "%s/%s", debugfs_path,	       	       \
 			sys_dirent.d_name) &&		       		       \
 	   (!stat(file, &st)) && (S_ISDIR(st.st_mode)) &&		       \
 	   (strcmp(sys_dirent.d_name, ".")) &&				       \
-	   (strcmp(sys_dirent.d_name, "..")))
+	   (strcmp(sys_dirent.d_name, "..")) &&				       \
+	   (is_tp_subsystem(&sys_dirent)))
 
 static int tp_event_has_id(struct dirent *sys_dir, struct dirent *evt_dir)
 {
#15 lore raw
Quoted 14 lines · Li Zefan
CC Peter, Ingo
Quoted 11 lines

Hey alright 2.6.31-rc6 booted.

I must admit, I've been afraid of doing
a git bisect for some time now. but after learning to do a bisect, I cant see a better solution to finding a bug(biggest issue I see is having a huge .config).

Anyways the culprit for my issue is this commit:

af6af30c0fcd77e621638e53ef8b176bca8bd3b4

Quoted 8 lines
Quoted 3 lines

reverting this on rc6 booted the machine up.

Thanks!

Hmm...I don't see how this commit can cause oops:

same here, perf is not enabled at all.
before the git bisect I manually reverted all perf
and came to the conclusion the perf had nothing to do with this. (must have missed this commit).

Quoted 28 lines
Quoted 4 lines
(quoted diff) +0 -0
@@ -119,11 +119,9 @@ struct ftrace_event_call {
         void                    *filter;
         void                    *mod;

Review struct ftrace_event_call { · line 125

Quoted 10 lines
(quoted diff) +3 -5
 
-#ifdef CONFIG_EVENT_PROFILE
-       atomic_t        profile_count;
-       int             (*profile_enable)(struct ftrace_event_call *);
-       void            (*profile_disable)(struct ftrace_event_call *);
-#endif
+       atomic_t                profile_count;
+       int                     (*profile_enable)(struct ftrace_event_call *);
+       void                    (*profile_disable)(struct ftrace_event_call *);
 

In your .config, CONFIG_EVENT_PROFILE=y, so this change makes no difference.

Quoted 9 lines
(quoted diff) +0 -0
  };
 
  #define MAX_FILTER_PRED                32
kernel/trace/trace_event_profile.c +0 -0
index 5b5895a..11ba5bb 100644
@@ -14,7 +14,7 @@ int ftrace_profile_enable(int event_id)
 
Quoted 9 lines

kernel/trace/trace_event_profile.c int ftrace_profile_enable(int event_id) · line 18

Quoted 5 lines
kernel/trace/trace_event_profile.c +1 -1
         mutex_lock(&event_mutex);
         list_for_each_entry(event,&ftrace_events, list) {
-               if (event->id == event_id) {
+               if (event->id == event_id&&  event->profile_enable) {
 

You will run into this hunk only when you run the perf tool, so should have nothing to do with the boot failure.

I haven't really looked into perf yet, nor have it enabled.

Quoted 12 lines
Quoted 12 lines
kernel/trace/trace_event_profile.c +0 -0
                         ret = event->profile_enable(event);
                         break;
                 }
kernel/trace/trace_events.c +0 -0
index 23d2972..e75276a 100644
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                 entry = trace_create_file("enable", 0644, call->dir, call,
                                           enable);
 
Quoted 6 lines

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 944

Quoted 3 lines
kernel/trace/trace_events.c +1 -1
-       if (call->id)
+       if (call->id&&  call->profile_enable)
 

We do an extra check on ->profile_enable, shouldn't cause bug..

maybe this has something to do with SELinux, since I saw with the oops something with inode_do_init_dentry, and some other stuff pertaining to SELinux(again though I'm lost as to why I'm hitting this).

Quoted 71 lines

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 4 lines
kernel/trace/trace_events.c +0 -0
                 entry = trace_create_file("id", 0444, call->dir, call,
                                           id);

Any way, I don't think this commit does the right thing:

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

can you try this patch:

---
  include/linux/ftrace_event.h   |    2 ++
  kernel/trace/trace_events.c    |    2 +-
  tools/perf/util/parse-events.c |    8 +++++++-
  3 files changed, 10 insertions(+), 2 deletions(-)
include/linux/ftrace_event.h +2 -0
index a81170d..398c925 100644
@@ -121,9 +121,11 @@ struct ftrace_event_call {
  	void			*filter;
  	void			*mod;
 
+#ifdef CONFIG_EVENT_PROFILE
  	atomic_t		profile_count;
  	int			(*profile_enable)(struct ftrace_event_call *);
  	void			(*profile_disable)(struct ftrace_event_call *);
+#endif
  };
 
  #define MAX_FILTER_PRED		32
kernel/trace/trace_events.c +1 -1
index e75276a..23d2972 100644
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentry *d_events,
  		entry = trace_create_file("enable", 0644, call->dir, call,
  					  enable);
 
-	if (call->id&&  call->profile_enable)
+	if (call->id)
  		entry = trace_create_file("id", 0444, call->dir, call,
  					  id);
 
tools/perf/util/parse-events.c +5 -0
index 0441784..1077ebd 100644
@@ -113,13 +113,19 @@ static unsigned long hw_cache_stat[C(MAX)] = {
   [C(BPU)]	= (CACHE_READ),
  };
 
+static int is_tp_subsystem(struct dirent *sys_dir)
+{
+	return !!strcmp(sys_dir->d_name, "ftrace");
+}
+
  #define for_each_subsystem(sys_dir, sys_dirent, sys_next, file, st)	       \
  	while (!readdir_r(sys_dir,&sys_dirent,&sys_next)&&  sys_next)	       \

tools/perf/util/parse-events.c static unsigned long hw_cache_stat[C(MAX)] = { · line 131

Quoted 10 lines
tools/perf/util/parse-events.c +2 -1
  	if (snprintf(file, MAXPATHLEN, "%s/%s", debugfs_path,	       	       \
  			sys_dirent.d_name)&&		       		\
  	   (!stat(file,&st))&&  (S_ISDIR(st.st_mode))&&		\
  	   (strcmp(sys_dirent.d_name, "."))&&				\
-	   (strcmp(sys_dirent.d_name, "..")))
+	   (strcmp(sys_dirent.d_name, ".."))&&				\
+	   (is_tp_subsystem(&sys_dirent)))
 
  static int tp_event_has_id(struct dirent *sys_dir, struct dirent *evt_dir)
  {

Alright, in the morning I try this out and let you know what happens.

Thanks for looking at this.

Justin P. Mattock

maintainer
#16 lore raw
In reply to Li Zefan
Quoted 14 lines · Li Zefan

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 943

Quoted 6 lines
kernel/trace/trace_events.c +1 -1
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                entry = trace_create_file("enable", 0644, call->dir, call,
                                          enable);
 
-       if (call->id)
+       if (call->id && call->profile_enable)

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 2 lines
kernel/trace/trace_events.c +0 -0
                entry = trace_create_file("id", 0444, call->dir, call,
                                          id);

Any way, I don't think this commit does the right thing:

Quoted 7 lines

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

The id files really must stay conditional.

Aside from that, you only add #ifdef fuzz back which with his config is moot, and shouldn't result in anything but a slightly bigger structure to begin with.

#17 lore raw
Quoted 22 lines · Peter Zijlstra
Quoted 12 lines · Li Zefan

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 943

Quoted 6 lines
kernel/trace/trace_events.c +1 -1
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                entry = trace_create_file("enable", 0644, call->dir, call,
                                          enable);
 
-       if (call->id)
+       if (call->id && call->profile_enable)

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 2 lines
kernel/trace/trace_events.c +0 -0
                entry = trace_create_file("id", 0444, call->dir, call,
                                          id);

Any way, I don't think this commit does the right thing:

Quoted 7 lines

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

Quoted 7 lines

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

But removing the id file from events/ftrace/ might break some ftrace binary parsers?

And this commit makes 'id' file disapeared with CONFIG_EVENT_PROFILE=n.

Quoted 4 lines

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

Agree.

Quoted 2 lines

The id files really must stay conditional.

I don't think it's a good idea to connect it with perfcounter this way. Wouldn't adding a 'profilable' file more intuitive?

Quoted 4 lines

Aside from that, you only add #ifdef fuzz back which with his config is moot, and shouldn't result in anything but a slightly bigger structure to begin with.

Yeah, actually I still fail to see where the bug is, maybe the commit that Justin bisected down is not the real culprit..

#18 lore raw
Quoted 60 lines · Li Zefan
Quoted 26 lines · Peter Zijlstra
Quoted 15 lines · Li Zefan

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 944

Quoted 7 lines
kernel/trace/trace_events.c +1 -1
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                 entry = trace_create_file("enable", 0644, call->dir, call,
                                           enable);
 
-       if (call->id)
+       if (call->id&&  call->profile_enable)
 

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 3 lines
kernel/trace/trace_events.c +0 -0
                 entry = trace_create_file("id", 0444, call->dir, call,
                                           id);

Any way, I don't think this commit does the right thing:

Quoted 8 lines

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

Quoted 8 lines

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

But removing the id file from events/ftrace/ might break some ftrace binary parsers?

And this commit makes 'id' file disapeared with CONFIG_EVENT_PROFILE=n.

Quoted 5 lines

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

Agree.

Quoted 3 lines

The id files really must stay conditional.

I don't think it's a good idea to connect it with perfcounter this way. Wouldn't adding a 'profilable' file more intuitive?

Quoted 11 lines
Quoted 5 lines

Aside from that, you only add #ifdef fuzz back which with his config is moot, and shouldn't result in anything but a slightly bigger structure to begin with.

Yeah, actually I still fail to see where the bug is, maybe the commit that Justin bisected down is not the real culprit..

FWIW, here is the git bisect log:

git bisect start
# bad: [6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ryusuke/nilfs2
git bisect bad 6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f
# good: [b600ffaebcc4791add19e04306f0478a963abe71] Merge branch 'irq-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect good b600ffaebcc4791add19e04306f0478a963abe71
# good: [ae83060026537885fd23737af161fee8afd04f4b] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6
git bisect good ae83060026537885fd23737af161fee8afd04f4b
# bad: [b637dc0dba6a243da2c74f5d02b42ba5eeb9425e] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound-2.6
git bisect bad b637dc0dba6a243da2c74f5d02b42ba5eeb9425e
# bad: [cf7fdd57f978d40ceb9a0f58a25f5cf9c84d6f33] USB: fix oops on disconnect in cdc-acm
git bisect bad cf7fdd57f978d40ceb9a0f58a25f5cf9c84d6f33
# good: [131f7340b4f93f9a4a8e5a65abbd352b34d0ee08] Merge branch 'drm-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6
git bisect good 131f7340b4f93f9a4a8e5a65abbd352b34d0ee08
# good: [c0c60c4b9ab45bb02b20796401dd6a90770fd0ee] ARM: 5639/1: arm: clkdev.c should include <linux/clk.h>
git bisect good c0c60c4b9ab45bb02b20796401dd6a90770fd0ee
# bad: [da758ddede96dd850945d3417ff75209a666ba0d] Merge branch 'perfcounters-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad da758ddede96dd850945d3417ff75209a666ba0d
# good: [389623fef0e8b088f293c437d3b7094fe82349fd] Merge git://git.infradead.org/mtd-2.6
git bisect good 389623fef0e8b088f293c437d3b7094fe82349fd
# bad: [af6af30c0fcd77e621638e53ef8b176bca8bd3b4] ftrace: Fix perf-tracepoint OOPS
git bisect bad af6af30c0fcd77e621638e53ef8b176bca8bd3b4
# good: [2cdbc46d7b2cb0acb68c3ecad93b000552121fa6] perf: Auto-detect libbfd
git bisect good 2cdbc46d7b2cb0acb68c3ecad93b000552121fa6
# good: [386c0b702b1ea81c0f54f5c9832a3d4a52270f14] perf report: Add missing command line options to man page
git bisect good 386c0b702b1ea81c0f54f5c9832a3d4a52270f14

all I know is with 2.6.31-rc5 booted fine, as soon I loaded 2.6.31-rc6 I hit this stuckage(I can write it manually down if you need it, seems ohci1394_dma gets a panic).

Justin P. Mattock

maintainer
#19 lore raw
In reply to Li Zefan
Quoted 27 lines · Li Zefan
Quoted 26 lines · Peter Zijlstra
Quoted 12 lines · Li Zefan

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 943

Quoted 6 lines
kernel/trace/trace_events.c +1 -1
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                entry = trace_create_file("enable", 0644, call->dir, call,
                                          enable);
 
-       if (call->id)
+       if (call->id && call->profile_enable)

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 2 lines
kernel/trace/trace_events.c +0 -0
                entry = trace_create_file("id", 0444, call->dir, call,
                                          id);

Any way, I don't think this commit does the right thing:

Quoted 7 lines

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

Quoted 8 lines
Quoted 3 lines

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

But removing the id file from events/ftrace/ might break some ftrace binary parsers?

And this commit makes 'id' file disapeared with CONFIG_EVENT_PROFILE=n.

I introduced the id file for EVENT_PROFILE, tough luck if someone else relies on it.

Quoted 5 lines
Quoted 4 lines

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

Quoted 7 lines

Agree.

Quoted 2 lines

The id files really must stay conditional.

I don't think it's a good idea to connect it with perfcounter this way. Wouldn't adding a 'profilable' file more intuitive?

its called: id ;-)

#20 lore raw
Quoted 56 lines · Peter Zijlstra
Quoted 32 lines · Li Zefan
Quoted 30 lines · Peter Zijlstra
Quoted 15 lines · Li Zefan

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 944

Quoted 7 lines
kernel/trace/trace_events.c +1 -1
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                 entry = trace_create_file("enable", 0644, call->dir, call,
                                           enable);
 
-       if (call->id)
+       if (call->id&&  call->profile_enable)
 

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 3 lines
kernel/trace/trace_events.c +0 -0
                 entry = trace_create_file("id", 0444, call->dir, call,
                                           id);

Any way, I don't think this commit does the right thing:

Quoted 8 lines

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

Quoted 9 lines
Quoted 4 lines

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

But removing the id file from events/ftrace/ might break some ftrace binary parsers?

And this commit makes 'id' file disapeared with CONFIG_EVENT_PROFILE=n.

I introduced the id file for EVENT_PROFILE, tough luck if someone else relies on it.

Quoted 8 lines
Quoted 5 lines

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

Agree.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

The id files really must stay conditional.

I don't think it's a good idea to connect it with perfcounter this way. Wouldn't adding a 'profilable' file more intuitive?

its called: id ;-)

Alright, I'm going to go back to the beginning
and start my build from scratch(make sure things are legit).

Justin P. Mattock

#21 lore raw
In reply to Peter Zijlstra
Quoted 56 lines · Peter Zijlstra
Quoted 32 lines · Li Zefan
Quoted 30 lines · Peter Zijlstra
Quoted 15 lines · Li Zefan

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentr · line 944

Quoted 7 lines
kernel/trace/trace_events.c +1 -1
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call, struct dentr
                 entry = trace_create_file("enable", 0644, call->dir, call,
                                           enable);
 
-       if (call->id)
+       if (call->id&&  call->profile_enable)
 

We do an extra check on ->profile_enable, shouldn't cause bug..

kernel/trace/trace_events.c event_create_dir(struct ftrace_event_call *call, struct dentry *d_events, · line 945

Quoted 3 lines
kernel/trace/trace_events.c +0 -0
                 entry = trace_create_file("id", 0444, call->dir, call,
                                           id);

Any way, I don't think this commit does the right thing:

Quoted 8 lines

- If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

- if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

Quoted 9 lines
Quoted 4 lines

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

But removing the id file from events/ftrace/ might break some ftrace binary parsers?

And this commit makes 'id' file disapeared with CONFIG_EVENT_PROFILE=n.

I introduced the id file for EVENT_PROFILE, tough luck if someone else relies on it.

Quoted 8 lines
Quoted 5 lines

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

Agree.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

The id files really must stay conditional.

I don't think it's a good idea to connect it with perfcounter this way. Wouldn't adding a 'profilable' file more intuitive?

its called: id ;-)

After looking at ps auxZ
I see xterm as root instead of my user name. This gives me enough evidence to throw this system away and start over.

sorry for the waste of time.

Justin P. Mattock

#22 lore raw
Quoted 83 lines · Justin P. Mattock
Quoted 74 lines · Peter Zijlstra
Quoted 46 lines · Li Zefan
Quoted 34 lines · Peter Zijlstra
Quoted 21 lines · Li Zefan
Quoted 10 lines
(quoted diff) +0 -0
@@ -940,7 +940,7 @@ event_create_dir(struct ftrace_event_call *call,

struct dentr
                entry = trace_create_file("enable", 0644, call->dir, call,
                                          enable);

-       if (call->id)
+       if (call->id&&  call->profile_enable)

We do an extra check on ->profile_enable, shouldn't cause bug..

Quoted 4 lines

                entry = trace_create_file("id", 0444, call->dir, call, id);

Any way, I don't think this commit does the right thing:

Quoted 8 lines

  - If CONFIG_EVENT_PROFILE=y, we'll create events/<dir>/<event>/id, except events/ftrace/<event>/id.

  - if CONFIG_EVENT_PROFILE=n, there's no 'id' file at all!

I think it's better to skip ftrace/ dir in perf tool code, instead of skipping creating id files in ftrace code.

Quoted 8 lines

No, it does do the right thing. Your patch breaks things because not all tracepoints are created through TRACE_EVENT() and will thus not have their profile_enable/disable hooks set.

By giving them an ID file, there is no way to distinguish good from bad tracepoints.

Quoted 5 lines

But removing the id file from events/ftrace/ might break some ftrace binary parsers?

And this commit makes 'id' file disapeared with CONFIG_EVENT_PROFILE=n.

I introduced the id file for EVENT_PROFILE, tough luck if someone else relies on it.

Quoted 15 lines
Quoted 6 lines

Expempting ftrace is no solid solution, suppose someone else does a TRACE_EVENT_FORMAT() tracepoint, how would you know you could use it as a profile event source?

Agree.

Quoted 4 lines

The id files really must stay conditional.

Quoted 7 lines
Quoted 3 lines

I don't think it's a good idea to connect it with perfcounter this way. Wouldn't adding a 'profilable' file more intuitive?

its called: id ;-)

Quoted 9 lines

After looking at ps auxZ
I see xterm as root instead of my user name. This gives me enough evidence to throw this system away and start over.

sorry for the waste of time.

Justin P. Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Signature · 1 line
Justin P. Mattock
maintainer
#23 lore raw
Quoted 7 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you share your .config so we could have a try?

#24 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

Quoted 5 lines
Quoted 2 lines

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you share your .config so we could have a try?

Justin sent his config in an earlier mail in this thread. But Steven didn't reproduce the bug with the config, neither me.

And we don't receive another bug report regarding to this crash.

maintainer
#25 lore raw
Quoted 14 lines · Li Zefan
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

Quoted 5 lines
Quoted 2 lines

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you share your .config so we could have a try?

Quoted 6 lines

Justin sent his config in an earlier mail in this thread. But Steven didn't reproduce the bug with the config, neither me.

And we don't receive another bug report regarding to this crash.

Right, weird story this..

#26 lore raw
In reply to Justin Mattock
Quoted 7 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

#27 lore raw
Quoted 25 lines · Ingo Molnar
Quoted 8 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 10 lines

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

shoot, I did not see your post here.
when looking at my bisect log, I guess after a
git bisect reset it clears?

Anyways after git bisect had finished
I looked manually at the commits that it had generated
the one which I had sent in a post previously, and this one:
9424edc2da097c8589fcc24a72552d33e54be161

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6
made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

At the moment I don't have a problem applying a patch
to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

Justin P. Mattock

#28 lore raw
Quoted 36 lines · Justin P. Mattock
Quoted 25 lines · Ingo Molnar
Quoted 8 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 10 lines

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

Quoted 8 lines

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

Quoted 5 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

Quoted 8 lines

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

Quoted 5 lines
Quoted 5 lines

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

Ingo

#29 lore raw
Quoted 45 lines · Ingo Molnar
Quoted 42 lines · Justin P. Mattock
Quoted 26 lines · Ingo Molnar
Quoted 9 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 11 lines

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 10 lines
Quoted 6 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 9 lines
Quoted 9 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 10 lines
Quoted 5 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

Ingo

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking the question of why.
In any case I still think I'm setting something wrong with either gcc, or something that might be causing this from userland.

Justin P. Mattock

#30 lore raw
Quoted 106 lines · Justin P. Mattock
Quoted 51 lines · Ingo Molnar
Quoted 47 lines · Justin P. Mattock
Quoted 29 lines · Ingo Molnar
Quoted 10 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 11 lines

[ _If_ that does not fix the bug then to save time you can 'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

       Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 10 lines
Quoted 6 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

 9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 10 lines
Quoted 10 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 10 lines
Quoted 5 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

       Ingo

Quoted 9 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

Signature · 1 line
Justin P. Mattock
#31 lore raw
Quoted 143 lines · Justin Mattock
Quoted 105 lines
Quoted 51 lines · Justin P. Mattock
Quoted 49 lines
Quoted 29 lines · Ingo Molnar
Quoted 10 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 11 lines

[ _If_ that does not fix the bug then to save time you can 'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

       Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 10 lines
Quoted 6 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

 9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 10 lines
Quoted 10 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 10 lines
Quoted 5 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

       Ingo

Quoted 9 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2

(quoted diff) +0 -2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"

CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

Quoted 9 lines

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

hi Justin,

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

thanks,

-Jason

include/asm-generic/vmlinux.lds.h +1 -0
index 6ad76bf..0029af4 100644
@@ -164,6 +164,7 @@
 	LIKELY_PROFILE()		       				\
 	BRANCH_PROFILE()						\
 	TRACE_PRINTKS()							\
+	. = ALIGN(32);							\
 	FTRACE_EVENTS()							\
 	TRACE_SYSCALLS()
 
include/linux/ftrace_event.h +1 -1
index a81170d..43f9f1e 100644
@@ -124,7 +124,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(32)));
 
 #define MAX_FILTER_PRED		32
 #define MAX_FILTER_STR_VAL	128
include/trace/ftrace.h +1 -1
index f64fbaa..4697fb6 100644
@@ -600,7 +600,7 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
+__attribute__((__aligned__(32)))					\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\
#32 lore raw
Quoted 156 lines · Jason Baron
Quoted 143 lines · Justin Mattock
Quoted 105 lines
Quoted 51 lines · Justin P. Mattock
Quoted 49 lines
Quoted 29 lines · Ingo Molnar
Quoted 10 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 11 lines

[ _If_ that does not fix the bug then to save time you can 'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

       Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 10 lines
Quoted 6 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

 9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 10 lines
Quoted 10 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 10 lines
Quoted 5 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

       Ingo

Quoted 9 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2

(quoted diff) +0 -2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"

CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

Quoted 9 lines

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

hi Justin,

Quoted 7 lines

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

Would be nice to know precisely what kind of problem is being hit here - we'd like to fix either the kernel or GCC - depending on where the bug lies.

Ingo

#33 lore raw
Quoted 167 lines · Ingo Molnar
Quoted 156 lines · Jason Baron
Quoted 143 lines · Justin Mattock
Quoted 105 lines
Quoted 51 lines · Justin P. Mattock
Quoted 49 lines
Quoted 29 lines · Ingo Molnar
Quoted 10 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 11 lines

[ _If_ that does not fix the bug then to save time you can 'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

       Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 10 lines
Quoted 6 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

 9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 10 lines
Quoted 10 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 10 lines
Quoted 5 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

  git revert af6af30c0f

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

       Ingo

Quoted 9 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2

(quoted diff) +0 -2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"

CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
 -O2 -pipe -fomit-frame-pointer" CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

Quoted 9 lines

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

hi Justin,

Quoted 7 lines

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

Quoted 6 lines

Would be nice to know precisely what kind of problem is being hit here - we'd like to fix either the kernel or GCC - depending on where the bug lies.

       Ingo

So I wasn't going crazy....
Anyways that system(clfs)
I still have, I can go ahead and
put it back on the machine and see if I hit this again(keep in mind, just got back from a 7hr drive, so it might be tomorrow).

Signature · 1 line
Justin P. Mattock
#34 lore raw
Quoted 178 lines · Justin Mattock
Quoted 168 lines · Ingo Molnar
Quoted 157 lines · Jason Baron
Quoted 143 lines · Justin Mattock
Quoted 105 lines
Quoted 50 lines · Justin P. Mattock
Quoted 49 lines
Quoted 28 lines · Ingo Molnar
Quoted 10 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 12 lines

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 11 lines
Quoted 7 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 9 lines
Quoted 9 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 11 lines
Quoted 6 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

Quoted 10 lines
Quoted 3 lines
Quoted 3 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

Ingo

Quoted 10 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2

(quoted diff) +0 -2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"

CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

Quoted 10 lines

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

hi Justin,

Quoted 8 lines

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

Quoted 7 lines

Would be nice to know precisely what kind of problem is being hit here - we'd like to fix either the kernel or GCC - depending on where the bug lies.

Ingo

Quoted 8 lines

So I wasn't going crazy....
Anyways that system(clfs)
I still have, I can go ahead and
put it back on the machine and see if I hit this again(keep in mind, just got back from a 7hr drive, so it might be tomorrow).

o.k. I put back on that system, and
hit the error. I add your patch to 2.6.31-rc6, and the latest git(a few days old).
I still am hitting this, but with your patch
I'm able to see the beginning of this panic:
(Ill write it manually)

[   2.523966] kernel panic - not syncing: No init found. try passing init= option
to the kernel
[   2.524394] Pid: 1, comm: swapper Not tainted 2.6.31-rc6 #6
[   2.524633] Call Trace:
[   2.524875] [<ffffffff813a5b72>] panic+0x75/0x120
[   2.525119] [<ffffffff8100910f>] init_post+0xef/0xf5
[   2.525357] [<ffffffff815f6cf0>] kernel_init+0x198/0x1a3
[   2.525600] [<ffffffff8102410a>] child_rip+0xa/0x20
[   2.525842] [<ffffffff815f6b58>] ? kernel_init+0x0/0x1a3
[   2.526084] [>ffffffff810224100>] ? child_rip+0x0/0x20

Seems I only hit this with using gcc 4.5.0 and compiling sysvinit with SELinux support to load the policy at boot. (here's the patch I used
http://readlist.com/lists/tycho.nsa.gov/selinux/3/15451.html).

Sound's like gcc is doing something(correct me if I'm
wrong) because the other systems I have are using the same packages except for and older version of gcc.
maybe I should update sysvinit with a better patch to load the policy.

Justin P. Mattock

maintainer
#35 lore raw
Quoted 9 lines · Justin P. Mattock

Justin Mattock wrote:

o.k. I put back on that system, and
hit the error. I add your patch to 2.6.31-rc6, and the latest git(a few days old).
I still am hitting this, but with your patch
I'm able to see the beginning of this panic:
(Ill write it manually)

Quoted 7 lines
[   2.523966] kernel panic - not syncing: No init found. try passing 
init= option
to the kernel
[   2.524394] Pid: 1, comm: swapper Not tainted 2.6.31-rc6 #6
[   2.524633] Call Trace:
[   2.524875] [<ffffffff813a5b72>] panic+0x75/0x120
[   2.525119] [<ffffffff8100910f>] init_post+0xef/0xf5

Strange. This panic is just telling you it could not find an "init" to execute.

-- Steve

Quoted 16 lines
[   2.525357] [<ffffffff815f6cf0>] kernel_init+0x198/0x1a3
[   2.525600] [<ffffffff8102410a>] child_rip+0xa/0x20
[   2.525842] [<ffffffff815f6b58>] ? kernel_init+0x0/0x1a3
[   2.526084] [>ffffffff810224100>] ? child_rip+0x0/0x20

Seems I only hit this with using gcc 4.5.0 and compiling sysvinit with SELinux support to load the policy at boot. (here's the patch I used
http://readlist.com/lists/tycho.nsa.gov/selinux/3/15451.html).

Sound's like gcc is doing something(correct me if I'm
wrong) because the other systems I have are using the same packages except for and older version of gcc.
maybe I should update sysvinit with a better patch to load the policy.

Justin P. Mattock

#36 lore raw
Quoted 22 lines · Steven Rostedt
Quoted 11 lines · Justin P. Mattock

Justin Mattock wrote:

o.k. I put back on that system, and
hit the error. I add your patch to 2.6.31-rc6, and the latest git(a few days old).
I still am hitting this, but with your patch
I'm able to see the beginning of this panic:
(Ill write it manually)

Quoted 8 lines
[   2.523966] kernel panic - not syncing: No init found. try passing
init= option
to the kernel
[   2.524394] Pid: 1, comm: swapper Not tainted 2.6.31-rc6 #6
[   2.524633] Call Trace:
[   2.524875] [<ffffffff813a5b72>] panic+0x75/0x120
[   2.525119] [<ffffffff8100910f>] init_post+0xef/0xf5
Quoted 4 lines

Strange. This panic is just telling you it could not find an "init" to execute.

It is strange, my only guess is something gcc is doing(like mentioned other systems I have run fine while using gcc 4.4)

Quoted 23 lines

-- Steve

Quoted 17 lines
[   2.525357] [<ffffffff815f6cf0>] kernel_init+0x198/0x1a3
[   2.525600] [<ffffffff8102410a>] child_rip+0xa/0x20
[   2.525842] [<ffffffff815f6b58>] ? kernel_init+0x0/0x1a3
[   2.526084] [>ffffffff810224100>] ? child_rip+0x0/0x20

Seems I only hit this with using gcc 4.5.0 and compiling sysvinit with SELinux support to load the policy at boot. (here's the patch I used
http://readlist.com/lists/tycho.nsa.gov/selinux/3/15451.html).

Sound's like gcc is doing something(correct me if I'm
wrong) because the other systems I have are using the same packages except for and older version of gcc.
maybe I should update sysvinit with a better patch to load the policy.

Justin P. Mattock

Justin P. Mattock

#37 lore raw
In reply to Justin P. Mattock
Quoted 187 lines · Justin P. Mattock
Quoted 178 lines · Justin Mattock
Quoted 168 lines · Ingo Molnar
Quoted 157 lines · Jason Baron
Quoted 143 lines · Justin Mattock
Quoted 105 lines
Quoted 50 lines · Justin P. Mattock
Quoted 49 lines
Quoted 28 lines · Ingo Molnar
Quoted 10 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 12 lines

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Quoted 11 lines
Quoted 7 lines

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 9 lines
Quoted 9 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

Quoted 11 lines
Quoted 6 lines

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

Quoted 10 lines
Quoted 3 lines
Quoted 3 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

Ingo

Quoted 10 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2

(quoted diff) +0 -2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"

CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

Quoted 10 lines

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

hi Justin,

Quoted 8 lines

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

Quoted 7 lines

Would be nice to know precisely what kind of problem is being hit here - we'd like to fix either the kernel or GCC - depending on where the bug lies.

Ingo

Quoted 8 lines

So I wasn't going crazy....
Anyways that system(clfs)
I still have, I can go ahead and
put it back on the machine and see if I hit this again(keep in mind, just got back from a 7hr drive, so it might be tomorrow).

Quoted 2 lines

o.k. I put back on that system, and
hit the error. I add your patch to 2.6.31-rc6,

ok. is that error, the same as the error below? The error below looks completely different from the posted previously. So, it almost looks like you the patch fixed one problem, only to reveal another one. Is that correct?

Quoted 28 lines

and the latest git(a few days old).
I still am hitting this, but with your patch
I'm able to see the beginning of this panic:
(Ill write it manually)

[   2.523966] kernel panic - not syncing: No init found. try passing  
init= option
to the kernel
[   2.524394] Pid: 1, comm: swapper Not tainted 2.6.31-rc6 #6
[   2.524633] Call Trace:
[   2.524875] [<ffffffff813a5b72>] panic+0x75/0x120
[   2.525119] [<ffffffff8100910f>] init_post+0xef/0xf5
[   2.525357] [<ffffffff815f6cf0>] kernel_init+0x198/0x1a3
[   2.525600] [<ffffffff8102410a>] child_rip+0xa/0x20
[   2.525842] [<ffffffff815f6b58>] ? kernel_init+0x0/0x1a3
[   2.526084] [>ffffffff810224100>] ? child_rip+0x0/0x20

Seems I only hit this with using gcc 4.5.0 and compiling sysvinit with SELinux support to load the policy at boot. (here's the patch I used
http://readlist.com/lists/tycho.nsa.gov/selinux/3/15451.html).

Sound's like gcc is doing something(correct me if I'm
wrong) because the other systems I have are using the same packages except for and older version of gcc.
maybe I should update sysvinit with a better patch to load the policy.

Justin P. Mattock

#38 lore raw
Quoted 210 lines · Jason Baron
Quoted 208 lines · Justin P. Mattock
Quoted 206 lines · Justin Mattock
Quoted 186 lines · Ingo Molnar
Quoted 173 lines · Jason Baron
Quoted 157 lines · Justin Mattock
Quoted 118 lines
Quoted 59 lines · Justin P. Mattock
Quoted 58 lines
Quoted 30 lines · Ingo Molnar
Quoted 11 lines · Justin Mattock

O.K. I feel better, deleted
my system, and threw in a minimal built system with only the bare essentials to boot. (just to make sure things are correct).

unfortunately after building rc6 I'm still hitting this. really am not sure why this is happening.

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

on the latest kernel and seeing whether that fixes the lockup?

Bisections are very efficient and hence very sensitive as well to minimal errors. Just one small mistake near the end of a bisection can blame the wrong commit.

So the best way to double-check such 100%-triggerable crashes is to do the revert. I tried the revert and it can be done fine here.

Quoted 13 lines

[ _If_ that does not fix the bug then to save time you can
'backtrack' the bisection, instead of re-doing it completely. I.e. you have your bisection log, re-check the final steps going backwards. Once you find a discrepancy (i.e. a 'bad' point that is 'good' or the other way around), redo the bisection log commands up to that point and continue it up to the end. ]

Ingo

shoot, I did not see your post here. when looking at my bisect log, I guess after a git bisect reset it clears?

Anyways after git bisect had finished I looked manually at the commits that it had generated the one which I had sent in a post previously, and this one:

Quoted 9 lines
Quoted 4 lines
   9424edc2da097c8589fcc24a72552d33e54be161

(this commit has no effect on your kernel image, at all.)

yep. but it was worth a try.

Quoted 16 lines
Quoted 5 lines

at the time looking at the commit, I see this to be more of the cause because of it being related to elf as so forth, but as soon as I reverted this on rc6 made no difference.(the previous commit fixes this for me, on a regular tar.ball as well as in git.

Quoted 11 lines

I think at this point since this system is a fresh from scratch build, I think something might be wrong that I'm doing (all the CFLAGS, and such are in a previous post).

At the moment I don't have a problem applying a patch to the kernel for this. especially since I'm the only one that seems to be hitting this, then if more and more reports of this happen then we can go from there.

Quoted 6 lines

What would be nice is to verify your bisection end result, i.e. do what i suggested:

yeah I've done this on both kernels three to be exact, and all boot after reverting
Fix perf-tracepoint OOPS.

As for my system, I'm still convinced that I might be doing something wrong over here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Could you please double-check the bisection result by doing this:

git revert af6af30c0f

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

on the latest kernel and seeing whether that fixes the lockup?

if this doesnt fix it on latest -git then this commit is not the cause of the lockup.

Ingo

Quoted 11 lines

This commit(Fix perf-tracepoint OOPS.)does fix my stuckage, but I'm left, as well as others asking
the question of why.
In any case I still think I'm setting something wrong with either gcc, or something
that might be causing this from userland.

Justin P. Mattock

O.k. here something awkward about this issue I was experiencing. at the moment I have two imac's here the descriptions:

imac A) the one with the problem

OS: built from the clfs book
x86_64 multilib with only lib64

built everything with these flags:
CFLAGS="-m64 -mtune=core2 -march=core2

(quoted diff) +0 -2
-mfpmath=both -O2 -pipe -fomit-frame-pointer
-fstack-protection"

CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
while compiling everything with
gcc version: 4.5.0 20090730

imac B) the one that works

OS: clfs(just built a few days ago)
x86_64 pure64 bit build
(lib with a symlink to lib64)
CFLAGS="-m64 -mtune=core2 -march=core2
-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}" MAKEOPTS="{-j3}"
gcc version: 4.4.1 (GCC for Cross-LFS 4.4.1.20090722)

Quoted 11 lines

The only things I can think of is either I hit something because of gcc, something goes wrong with the libraries, or there something happening with either the option of mfpmath=both or stackprotection.

At this point since the kernel seems to be running fine, is to just trash the system that has this issue and just leave it at, I was hitting some weird anomaly.

hi Justin,

Quoted 9 lines

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

Quoted 8 lines

Would be nice to know precisely what kind of problem is being hit here - we'd like to fix either the kernel or GCC - depending on where the bug lies.

Ingo

So I wasn't going crazy....
Anyways that system(clfs)
I still have, I can go ahead and
put it back on the machine and see if I hit this again(keep in mind, just got back from a 7hr drive, so it might be tomorrow).

Quoted 10 lines
Quoted 3 lines

o.k. I put back on that system, and
hit the error. I add your patch to 2.6.31-rc6,

ok. is that error, the same as the error below? The error below looks completely different from the posted previously. So, it almost looks like you the patch fixed one problem, only to reveal another one. Is that correct?

Could be a different error, the problem I have is capturing this error i.g. tried ieee1394_dma=early to capture this, but that mechanism seems to error out.(ssh no go either because this happens so early). I think this is the top part of the error, because before adding your patch the system would boot a little farther(to fast to read anything)down the line, and I did see something in there about a kernel panic.

If you have any ideas on how I can capture this early, would be appreciated. (getting anything to log this early is a bit tricky).

Quoted 22 lines
Quoted 22 lines

and the latest git(a few days old).
I still am hitting this, but with your patch
I'm able to see the beginning of this panic:
(Ill write it manually)

[   2.523966] kernel panic - not syncing: No init found. try passing
init= option
to the kernel
[   2.524394] Pid: 1, comm: swapper Not tainted 2.6.31-rc6 #6
[   2.524633] Call Trace:
[   2.524875] [<ffffffff813a5b72>] panic+0x75/0x120
[   2.525119] [<ffffffff8100910f>] init_post+0xef/0xf5
[   2.525357] [<ffffffff815f6cf0>] kernel_init+0x198/0x1a3
[   2.525600] [<ffffffff8102410a>] child_rip+0xa/0x20
[   2.525842] [<ffffffff815f6b58>] ? kernel_init+0x0/0x1a3
[   2.526084] [>ffffffff810224100>] ? child_rip+0x0/0x20

Seems I only hit this with using gcc 4.5.0 and compiling sysvinit with SELinux support to load the policy at boot. (here's the patch I used
http://readlist.com/lists/tycho.nsa.gov/selinux/3/15451.html).

Quoted 9 lines
Quoted 7 lines

Sound's like gcc is doing something(correct me if I'm
wrong) because the other systems I have are using the same packages except for and older version of gcc.
maybe I should update sysvinit with a better patch to load the policy.

Justin P. Mattock

As a test Ill throw in a kernel that was compiled with gcc 4.4.0 just to see if this is a compiler/kernel issue.

Justin P. Mattock

maintainer
#39 lore raw
In reply to Jason Baron
Quoted 45 lines · Jason Baron

hi Justin,

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

thanks,

(quoted diff) +0 -1
-Jason
include/asm-generic/vmlinux.lds.h +1 -0
index 6ad76bf..0029af4 100644
@@ -164,6 +164,7 @@
 	LIKELY_PROFILE()		       				\
 	BRANCH_PROFILE()						\
 	TRACE_PRINTKS()							\
+	. = ALIGN(32);							\
 	FTRACE_EVENTS()							\
 	TRACE_SYSCALLS()
 
include/linux/ftrace_event.h +1 -1
index a81170d..43f9f1e 100644
@@ -124,7 +124,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(32)));
 
 #define MAX_FILTER_PRED		32
 #define MAX_FILTER_STR_VAL	128
include/trace/ftrace.h +0 -0
index f64fbaa..4697fb6 100644

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 606

Quoted 8 lines
include/trace/ftrace.h +1 -1
@@ -600,7 +600,7 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
+__attribute__((__aligned__(32)))					\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\

Are all alignments needed? Or just adding one might help. Or removing the one directly above?

-- Steve

#40 lore raw
Quoted 59 lines · Steven Rostedt
Quoted 45 lines · Jason Baron

hi Justin,

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

thanks,

(quoted diff) +0 -1
-Jason
include/asm-generic/vmlinux.lds.h +1 -0
index 6ad76bf..0029af4 100644
@@ -164,6 +164,7 @@
 	LIKELY_PROFILE()		       				\
 	BRANCH_PROFILE()						\
 	TRACE_PRINTKS()							\
+	. = ALIGN(32);							\
 	FTRACE_EVENTS()							\
 	TRACE_SYSCALLS()
 
include/linux/ftrace_event.h +1 -1
index a81170d..43f9f1e 100644
@@ -124,7 +124,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(32)));
 
 #define MAX_FILTER_PRED		32
 #define MAX_FILTER_STR_VAL	128
include/trace/ftrace.h +0 -0
index f64fbaa..4697fb6 100644

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 606

Quoted 8 lines
include/trace/ftrace.h +1 -1
@@ -600,7 +600,7 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
+__attribute__((__aligned__(32)))					\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\

Are all alignments needed? Or just adding one might help. Or removing the one directly above?

Quoted 2 lines

-- Steve

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +4 -1
index a81170d..7182f03 100644
@@ -124,7 +124,10 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */
 
 #define MAX_FILTER_PRED		32
 #define MAX_FILTER_STR_VAL	128
include/trace/ftrace.h +0 -1
index f64fbaa..e344e81 100644
@@ -600,7 +600,6 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\
#41 lore raw
Quoted 125 lines · Jason Baron
Quoted 60 lines · Steven Rostedt
Quoted 46 lines · Jason Baron

hi Justin,

I've been playing around with gcc '4.5' as well and hit a panic that looks very similar to what you've seen with stock 2.6.31 - I haven't seen it anywhere else. Anyways, it seems to be some sort of alignment issue with the 'struct ftrace_event_call'. I'm not sure yet if this is a compiler or kernel issue. But the following kernel patch fixes the issue for me. It would be interesting to verify if the patch also resolves the issue for you.

thanks,

(quoted diff) +0 -1
-Jason
include/asm-generic/vmlinux.lds.h +1 -0
index 6ad76bf..0029af4 100644
@@ -164,6 +164,7 @@
  	LIKELY_PROFILE()		       				\
  	BRANCH_PROFILE()						\
  	TRACE_PRINTKS()							\
+	. = ALIGN(32);							\
  	FTRACE_EVENTS()							\
  	TRACE_SYSCALLS()
 
include/linux/ftrace_event.h +1 -1
index a81170d..43f9f1e 100644
@@ -124,7 +124,7 @@ struct ftrace_event_call {
  	atomic_t		profile_count;
  	int			(*profile_enable)(struct ftrace_event_call *);
  	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(32)));
 
  #define MAX_FILTER_PRED		32
  #define MAX_FILTER_STR_VAL	128
include/trace/ftrace.h +0 -0
index f64fbaa..4697fb6 100644

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 606

Quoted 8 lines
include/trace/ftrace.h +1 -1
@@ -600,7 +600,7 @@ static int ftrace_raw_init_event_##call(void)				\
  }									\
  									\
  static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
+__attribute__((__aligned__(32)))					\
  __attribute__((section("_ftrace_events"))) event_##call = {		\
  	.name			= #call,				\
  	.system			= __stringify(TRACE_SYSTEM),		\

Are all alignments needed? Or just adding one might help. Or removing the one directly above?

Quoted 3 lines

-- Steve

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +4 -1
index a81170d..7182f03 100644
@@ -124,7 +124,10 @@ struct ftrace_event_call {
  	atomic_t		profile_count;
  	int			(*profile_enable)(struct ftrace_event_call *);
  	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */
 
  #define MAX_FILTER_PRED		32
  #define MAX_FILTER_STR_VAL	128
include/trace/ftrace.h +0 -0
index f64fbaa..e344e81 100644

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 605

Quoted 7 lines
include/trace/ftrace.h +0 -1
@@ -600,7 +600,6 @@ static int ftrace_raw_init_event_##call(void)				\
  }									\
  									\
  static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
  __attribute__((section("_ftrace_events"))) event_##call = {		\
  	.name			= #call,				\
  	.system			= __stringify(TRACE_SYSTEM),		\

shoot I don't know why this is still hitting. tried both patches and still.
As of now the only thing I can think of besides looking at kernel/compiler is the patch for sysvinit to load the policy(maybe something in there is old/outdated).

(BTW: not sure if it means anything but this system is x86_64 built from the multilib clfs, but with no 32 bit libs, pretty much how fedora11 has there system built)

Justin P. Mattock

maintainer
#42 lore raw
In reply to Jason Baron
Quoted 38 lines · Jason Baron

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +0 -0
index a81170d..7182f03 100644

include/linux/ftrace_event.h struct ftrace_event_call { · line 130

Quoted 8 lines
include/linux/ftrace_event.h +4 -1
@@ -124,7 +124,10 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */

Is this true for i386?

I just tried this patch and it seems to work. Can you give it a try.

include/linux/ftrace_event.h +1 -1
index 4ec5e67..044b70d 100644
@@ -133,7 +133,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(void);
 	void			(*profile_disable)(void);
-};
+} __attribute__((aligned(sizeof(struct list_head))));
 
 #define FTRACE_MAX_PROFILE_SIZE	2048
 
include/trace/ftrace.h +0 -2
index cc0d966..31e7637 100644
@@ -501,7 +501,6 @@ static void ftrace_profile_disable_##call(void)				\
  * }
  *
  * static struct ftrace_event_call __used
- * __attribute__((__aligned__(4)))
  * __attribute__((section("_ftrace_events"))) event_<call> = {
  *	.name			= "<call>",
  *	.system			= "<system>",
@@ -619,7 +618,6 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\
#43 lore raw
Quoted 88 lines · Steven Rostedt
Quoted 38 lines · Jason Baron

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +0 -0
index a81170d..7182f03 100644

include/linux/ftrace_event.h struct ftrace_event_call { · line 131

Quoted 9 lines
include/linux/ftrace_event.h +4 -1
@@ -124,7 +124,10 @@ struct ftrace_event_call {
  	atomic_t		profile_count;
  	int			(*profile_enable)(struct ftrace_event_call *);
  	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */
 

Is this true for i386?

I just tried this patch and it seems to work. Can you give it a try.

include/linux/ftrace_event.h +1 -1
index 4ec5e67..044b70d 100644
@@ -133,7 +133,7 @@ struct ftrace_event_call {
  	atomic_t		profile_count;
  	int			(*profile_enable)(void);
  	void			(*profile_disable)(void);
-};
+} __attribute__((aligned(sizeof(struct list_head))));
 
  #define FTRACE_MAX_PROFILE_SIZE	2048
 
include/trace/ftrace.h +0 -1
index cc0d966..31e7637 100644
@@ -501,7 +501,6 @@ static void ftrace_profile_disable_##call(void)				\
   * }
   *
   * static struct ftrace_event_call __used
- * __attribute__((__aligned__(4)))
   * __attribute__((section("_ftrace_events"))) event_<call>  = {
   *	.name			= "<call>",
   *	.system			= "<system>",

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 623

Quoted 7 lines
include/trace/ftrace.h +0 -1
@@ -619,7 +618,6 @@ static int ftrace_raw_init_event_##call(void)				\
  }									\
  									\
  static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
  __attribute__((section("_ftrace_events"))) event_##call = {		\
  	.name			= #call,				\
  	.system			= __stringify(TRACE_SYSTEM),		\

o.k. applied your patch, but unfortunantly
I still am hitting this kernel panic.

must admit I have no idea why this is doing this. (but am willing to sit through this, because eventually sooner or later will hit this if I update gcc).

Justin P. Mattock

maintainer
#44 lore raw
Quoted 7 lines · Justin P. Mattock
Quoted 1 line

o.k. applied your patch, but unfortunantly
I still am hitting this kernel panic.

must admit I have no idea why this is doing this. (but am willing to sit through this, because eventually sooner or later will hit this if I update gcc).

But the panic you showed was that it could not find an init to execute. Which looks like a setup issue and not a kernel bug.

-- Steve

#45 lore raw
Quoted 13 lines · Steven Rostedt
Quoted 9 lines · Justin P. Mattock
Quoted 2 lines

o.k. applied your patch, but unfortunantly
I still am hitting this kernel panic.

must admit I have no idea why this is doing this. (but am willing to sit through this, because eventually sooner or later will hit this if I update gcc).

Quoted 8 lines

But the panic you showed was that it could not find an init to execute. Which looks like a setup issue and not a kernel bug.

-- Steve

That's whats getting me, i.g. if I compile sysvinit normally without adding an SELinux patch the system boots, as soon as I compile sysvinit with SELinux support to load the policy, bam... I hit this.

I can send a post to SELinux and see what they think, and the go from there.

Justin P. Mattock

maintainer
#46 lore raw
Quoted 22 lines · Justin P. Mattock
Quoted 13 lines · Steven Rostedt
Quoted 9 lines · Justin P. Mattock
Quoted 2 lines

o.k. applied your patch, but unfortunantly
I still am hitting this kernel panic.

must admit I have no idea why this is doing this. (but am willing to sit through this, because eventually sooner or later will hit this if I update gcc).

Quoted 8 lines

But the panic you showed was that it could not find an init to execute. Which looks like a setup issue and not a kernel bug.

-- Steve

Quoted 7 lines

That's whats getting me, i.g. if I compile sysvinit normally without adding an SELinux patch the system boots, as soon as I compile sysvinit with SELinux support to load the policy, bam... I hit this.

I can send a post to SELinux and see what they think, and the go from there.

Oh! It's an SELinux thing. It probably prevents you from executing init ;-)

-- Steve

#47 lore raw
Quoted 32 lines · Steven Rostedt
Quoted 30 lines · Justin P. Mattock
Quoted 14 lines · Steven Rostedt
Quoted 10 lines · Justin P. Mattock
Quoted 2 lines

o.k. applied your patch, but unfortunantly
I still am hitting this kernel panic.

must admit I have no idea why this is doing this. (but am willing to sit through this, because eventually sooner or later will hit this if I update gcc).

Quoted 9 lines

But the panic you showed was that it could not find an init to execute. Which looks like a setup issue and not a kernel bug.

-- Steve

That's whats getting me, i.g. if I compile sysvinit normally without adding an SELinux patch the system boots, as soon as I compile sysvinit with SELinux support to load the policy, bam... I hit this.

Quoted 12 lines
Quoted 3 lines

I can send a post to SELinux and see what they think, and the go from there.

Oh! It's an SELinux thing. It probably prevents you from executing init ;-)

-- Steve

What I think is happening is while building
sysvinit with the SELinux patch, sysvinit is looking in /lib for libselinux(but could be wrong) but libselinux is in /lib64.
(here's my excuse as a newbie:)
part of a pain when building an x86_64 multilib is building
everything to point to lib64(pure64 with the soft link lib64 -> lib makes life so much easier).

I'm going to look at that patch and see how it tells -lselinux -lsepol to find the libs.
(if this is the case then I must admit I am a real newbie, and apologize for the confusion).

Justin P. Mattock

maintainer
#48 lore raw
Quoted 10 lines · Justin P. Mattock
Quoted 1 line

What I think is happening is while building
sysvinit with the SELinux patch, sysvinit is looking in /lib for libselinux(but could be wrong) but libselinux is in /lib64.
(here's my excuse as a newbie:)
part of a pain when building an x86_64 multilib is building
everything to point to lib64(pure64 with the soft link lib64 -> lib makes life so much easier).

Quoted 4 lines

I'm going to look at that patch and see how it tells -lselinux -lsepol to find the libs.
(if this is the case then I must admit I am a real newbie, and apologize for the confusion).

Justin,

Just being able to monkey around with the init code on top of SELinux already qualifies you to be well beyond a newbie ;-)

-- Steve

#49 lore raw
Quoted 20 lines · Steven Rostedt
Quoted 11 lines · Justin P. Mattock
Quoted 2 lines

What I think is happening is while building
sysvinit with the SELinux patch, sysvinit is looking in /lib for libselinux(but could be wrong) but libselinux is in /lib64.
(here's my excuse as a newbie:)
part of a pain when building an x86_64 multilib is building
everything to point to lib64(pure64 with the soft link lib64 -> lib makes life so much easier).

Quoted 5 lines

I'm going to look at that patch and see how it tells -lselinux -lsepol to find the libs.
(if this is the case then I must admit I am a real newbie, and apologize for the confusion).

Quoted 10 lines

Justin,

Just being able to monkey around with the init code on top of SELinux already qualifies you to be well beyond a newbie ;-)

-- Steve

Thanks man!!(I really appreciate that).
Im looking at sysvinit right now
Im thinking all it really needs is
LIBDIR/LDFLAGS=-L/lib64 -lselinux -lsepol
in the Makefile or something in that area.

Justin P. Mattock

#50 lore raw
In reply to Steven Rostedt
Quoted 20 lines · Steven Rostedt
Quoted 11 lines · Justin P. Mattock
Quoted 2 lines

What I think is happening is while building
sysvinit with the SELinux patch, sysvinit is looking in /lib for libselinux(but could be wrong) but libselinux is in /lib64.
(here's my excuse as a newbie:)
part of a pain when building an x86_64 multilib is building
everything to point to lib64(pure64 with the soft link lib64 -> lib makes life so much easier).

Quoted 5 lines

I'm going to look at that patch and see how it tells -lselinux -lsepol to find the libs.
(if this is the case then I must admit I am a real newbie, and apologize for the confusion).

Quoted 10 lines

Justin,

Just being able to monkey around with the init code on top of SELinux already qualifies you to be well beyond a newbie ;-)

-- Steve

yep like I said I owe you guys an apology, for
my mistake(thanks for taking the time).

From what it looks like init was looking for libselinux in /lib instead of /lib64.
after adjusting the Makefile and adding:
LDFLAGS = -s -L/lib64 -lselinux -lsepol
LIBDIR=/lib64
the system boots right up.
(sh*t built everything to point to lib64(xserver and all).., and forgot the modify init.c to do the same).

Justin P. Mattock

maintainer
#51 lore raw
Quoted 3 lines · Justin P. Mattock
Quoted 1 line

yep like I said I owe you guys an apology, for
my mistake(thanks for taking the time).

Well, thank you for reporting it. If nobody reports any problems, we just assume everything is OK, and as people curse us out and look for alternatives, we will be sipping our mai-tais in ignorant bliss.

-- Steve

Quoted 12 lines

From what it looks like init was looking for libselinux
in /lib instead of /lib64.
after adjusting the Makefile and adding:
LDFLAGS = -s -L/lib64 -lselinux -lsepol
LIBDIR=/lib64
the system boots right up.
(sh*t built everything to point to lib64(xserver and all).., and forgot the modify init.c to do the same).

Justin P. Mattock

#52 lore raw
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · Justin P. Mattock
Quoted 2 lines

yep like I said I owe you guys an apology, for
my mistake(thanks for taking the time).

Quoted 7 lines

Well, thank you for reporting it. If nobody reports any problems, we just assume everything is OK, and as people curse us out and look for alternatives, we will be sipping our mai-tais in ignorant bliss.

-- Steve

agree,
my main concern when reporting is to make sure that it's a real issue, and not a mistake like this one. Anyways, a mai-tais sure does sound good
(sign me up!!)

Quoted 9 lines
Quoted 9 lines

From what it looks like init was looking for libselinux
in /lib instead of /lib64.
after adjusting the Makefile and adding:
LDFLAGS = -s -L/lib64 -lselinux -lsepol
LIBDIR=/lib64
the system boots right up.
(sh*t built everything to point to lib64(xserver and all).., and forgot the modify init.c to do the same).

Quoted 6 lines
Quoted 3 lines

Justin P. Mattock

Justin P. mattock

#53 lore raw
Quoted 14 lines · Justin P. Mattock
Quoted 13 lines · Steven Rostedt
Quoted 5 lines · Justin P. Mattock
Quoted 2 lines

yep like I said I owe you guys an apology, for
my mistake(thanks for taking the time).

Well, thank you for reporting it. If nobody reports any problems, we just assume everything is OK, and as people curse us out and look for alternatives, we will be sipping our mai-tais in ignorant bliss.

Quoted 6 lines
Quoted 3 lines

-- Steve

agree, my main concern when reporting is to make sure that it's a real issue, and not a mistake like this one.

Well, it can be really hard to filter out whether a problem is somehow self-inflicted or caused by the kernel proper - thus we generally prefer over-reporting over under-reporting.

Even if it's self-inflicted it's useful on a meta level: if many people report it then we might end up adding more safety guards to disable a particular common pattern of shoot-foot-self incidents.

Ingo

#54 lore raw
Quoted 31 lines · Ingo Molnar
Quoted 16 lines · Justin P. Mattock
Quoted 14 lines · Steven Rostedt
Quoted 6 lines · Justin P. Mattock
Quoted 2 lines

yep like I said I owe you guys an apology, for
my mistake(thanks for taking the time).

Well, thank you for reporting it. If nobody reports any problems, we just assume everything is OK, and as people curse us out and look for alternatives, we will be sipping our mai-tais in ignorant bliss.

Quoted 7 lines
Quoted 4 lines

-- Steve

agree, my main concern when reporting is to make sure that it's a real issue, and not a mistake like this one.

Well, it can be really hard to filter out whether a problem is somehow self-inflicted or caused by the kernel proper - thus we generally prefer over-reporting over under-reporting.

Quoted 7 lines

Even if it's self-inflicted it's useful on a meta level: if many people report it then we might end up adding more safety guards to disable a particular common pattern of shoot-foot-self incidents.

Ingo

CONFIG_INGO=y

Thanks man, yeah I agree the more over reporting the better. but in this case I did shoot myself in the foot(not by choice though). In any case I really appreciate all of you guys
taking the time to look at this. I will try and be more careful the next time in reporting.

Justin P. Mattock

#55 lore raw
In reply to Steven Rostedt
Quoted 82 lines · Steven Rostedt
Quoted 38 lines

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +0 -0
index a81170d..7182f03 100644

include/linux/ftrace_event.h struct ftrace_event_call { · line 130

Quoted 8 lines
include/linux/ftrace_event.h +4 -1
@@ -124,7 +124,10 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */

Is this true for i386?

I just tried this patch and it seems to work. Can you give it a try.

include/linux/ftrace_event.h +1 -1
index 4ec5e67..044b70d 100644
@@ -133,7 +133,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(void);
 	void			(*profile_disable)(void);
-};
+} __attribute__((aligned(sizeof(struct list_head))));
 
 #define FTRACE_MAX_PROFILE_SIZE	2048
 
include/trace/ftrace.h +0 -1
index cc0d966..31e7637 100644
@@ -501,7 +501,6 @@ static void ftrace_profile_disable_##call(void)				\
  * }
  *
  * static struct ftrace_event_call __used
- * __attribute__((__aligned__(4)))
  * __attribute__((section("_ftrace_events"))) event_<call> = {
  *	.name			= "<call>",
  *	.system			= "<system>",

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 623

Quoted 7 lines
include/trace/ftrace.h +0 -1
@@ -619,7 +618,6 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\

indeed your patch works as well for me, its much cleaner!

However, I want to make sure this fix is sufficient and is the best way to address this type of issue in general. For example, I know tracepoints are using the aligned attribute in all 3 places -> definition, usage, and linker alignment. (adding Mathieu to 'cc list). Is just the definition 'aligned' sufficient? Also, once we find a method for solving these issues in general, we need to review all users of this kind of technique to make sure they are consistent. I also think your patch above needs to add a comment to say what its doing.

thanks,

-Jason

#56 lore raw
Quoted 55 lines · * Jason Baron ([email protected]) wrote:
Quoted 54 lines · Steven Rostedt
Quoted 38 lines

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +0 -0
index a81170d..7182f03 100644

include/linux/ftrace_event.h struct ftrace_event_call { · line 130

Quoted 8 lines
include/linux/ftrace_event.h +4 -1
@@ -124,7 +124,10 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */

Is this true for i386?

I just tried this patch and it seems to work. Can you give it a try.

Quoted 10 lines

include/linux/ftrace_event.h struct ftrace_event_call { · line 136

Quoted 10 lines
include/linux/ftrace_event.h +1 -1
index 4ec5e67..044b70d 100644
@@ -133,7 +133,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(void);
 	void			(*profile_disable)(void);
-};
+} __attribute__((aligned(sizeof(struct list_head))));

I don't like that.

Basically, the vmlinux.lds.h linker script must have alignment statements before each section, which match the alignment of the section structures. Failure to do so would put padding at the beginning of the section, which is definitely not working at all. I don't see how we can automatically pass sizeof(struct list_head) to a linker script :/

Mathieu

Quoted 42 lines
Quoted 18 lines
include/linux/ftrace_event.h +0 -0
 #define FTRACE_MAX_PROFILE_SIZE	2048
 
include/trace/ftrace.h +0 -1
index cc0d966..31e7637 100644
@@ -501,7 +501,6 @@ static void ftrace_profile_disable_##call(void)				\
  * }
  *
  * static struct ftrace_event_call __used
- * __attribute__((__aligned__(4)))
  * __attribute__((section("_ftrace_events"))) event_<call> = {
  *	.name			= "<call>",
  *	.system			= "<system>",

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 623

Quoted 7 lines
include/trace/ftrace.h +0 -1
@@ -619,7 +618,6 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\

indeed your patch works as well for me, its much cleaner!

However, I want to make sure this fix is sufficient and is the best way to address this type of issue in general. For example, I know tracepoints are using the aligned attribute in all 3 places -> definition, usage, and linker alignment. (adding Mathieu to 'cc list). Is just the definition 'aligned' sufficient? Also, once we find a method for solving these issues in general, we need to review all users of this kind of technique to make sure they are consistent. I also think your patch above needs to add a comment to say what its doing.

thanks,

-Jason

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#57 lore raw
Quoted 69 lines · Mathieu Desnoyers
Quoted 55 lines · * Jason Baron ([email protected]) wrote:
Quoted 54 lines · Steven Rostedt
Quoted 38 lines

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +0 -0
index a81170d..7182f03 100644

include/linux/ftrace_event.h struct ftrace_event_call { · line 130

Quoted 8 lines
include/linux/ftrace_event.h +4 -1
@@ -124,7 +124,10 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */

Is this true for i386?

I just tried this patch and it seems to work. Can you give it a try.

Quoted 10 lines

include/linux/ftrace_event.h struct ftrace_event_call { · line 136

Quoted 10 lines
include/linux/ftrace_event.h +1 -1
index 4ec5e67..044b70d 100644
@@ -133,7 +133,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(void);
 	void			(*profile_disable)(void);
-};
+} __attribute__((aligned(sizeof(struct list_head))));

I don't like that.

Quoted 5 lines

Basically, the vmlinux.lds.h linker script must have alignment statements before each section, which match the alignment of the section structures. Failure to do so would put padding at the beginning of the section, which is definitely not working at all. I don't see how we can automatically pass sizeof(struct list_head) to a linker script :/

OK, what about __attribute__((aligned((BITS_PER_LONG/8)*2)))

That should also work in the linker script as well.

With the added comment:

/*
* We must aligned by the largest item in the structure. This happens
* to be the list_head, which consists of two pointers.
*/

Quoted 39 lines

Mathieu

Quoted 36 lines
Quoted 18 lines
include/linux/ftrace_event.h +0 -0
 #define FTRACE_MAX_PROFILE_SIZE	2048
 
include/trace/ftrace.h +0 -1
index cc0d966..31e7637 100644
@@ -501,7 +501,6 @@ static void ftrace_profile_disable_##call(void)				\
  * }
  *
  * static struct ftrace_event_call __used
- * __attribute__((__aligned__(4)))
  * __attribute__((section("_ftrace_events"))) event_<call> = {
  *	.name			= "<call>",
  *	.system			= "<system>",

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 623

Quoted 7 lines
include/trace/ftrace.h +0 -1
@@ -619,7 +618,6 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\

indeed your patch works as well for me, its much cleaner!

However, I want to make sure this fix is sufficient and is the best way to address this type of issue in general. For example, I know tracepoints are using the aligned attribute in all 3 places -> definition, usage, and linker alignment. (adding Mathieu to 'cc list). Is just the definition 'aligned' sufficient? Also, once we find a method for solving these issues in general, we need to review all users of this kind of technique to make sure they are consistent. I also think your patch above needs to add a comment to say what its doing.

Yes, I forgot to add the comment. One really does belong there.

-- Steve

Quoted 7 lines
Quoted 6 lines

thanks,

-Jason

#58 lore raw
Quoted 78 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 69 lines · Mathieu Desnoyers
Quoted 55 lines · * Jason Baron ([email protected]) wrote:
Quoted 54 lines · Steven Rostedt
Quoted 38 lines

So the problem I'm seeing is an oops on boot caused by the call->system pointer deference in event_create_dir(). The 'call' variable is of type 'struct ftrace_event_call'.

What's going on is that the 'struct ftrace_event_call' is of size 168 bytes (sizeof(struct ftrace_event_call)) = 168 = 0xA8. However, in memory the structures are 16-byte aligned. Thus, the stride for walking through the pointers needs to be 176 (0xB0), but instead its 168 causing the oops.

I've only seen this issue while using gcc (GCC) 4.5.0 20090916, on a vanilla 2.6.31 kernel.

That said, I'm not sure the compiler is doing the wrong thing here. The 'struct ftrace_event_call' contains an embedded 'struct list_head' which is 16 bytes. According to the gcc docs, the aligned attribute, 'specifies a minimum alignment for the variable or structure field, measured in bytes'. Thus, at least according to the docs, gcc can increase the alignment of the 'struct ftrace_event_call', from its original specification of 4, to 16. Even in the case where we are working corectly the structures are 8-byte aligned.

Thus, I would reccommend the patch below as a preventive measure. Its the minimal patch I've found to resolve this issue. In general, if we are going to walk data structures embedded in a special elf section, I think the general rules needs to be to set the alignment to the power of two which is greater than or equal to the largest item in the structure.

thanks,

-Jason

include/linux/ftrace_event.h +0 -0
index a81170d..7182f03 100644

include/linux/ftrace_event.h struct ftrace_event_call { · line 130

Quoted 8 lines
include/linux/ftrace_event.h +4 -1
@@ -124,7 +124,10 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(struct ftrace_event_call *);
 	void			(*profile_disable)(struct ftrace_event_call *);
-};
+} __attribute__((aligned(16)));
+
+/* Align to the largest field in the data structure:
+ * sizeof(struct list_head) = 16 */

Is this true for i386?

I just tried this patch and it seems to work. Can you give it a try.

Quoted 10 lines

include/linux/ftrace_event.h struct ftrace_event_call { · line 136

Quoted 10 lines
include/linux/ftrace_event.h +1 -1
index 4ec5e67..044b70d 100644
@@ -133,7 +133,7 @@ struct ftrace_event_call {
 	atomic_t		profile_count;
 	int			(*profile_enable)(void);
 	void			(*profile_disable)(void);
-};
+} __attribute__((aligned(sizeof(struct list_head))));

I don't like that.

Quoted 5 lines

Basically, the vmlinux.lds.h linker script must have alignment statements before each section, which match the alignment of the section structures. Failure to do so would put padding at the beginning of the section, which is definitely not working at all. I don't see how we can automatically pass sizeof(struct list_head) to a linker script :/

OK, what about __attribute__((aligned((BITS_PER_LONG/8)*2)))

Quoted 9 lines

That should also work in the linker script as well.

With the added comment:

/*
* We must aligned by the largest item in the structure. This happens
* to be the list_head, which consists of two pointers.
*/

Yep, sounds good. Oddly we have to keep these in sync manually. I'd also add a comment in the C code to tell whoever want to change the size of the structure to also check the linker script.

Also adding a BUILD_BUG_ON() that checks the structure sizeof() would be a nice safety-net (this should probably be added to tracepoints too eventually).

Mathieu

Quoted 52 lines
Quoted 39 lines

Mathieu

Quoted 36 lines
Quoted 18 lines
include/linux/ftrace_event.h +0 -0
 #define FTRACE_MAX_PROFILE_SIZE	2048
 
include/trace/ftrace.h +0 -1
index cc0d966..31e7637 100644
@@ -501,7 +501,6 @@ static void ftrace_profile_disable_##call(void)				\
  * }
  *
  * static struct ftrace_event_call __used
- * __attribute__((__aligned__(4)))
  * __attribute__((section("_ftrace_events"))) event_<call> = {
  *	.name			= "<call>",
  *	.system			= "<system>",

include/trace/ftrace.h static int ftrace_raw_init_event_##call(void) \ · line 623

Quoted 7 lines
include/trace/ftrace.h +0 -1
@@ -619,7 +618,6 @@ static int ftrace_raw_init_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
-__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name			= #call,				\
 	.system			= __stringify(TRACE_SYSTEM),		\

indeed your patch works as well for me, its much cleaner!

However, I want to make sure this fix is sufficient and is the best way to address this type of issue in general. For example, I know tracepoints are using the aligned attribute in all 3 places -> definition, usage, and linker alignment. (adding Mathieu to 'cc list). Is just the definition 'aligned' sufficient? Also, once we find a method for solving these issues in general, we need to review all users of this kind of technique to make sure they are consistent. I also think your patch above needs to add a comment to say what its doing.

Yes, I forgot to add the comment. One really does belong there.

-- Steve

Quoted 7 lines
Quoted 6 lines

thanks,

-Jason

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#59 lore raw
Quoted 3 lines · Mathieu Desnoyers

Also adding a BUILD_BUG_ON() that checks the structure sizeof() would be a nice safety-net (this should probably be added to tracepoints too eventually).

Yeah, I was thinking the same thing. But that would have to be added somewhere in a C file.

-- Steve

maintainer
#60 lore raw
In reply to Justin P. Mattock
Quoted 4 lines · Justin P. Mattock

I must admit, I've been afraid of doing
a git bisect for some time now. but after learning to do a bisect, I cant see a better solution to finding a bug(biggest issue I see is having a huge .config).

http://rostedt.homelinux.com/code/streamline_config.pl

It will remove any module from your config that is not loaded. Makes compiling kernels much faster.

(also featured in LWN this week: http://lwn.net/Articles/347713/)

-- Steve

#61 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · Justin P. Mattock

I must admit, I've been afraid of doing
a git bisect for some time now. but after learning to do a bisect, I cant see a better solution to finding a bug(biggest issue I see is having a huge .config).

 http://rostedt.homelinux.com/code/streamline_config.pl

ahh noway!! finally a nice and small .config
Ill try it out.

Quoted 9 lines

It will remove any module from your config that is not loaded. Makes compiling kernels much faster.

(also featured in LWN this week: http://lwn.net/Articles/347713/)

-- Steve

Ill have to look at that feature.
(need to register).

Signature · 1 line
Justin P. Mattock