Skip to contentSkip to conversation

: x86 AMD 64

oops -tip Jaswinder Singh Rajput 7 participants 31 messages Mar 18, 2009 – Mar 23, 2009 Tracing

lore
Share & tools
lore
#1 lore raw

oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

Quoted 6 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

Quoted 7 lines

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

Ingo

#2 lore raw

Re: oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

Quoted 17 lines · Ingo Molnar
Quoted 6 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

Quoted 7 lines

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

<freeze>

Quoted 7 lines

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

Ingo

Darn, ok I test this one.

#3 lore raw

Re: oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

In reply to Ingo Molnar
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 8 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

This fixed the oops for me, Is this looks OK to you:

Subject: [PATCH] x86: tracepoint.c fix oops

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
PGD 13d5fb067 PUD 13d688067 PMD 0
Oops: 0000 [#1] SMP
---
 kernel/tracepoint.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/tracepoint.c +3 -0
index 7960274..80d1353 100644
@@ -280,6 +280,8 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
+		if (!iter)
+			goto out;
 		mark_entry = get_tracepoint(iter->name);
 		if (mark_entry) {
 			set_tracepoint(&mark_entry, iter,
@@ -288,6 +290,7 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 			disable_tracepoint(iter);
 		}
 	}
+out:
 	mutex_unlock(&tracepoints_mutex);
 }
 
Signature · 1 line
1.6.0.6
#4 lore raw

Re: oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

Quoted 52 lines · Jaswinder Singh Rajput
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 8 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

This fixed the oops for me, Is this looks OK to you:

Subject: [PATCH] x86: tracepoint.c fix oops

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
PGD 13d5fb067 PUD 13d688067 PMD 0
Oops: 0000 [#1] SMP
---
 kernel/tracepoint.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/tracepoint.c +2 -0
index 7960274..80d1353 100644
@@ -280,6 +280,8 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
+		if (!iter)
+			goto out;
 		mark_entry = get_tracepoint(iter->name);
 		if (mark_entry) {
 			set_tracepoint(&mark_entry, iter,

kernel/tracepoint.c void tracepoint_update_probe_range(struct tracepoint *begin, · line 294

Quoted 5 lines
kernel/tracepoint.c +1 -0
@@ -288,6 +290,7 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 			disable_tracepoint(iter);
 		}
 	}
+out:
 	mutex_unlock(&tracepoints_mutex);

hm, that does not look right - how can 'iter' become zero? It's a text-ish symbol.

I think what we might have here is a corruption of the tracepoint data structures.

Ingo

#5 lore raw

Re: oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

In reply to Jaswinder Singh Rajput
Quoted 52 lines · Jaswinder Singh Rajput
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 8 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

This fixed the oops for me, Is this looks OK to you:

Subject: [PATCH] x86: tracepoint.c fix oops

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
PGD 13d5fb067 PUD 13d688067 PMD 0
Oops: 0000 [#1] SMP
---
 kernel/tracepoint.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/tracepoint.c +2 -0
index 7960274..80d1353 100644
@@ -280,6 +280,8 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
+		if (!iter)
+			goto out;
 		mark_entry = get_tracepoint(iter->name);
 		if (mark_entry) {
 			set_tracepoint(&mark_entry, iter,

kernel/tracepoint.c void tracepoint_update_probe_range(struct tracepoint *begin, · line 295

Quoted 6 lines
kernel/tracepoint.c +1 -0
@@ -288,6 +290,7 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 			disable_tracepoint(iter);
 		}
 	}
+out:
 	mutex_unlock(&tracepoints_mutex);
 }

Ok, it should fix the crash.
But I think the real problem remains: iter is not supposed to point to NULL, this is a section range:

tracepoint_update_probe_range(__start___tracepoints,

		__stop___tracepoints);

It seems to mean that the section is empty.

Quoted 5 lines
Signature · 1 line
1.6.0.6
#6 lore raw

Re: oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

Quoted 65 lines · Frederic Weisbecker
Quoted 52 lines · Jaswinder Singh Rajput
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 8 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

This fixed the oops for me, Is this looks OK to you:

Subject: [PATCH] x86: tracepoint.c fix oops

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
PGD 13d5fb067 PUD 13d688067 PMD 0
Oops: 0000 [#1] SMP
---
 kernel/tracepoint.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/tracepoint.c +2 -0
index 7960274..80d1353 100644
@@ -280,6 +280,8 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
+		if (!iter)
+			goto out;
 		mark_entry = get_tracepoint(iter->name);
 		if (mark_entry) {
 			set_tracepoint(&mark_entry, iter,

kernel/tracepoint.c void tracepoint_update_probe_range(struct tracepoint *begin, · line 295

Quoted 6 lines
kernel/tracepoint.c +1 -0
@@ -288,6 +290,7 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 			disable_tracepoint(iter);
 		}
 	}
+out:
 	mutex_unlock(&tracepoints_mutex);
 }

Ok, it should fix the crash.
But I think the real problem remains: iter is not supposed to point to NULL, this is a section range:

Quoted 4 lines

tracepoint_update_probe_range(__start___tracepoints,

		__stop___tracepoints);

It seems to mean that the section is empty.

OK - so checking for !iter on entry and emitting a WARN_ONCE in that case ought to change the crash for a warning, right?

Ingo

#7 lore raw

Re: oops in tracepoint_update_probe_range() (was: Re: [oops -tip] : x86 AMD 64)

Quoted 71 lines · Ingo Molnar
Quoted 68 lines · Frederic Weisbecker
Quoted 52 lines · Jaswinder Singh Rajput
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 8 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

This fixed the oops for me, Is this looks OK to you:

Subject: [PATCH] x86: tracepoint.c fix oops

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
PGD 13d5fb067 PUD 13d688067 PMD 0
Oops: 0000 [#1] SMP
---
 kernel/tracepoint.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/tracepoint.c +2 -0
index 7960274..80d1353 100644
@@ -280,6 +280,8 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
+		if (!iter)
+			goto out;
 		mark_entry = get_tracepoint(iter->name);
 		if (mark_entry) {
 			set_tracepoint(&mark_entry, iter,

kernel/tracepoint.c void tracepoint_update_probe_range(struct tracepoint *begin, · line 295

Quoted 6 lines
kernel/tracepoint.c +1 -0
@@ -288,6 +290,7 @@ void tracepoint_update_probe_range(struct tracepoint *begin,
 			disable_tracepoint(iter);
 		}
 	}
+out:
 	mutex_unlock(&tracepoints_mutex);
 }

Ok, it should fix the crash.
But I think the real problem remains: iter is not supposed to point to NULL, this is a section range:

tracepoint_update_probe_range(__start___tracepoints,

		__stop___tracepoints);
Quoted 6 lines
Quoted 1 line

It seems to mean that the section is empty.

OK - so checking for !iter on entry and emitting a WARN_ONCE in that case ought to change the crash for a warning, right?

Ingo

The real bug is elsewhere, Jaswinder has one user of tracepoints which is blktrace.
So this section is not supposed to be empty.

But, I guess it is possible to raise via a randconfig by enabling CONFIG_TRACEPOINTS without any user of it because when a module is loaded, the section is checked without the !NULL safety.

Thus I think Jaswinder's patch could be picked without WARN or anything, because this situation can happen in a normal randconfig case.

Concerning the real bug (there is blktrace here, so this NULL is weird) I'm building Jaswinder's config, may be I will find something...

#8 lore raw

[tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Commit-ID:  7d1832698e6e422cc2cf0d80b9c50ca567e758a3
Gitweb:     http://git.kernel.org/tip/7d1832698e6e422cc2cf0d80b9c50ca567e758a3
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:30:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +7 -2
index 7960274..8bc1a06 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		goto out;
+	}
+
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
 		mark_entry = get_tracepoint(iter->name);
#9 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

Quoted 41 lines · Jaswinder Singh Rajput
Commit-ID:  7d1832698e6e422cc2cf0d80b9c50ca567e758a3
Gitweb:     http://git.kernel.org/tip/7d1832698e6e422cc2cf0d80b9c50ca567e758a3
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:30:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +1 -2
index 7960274..8bc1a06 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void

kernel/tracepoint.c static void disable_tracepoint(struct tracepoint *elem) · line 285

Quoted 10 lines
kernel/tracepoint.c +6 -0
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		goto out;
+	}
+

There is no out, it should be :

+ if (!iter) {
+ WARN_ON_ONCE(1);
+ return;
+ }
+

#11 lore raw

[tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Commit-ID:  966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Gitweb:     http://git.kernel.org/tip/966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:48:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +7 -2
index 7960274..dd15df9 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		return;
+	}
+
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
 		mark_entry = get_tracepoint(iter->name);
#10 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Quoted 56 lines · Jaswinder Singh Rajput
Quoted 41 lines · Jaswinder Singh Rajput
Commit-ID:  7d1832698e6e422cc2cf0d80b9c50ca567e758a3
Gitweb:     http://git.kernel.org/tip/7d1832698e6e422cc2cf0d80b9c50ca567e758a3
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:30:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +1 -2
index 7960274..8bc1a06 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void

kernel/tracepoint.c static void disable_tracepoint(struct tracepoint *elem) · line 285

Quoted 10 lines
kernel/tracepoint.c +6 -0
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		goto out;
+	}
+

There is no out, it should be :

kernel/tracepoint.c static void disable_tracepoint(struct tracepoint *elem) · line 285

Quoted 5 lines
kernel/tracepoint.c +5 -0
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		return;
+	}
+

hmm, who is setting iter :p

--
JSR

#12 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Quoted 44 lines · Jaswinder Singh Rajput
Commit-ID:  966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Gitweb:     http://git.kernel.org/tip/966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:48:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +2 -2
index 7960274..dd15df9 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;

kernel/tracepoint.c static void disable_tracepoint(struct tracepoint *elem) · line 288

Quoted 10 lines
kernel/tracepoint.c +5 -0
 	struct tracepoint_entry *mark_entry;
 
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		return;
+	}
+
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
 		mark_entry = get_tracepoint(iter->name);

my original patch was correct.

--
JSR

#18 lore raw

[tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Commit-ID:  09933a108e6730a464a1ab676c9decc11aee0edc
Gitweb:     http://git.kernel.org/tip/09933a108e6730a464a1ab676c9decc11aee0edc
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:54:39 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +7 -2
index 7960274..adf2873 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
+	if (!begin) {
+		WARN_ON_ONCE(1);
+		return;
+	}
+
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
 		mark_entry = get_tracepoint(iter->name);
#13 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Quoted 58 lines · Jaswinder Singh Rajput
Quoted 44 lines · Jaswinder Singh Rajput
Commit-ID:  966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Gitweb:     http://git.kernel.org/tip/966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:48:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +2 -2
index 7960274..dd15df9 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;

kernel/tracepoint.c static void disable_tracepoint(struct tracepoint *elem) · line 288

Quoted 10 lines
kernel/tracepoint.c +5 -0
 	struct tracepoint_entry *mark_entry;
 
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		return;
+	}
+
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
 		mark_entry = get_tracepoint(iter->name);

my original patch was correct.

Quoted 4 lines

--
JSR

Jaswinder, It's hard for me to reproduce it via your config. May be it's because I had to update it to match latest -tip tree and then I inserted some noise inside.

Could you please send me your bad vmlinux, so that I can have a first look at your elf sections and see if there is something helpful inside.

Thanks.

#14 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

Quoted 7 lines · Frederic Weisbecker

Jaswinder, It's hard for me to reproduce it via your config. May be it's because I had to update it to match latest -tip tree and then I inserted some noise inside.

Could you please send me your bad vmlinux, so that I can have a first look at your elf sections and see if there is something helpful inside.

OK I am uploading vmlinux (file size 14729799) and dmesg on :

http://userweb.kernel.org/~jaswinder/oops_20090318/

Are you testing on 32 bit or 64 bit machine.

I am getting this error on 64 bit AMD box.

Thanks
--
JSR

#19 lore raw

[tip:tracing/ftrace] tracepoints: dont update zero-sized tracepoint sections

In reply to Jaswinder Singh Rajput
Commit-ID:  ec625cb29e66824f7ce41082617aeb93fa4e42e2
Gitweb:     http://git.kernel.org/tip/ec625cb29e66824f7ce41082617aeb93fa4e42e2
Author:     Ingo Molnar <[email protected]>
AuthorDate: Wed, 18 Mar 2009 19:54:04 +0100
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 19:55:00 +0100

tracepoints: dont update zero-sized tracepoint sections

Zero-sized tracepoint sections can occur if tracing is enabled but no tracepoint is defined. Do not emit a warning in that case.

Cc: Frederic Weisbecker <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Jaswinder Singh Rajput <[email protected]>
LKML-Reference: <[email protected]>

---
 kernel/tracepoint.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)
kernel/tracepoint.c +1 -3
index adf2873..1ef5d3a 100644
@@ -278,10 +278,8 @@ tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
-	if (!begin) {
-		WARN_ON_ONCE(1);
+	if (!begin)
 		return;
-	}
 
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
#16 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Quoted 47 lines · Jaswinder Singh Rajput
Quoted 46 lines · Jaswinder Singh Rajput
Commit-ID:  966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Gitweb:     http://git.kernel.org/tip/966a6fdf6210e3ac8ce00b61cd1107cdf97ce744
Author:     Jaswinder Singh Rajput <[email protected]>
AuthorDate: Wed, 18 Mar 2009 22:18:56 +0530
Commit:     Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Mar 2009 18:48:43 +0100

tracing: fix oops in tracepoint_update_probe_range()

Change this crash:

 BUG: unable to handle kernel NULL pointer dereference at (null)
 IP: [<ffffffff8107d4de>] tracepoint_update_probe_range+0x1f/0x9b
 PGD 13d5fb067 PUD 13d688067 PMD 0
 Oops: 0000 [#1] SMP

To a more debuggable WARN_ONCE().

Cc Frederic Weisbecker <[email protected]>
Cc Steven Rostedt <[email protected]>

LKML-Reference: <[email protected]>

[ moved the check outside the lock and added a WARN_ON(). ]
---
 kernel/tracepoint.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/tracepoint.c +2 -2
index 7960274..dd15df9 100644
@@ -272,12 +272,17 @@ static void disable_tracepoint(struct tracepoint *elem)
  *
  * Updates the probe callback corresponding to a range of tracepoints.
  */
-void tracepoint_update_probe_range(struct tracepoint *begin,
-	struct tracepoint *end)
+void
+tracepoint_update_probe_range(struct tracepoint *begin, struct tracepoint *end)
 {
 	struct tracepoint *iter;
 	struct tracepoint_entry *mark_entry;
 
Quoted 10 lines

kernel/tracepoint.c static void disable_tracepoint(struct tracepoint *elem) · line 288

Quoted 8 lines
kernel/tracepoint.c +5 -0
+	if (!iter) {
+		WARN_ON_ONCE(1);
+		return;
+	}
+
 	mutex_lock(&tracepoints_mutex);
 	for (iter = begin; iter < end; iter++) {
 		mark_entry = get_tracepoint(iter->name);

my original patch was correct.

It might have worked but it was rather ugly: it took the tracepoints_mutex for no reason.

The clean fix to skip zero-sized sections early in the function, without taking any lock, and without emitting a warning.

Ingo

#17 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

Quoted 6 lines · Ingo Molnar

It might have worked but it was rather ugly: it took the tracepoints_mutex for no reason.

The clean fix to skip zero-sized sections early in the function, without taking any lock, and without emitting a warning.

Yes, now it looks better, thanks :-)

--
JSR

#15 lore raw

Re: [tip:tracing/ftrace] tracing: fix oops in tracepoint_update_probe_range()

In reply to Jaswinder Singh Rajput
Quoted 14 lines · Jaswinder Singh Rajput
Quoted 7 lines · Frederic Weisbecker

Jaswinder, It's hard for me to reproduce it via your config. May be it's because I had to update it to match latest -tip tree and then I inserted some noise inside.

Could you please send me your bad vmlinux, so that I can have a first look at your elf sections and see if there is something helpful inside.

OK I am uploading vmlinux (file size 14729799) and dmesg on :

http://userweb.kernel.org/~jaswinder/oops_20090318/

Quoted 8 lines

Are you testing on 32 bit or 64 bit machine.

I am getting this error on 64 bit AMD box.

Thanks
--
JSR

Actually I haven't tested but only looked at my elf sections and haven't seen anything weird. (I run a 64 too).

Your sections are normal too:

$ objdump -t ./vmlinux | grep __tracepoint | sort -d

ffffffff81623b00 l O __ksymtab_gpl 0000000000000010 __ksymtab___tracepoint_block_remap
ffffffff8162eeb3 l O __ksymtab_strings 0000000000000019 __kstrtab___tracepoint_block_remap
ffffffff816fb800 g .data 0000000000000000 __start___tracepoints

ffffffff816fb800 g     O .data	0000000000000020 __tracepoint_power_start
ffffffff816fb820 g     O .data	0000000000000020 __tracepoint_power_end
ffffffff816fb840 g     O .data	0000000000000020 __tracepoint_power_mark
ffffffff816fb860 g     O .data	0000000000000020 __tracepoint_sched_wait_task
ffffffff816fb880 g     O .data	0000000000000020 __tracepoint_sched_wakeup
ffffffff816fb8a0 g     O .data	0000000000000020 __tracepoint_sched_wakeup_new
ffffffff816fb8c0 g     O .data	0000000000000020 __tracepoint_sched_switch
ffffffff816fb8e0 g     O .data	0000000000000020 __tracepoint_sched_migrate_task
ffffffff816fb900 g     O .data	0000000000000020 __tracepoint_sched_process_fork
ffffffff816fb920 g     O .data	0000000000000020 __tracepoint_sched_process_free
ffffffff816fb940 g     O .data	0000000000000020 __tracepoint_sched_process_exit
ffffffff816fb960 g     O .data	0000000000000020 __tracepoint_sched_process_wait
ffffffff816fb980 g     O .data	0000000000000020 __tracepoint_softirq_entry
ffffffff816fb9a0 g     O .data	0000000000000020 __tracepoint_softirq_exit
ffffffff816fb9c0 g     O .data	0000000000000020 __tracepoint_sched_signal_send
ffffffff816fb9e0 g     O .data	0000000000000020 __tracepoint_workqueue_insertion
ffffffff816fba00 g     O .data	0000000000000020 __tracepoint_workqueue_execution
ffffffff816fba20 g     O .data	0000000000000020 __tracepoint_workqueue_creation
ffffffff816fba40 g     O .data	0000000000000020 __tracepoint_workqueue_destruction
ffffffff816fba60 g     O .data	0000000000000020 __tracepoint_sched_kthread_stop
ffffffff816fba80 g     O .data	0000000000000020 __tracepoint_sched_kthread_stop_ret
ffffffff816fbaa0 g     O .data	0000000000000020 __tracepoint_irq_handler_entry
ffffffff816fbac0 g     O .data	0000000000000020 __tracepoint_irq_handler_exit
ffffffff816fbae0 g     O .data	0000000000000020 __tracepoint_block_bio_bounce
ffffffff816fbb00 g     O .data	0000000000000020 __tracepoint_block_split
ffffffff816fbb20 g     O .data	0000000000000020 __tracepoint_block_rq_abort
ffffffff816fbb40 g     O .data	0000000000000020 __tracepoint_block_rq_insert
ffffffff816fbb60 g     O .data	0000000000000020 __tracepoint_block_rq_issue
ffffffff816fbb80 g     O .data	0000000000000020 __tracepoint_block_plug
ffffffff816fbba0 g     O .data	0000000000000020 __tracepoint_block_unplug_io
ffffffff816fbbc0 g     O .data	0000000000000020 __tracepoint_block_unplug_timer
ffffffff816fbbe0 g     O .data	0000000000000020 __tracepoint_block_getrq
ffffffff816fbc00 g     O .data	0000000000000020 __tracepoint_block_sleeprq
ffffffff816fbc20 g     O .data	0000000000000020 __tracepoint_block_rq_requeue
ffffffff816fbc40 g     O .data	0000000000000020 __tracepoint_block_bio_backmerge
ffffffff816fbc60 g     O .data	0000000000000020 __tracepoint_block_bio_frontmerge
ffffffff816fbc80 g     O .data	0000000000000020 __tracepoint_block_bio_queue
ffffffff816fbca0 g     O .data	0000000000000020 __tracepoint_block_rq_complete
ffffffff816fbcc0 g     O .data	0000000000000020 __tracepoint_block_remap
ffffffff816fbce0 g     O .data	0000000000000020 __tracepoint_block_bio_complete

ffffffff816fbd00 g .data 0000000000000000 __stop___tracepoints

You see? __start___tracepoints and __stop___tracepoints embrace very well a good number of tracepoints, so it's not empty.

You even have tracepoints for options that you haven't selected. It seems that if CONFIG_TRACEPOINTS is enabled, all tracepoints that are found in a built file will be configured.

And there are things such as workqueues that are always built in a kernel, so the tracepoint section is _never_ supposed to be empty if CONFIG_TRACEPOINTS=y

Something screwed up somewhere...

#20 lore raw

Re: oops in tracepoint_update_probe_range()

In reply to Ingo Molnar
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 9 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

I look up the jpg files, this oops is occurred when a new module is being loaded.

tracepoint_module_notify() is added by Mathieu Desnoyers on the suggestion of me.

tracepoint_update_probe_range() and tracepoint_module_notify() can not trigger this oops if the arguments are correct.

If @begin is NULL, @end is NULL too, it's ensued by kernel/module.c.

load_module(...):

	mod->tracepoints = section_objs(hdr, sechdrs, secstrings,
					"__tracepoints",
					sizeof(*mod->tracepoints),
					&mod->num_tracepoints);
static void *section_objs(...)
{
	unsigned int sec = find_sec(hdr, sechdrs, secstrings, name);

	/* Section 0 has sh_addr 0 and sh_size 0. */
	*num = sechdrs[sec].sh_size / object_size;
	return (void *)sechdrs[sec].sh_addr;
}

If the module has not "__tracepoints" section, find_sec() returns 0. So I think, sechdrs[0].sh_size is corrupted.

Is the following fix fixed the oops for you?
---

kernel/module.c +1 -0
index 7fa134e..2ee47ff 100644
@@ -1950,6 +1950,7 @@ static noinline struct module *load_module(void __user *umod,
 	sechdrs = (void *)hdr + hdr->e_shoff;
 	secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 	sechdrs[0].sh_addr = 0;
+	sechdrs[0].sh_size = 0;
 
 	for (i = 1; i < hdr->e_shnum; i++) {
 		if (sechdrs[i].sh_type != SHT_NOBITS
#21 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 60 lines · Lai Jiangshan
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 9 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

I look up the jpg files, this oops is occurred when a new module is being loaded.

tracepoint_module_notify() is added by Mathieu Desnoyers on the suggestion of me.

tracepoint_update_probe_range() and tracepoint_module_notify() can not trigger this oops if the arguments are correct.

If @begin is NULL, @end is NULL too, it's ensued by kernel/module.c.

load_module(...):

	mod->tracepoints = section_objs(hdr, sechdrs, secstrings,
					"__tracepoints",
					sizeof(*mod->tracepoints),
					&mod->num_tracepoints);
static void *section_objs(...)
{
	unsigned int sec = find_sec(hdr, sechdrs, secstrings, name);

	/* Section 0 has sh_addr 0 and sh_size 0. */
	*num = sechdrs[sec].sh_size / object_size;
	return (void *)sechdrs[sec].sh_addr;
}

If the module has not "__tracepoints" section, find_sec() returns 0. So I think, sechdrs[0].sh_size is corrupted.

Is the following fix fixed the oops for you?
---

kernel/module.c +0 -0
index 7fa134e..2ee47ff 100644

kernel/module.c static noinline struct module *load_module(void __user *umod, · line 1956

Quoted 7 lines
kernel/module.c +1 -0
@@ -1950,6 +1950,7 @@ static noinline struct module *load_module(void __user *umod,
 	sechdrs = (void *)hdr + hdr->e_shoff;
 	secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 	sechdrs[0].sh_addr = 0;
+	sechdrs[0].sh_size = 0;
 
 	for (i = 1; i < hdr->e_shnum; i++) {
 		if (sechdrs[i].sh_type != SHT_NOBITS

Jaswinder, could you please try the fix from Lai, but first do:

git revert ec625cb # tracepoints: dont update zero-sized tracepoint sections git revert 09933a1 # tracing: fix oops in tracepoint_update_probe_range()

?

Ingo

#22 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 67 lines · Ingo Molnar
Quoted 65 lines · Lai Jiangshan
Quoted 14 lines · Ingo Molnar
Quoted 12 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

Quoted 9 lines
Quoted 1 line

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

I look up the jpg files, this oops is occurred when a new module is being loaded.

tracepoint_module_notify() is added by Mathieu Desnoyers on the suggestion of me.

tracepoint_update_probe_range() and tracepoint_module_notify() can not trigger this oops if the arguments are correct.

If @begin is NULL, @end is NULL too, it's ensued by kernel/module.c.

load_module(...):

	mod->tracepoints = section_objs(hdr, sechdrs, secstrings,
					"__tracepoints",
					sizeof(*mod->tracepoints),
					&mod->num_tracepoints);
static void *section_objs(...)
{
	unsigned int sec = find_sec(hdr, sechdrs, secstrings, name);

	/* Section 0 has sh_addr 0 and sh_size 0. */
	*num = sechdrs[sec].sh_size / object_size;
	return (void *)sechdrs[sec].sh_addr;
}

If the module has not "__tracepoints" section, find_sec() returns 0. So I think, sechdrs[0].sh_size is corrupted.

Is the following fix fixed the oops for you?
---

kernel/module.c +1 -0
index 7fa134e..2ee47ff 100644
@@ -1950,6 +1950,7 @@ static noinline struct module *load_module(void __user *umod,
 	sechdrs = (void *)hdr + hdr->e_shoff;
 	secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 	sechdrs[0].sh_addr = 0;
+	sechdrs[0].sh_size = 0;
 
Quoted 8 lines

kernel/module.c static noinline struct module *load_module(void __user *umod, · line 1956

Quoted 2 lines
kernel/module.c +0 -0
 	for (i = 1; i < hdr->e_shnum; i++) {
 		if (sechdrs[i].sh_type != SHT_NOBITS

Jaswinder, could you please try the fix from Lai, but first do:

git revert ec625cb # tracepoints: dont update zero-sized tracepoint sections git revert 09933a1 # tracing: fix oops in tracepoint_update_probe_range()

After reverting above two commits and applying Lai's patch still gives me oops:

[    5.027136] hub 6-0:1.0: state 7 ports 3 chg 0000 evt 0000
[    5.444563] BUG: unable to handle kernel NULL pointer dereference at (null)
[    5.444906] IP: [<ffffffff8107d50a>] tracepoint_update_probe_range+0x1f/0x9b
[    5.445155] PGD 13d5a8067 PUD 13d5ea067 PMD 0 
[    5.445376] Oops: 0000 [#1] SMP 
[    5.445437] last sysfs file: /sys/class/firmware/timeout
[    5.445437] CPU 0 
[    5.445437] Modules linked in: scsi_wait_scan(+)
[    5.445437] Pid: 877, comm: modprobe Not tainted 2.6.29-rc8-tip #383 HP Pavilion dv5 Notebook PC
[    5.445437] RIP: 0010:[<ffffffff8107d50a>]  [<ffffffff8107d50a>] tracepoint_update_probe_range+0x1f/0x9b
[    5.445437] RSP: 0018:ffff88013d5ede78  EFLAGS: 00010287
[    5.445437] RAX: ffff88013d5ec000 RBX: 0000000000000000 RCX: ffffffff81650940
[    5.445437] RDX: ffffffffa0000300 RSI: 0000001400000000 RDI: ffffffff81650960
[    5.445437] RBP: ffff88013d5ede98 R08: ffffc200006799c8 R09: ffff88013d5eddb8
[    5.445437] R10: dead000000200200 R11: 6db6db6db6db6db7 R12: 00000000fffffffc
[    5.445437] R13: 0000000000000000 R14: 0000001400000000 R15: 0000000000000001
[    5.445437] FS:  00007f9d94e4f6f0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
[    5.445437] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    5.445437] CR2: 0000000000000000 CR3: 000000013d5cf000 CR4: 00000000000006a0
[    5.445437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    5.445437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    5.445437] Process modprobe (pid: 877, threadinfo ffff88013d5ec000, task ffff88013e3d2710)
[    5.445437] Stack:
[    5.445437]  0000000000000000 00000000fffffffc 0000000000000000 ffffffffa0000300
[    5.445437]  ffff88013d5edea8 ffffffff8107d5b0 ffff88013d5edee8 ffffffff8143e834
[    5.445437]  ffffffff8164e4d0 ffffffff8164e4d0 0000000000000000 0000000000000000
[    5.445437] Call Trace:
[    5.445437]  [<ffffffff8107d5b0>] tracepoint_module_notify+0x2a/0x2e
[    5.445437]  [<ffffffff8143e834>] notifier_call_chain+0x33/0x5b
[    5.445437]  [<ffffffff8105377c>] __blocking_notifier_call_chain+0x4d/0x6a
[    5.445437]  [<ffffffff810537a8>] blocking_notifier_call_chain+0xf/0x11
[    5.445437]  [<ffffffff81061c90>] sys_init_module+0x94/0x1c8
[    5.445437]  [<ffffffff8100bb2b>] system_call_fastpath+0x16/0x1b
[    5.445437] Code: e8 05 df 3b 00 31 c0 5b 41 5c c9 c3 55 48 89 e5 41 56 49 89 f6 41 55 41 54 53 48 89 fb 48 c7 c7 60 09 65 81 e8 95 e1 3b 00 eb 62 <48> 8b 3b e8 9d fa ff ff 48 85 c0 49 89 c4 74 3f 48 8b 33 48 8d 
[    5.445437] RIP  [<ffffffff8107d50a>] tracepoint_update_probe_range+0x1f/0x9b
[    5.445437]  RSP <ffff88013d5ede78>
[    5.445437] CR2: 0000000000000000
[    5.450260] ---[ end trace 20c410fa785114f0 ]---

Thanks,
--
JSR

#23 lore raw

Re: oops in tracepoint_update_probe_range()

In reply to Ingo Molnar
Quoted 20 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 17 lines · Lai Jiangshan
Quoted 16 lines · Ingo Molnar
Quoted 13 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

<freeze>

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

in a modular kernel apparently.

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

It does not matter if the core kernel __tracepoint section is there or not, what really matters here is if the problematic module has a correct module header.

Also, I wonder which module is being loaded when your crash occurs. Does this specific module contain tracepoints or not ? Maybe we could add a little printk() to show the current module being loaded before the crash ?

As Lai said, the loop should be skipped because begin and end will both be the same value when there are no tracepoints in a given module. So adding extra length check will not help anything here.

The "return if NULL" test probably works just because the specific module header mismatch we have here happen to put this field to NULL for some reason.

Mathieu

Quoted 51 lines
Quoted 35 lines

I look up the jpg files, this oops is occurred when a new module is being loaded.

tracepoint_module_notify() is added by Mathieu Desnoyers on the suggestion of me.

tracepoint_update_probe_range() and tracepoint_module_notify() can not trigger this oops if the arguments are correct.

If @begin is NULL, @end is NULL too, it's ensued by kernel/module.c.

load_module(...):

	mod->tracepoints = section_objs(hdr, sechdrs, secstrings,
					"__tracepoints",
					sizeof(*mod->tracepoints),
					&mod->num_tracepoints);
static void *section_objs(...)
{
	unsigned int sec = find_sec(hdr, sechdrs, secstrings, name);

	/* Section 0 has sh_addr 0 and sh_size 0. */
	*num = sechdrs[sec].sh_size / object_size;
	return (void *)sechdrs[sec].sh_addr;
}

If the module has not "__tracepoints" section, find_sec() returns 0. So I think, sechdrs[0].sh_size is corrupted.

Is the following fix fixed the oops for you?
---

kernel/module.c +0 -0
index 7fa134e..2ee47ff 100644

kernel/module.c static noinline struct module *load_module(void __user *umod, · line 1956

Quoted 7 lines
kernel/module.c +1 -0
@@ -1950,6 +1950,7 @@ static noinline struct module *load_module(void __user *umod,
 	sechdrs = (void *)hdr + hdr->e_shoff;
 	secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 	sechdrs[0].sh_addr = 0;
+	sechdrs[0].sh_size = 0;
 
 	for (i = 1; i < hdr->e_shnum; i++) {
 		if (sechdrs[i].sh_type != SHT_NOBITS

Jaswinder, could you please try the fix from Lai, but first do:

git revert ec625cb # tracepoints: dont update zero-sized tracepoint sections git revert 09933a1 # tracing: fix oops in tracepoint_update_probe_range()

?

Ingo

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#24 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 3 lines · Mathieu Desnoyers

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

Isn't there a module versioning that prevents such things?

-- Steve

#25 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

Isn't there a module versioning that prevents such things?

-- Steve

I think it's only enabled when CONFIG_MODVERSIONS is on, which is not the case for his config :

# CONFIG_MODVERSIONS is not set

So I might be wrong, but I suspect the compatibility check in this case would only be based on kernel version.

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#30 lore raw

Re: oops in tracepoint_update_probe_range()

In reply to Mathieu Desnoyers
Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 21 lines · Lai Jiangshan
Quoted 20 lines · Ingo Molnar
Quoted 6 lines · Jaswinder Singh Rajput

Good: f4c3c4cdb1de232
Bad : 1e08816af0bc345

Config:
http://userweb.kernel.org/~jaswinder/oops_20090318/config-hpdv5-tip-bad-20090318

Quoted 7 lines

oops:
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page1.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page2.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page3.jpg
http://userweb.kernel.org/~jaswinder/oops_20090318/oops_page4.jpg

<freeze>

Steve, Frederic - the crashes above are in:

tracepoint_update_probe_range()

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

in a modular kernel apparently.

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

yes, it was having old modules, after deleting old modules.

[root@hpdv5 linux-2.6-tip]# make modules_install
INSTALL arch/x86/kernel/test_nx.ko
INSTALL drivers/hid/hid-dummy.ko
INSTALL drivers/scsi/scsi_wait_scan.ko
DEPMOD 2.6.29-rc8-tip
[root@hpdv5 jaswinder-git]# /sbin/mkinitrd /boot/initrd-2.6-tip.img 2.6.29-rc8-tip [root@hpdv5 linux-2.6-tip]#

By following above steps, I get rid of oops, and here is my lsmod:

[jaswinder@hpdv5]$ /sbin/lsmod 
Module                  Size  Used by
[jaswinder@hpdv5]$ 

--
JSR

#26 lore raw

Re: oops in tracepoint_update_probe_range()

In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · Mathieu Desnoyers

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

Isn't there a module versioning that prevents such things?

It can be overriden. In any case, the NULL check we have there now makes sense.

Ingo

#27 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 8 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 5 lines · Steven Rostedt
Quoted 3 lines · Mathieu Desnoyers

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

Quoted 6 lines
Quoted 1 line

Isn't there a module versioning that prevents such things?

It can be overriden. In any case, the NULL check we have there now makes sense.

Ingo

Well, it duplicates the check for begin == end. Actually, if begin != end _and_ being is NULL, this should be a WARN_ON or BUG_ON, because the kernel would be trying to load a module with incompatible struct module.

Are we supposed to assume that module.c allows loading modules with incompatible struct module at all ? That sounds like we would be trying to fix up things broken by the module loader in the first place.

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#28 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 16 lines · Mathieu Desnoyers
Quoted 8 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 5 lines · Steven Rostedt
Quoted 3 lines · Mathieu Desnoyers

Jaswinder : maybe you have old modules in your /lib/modules/uname -r directory which have the correct version, but the wrong module.h header ?

Quoted 6 lines
Quoted 1 line

Isn't there a module versioning that prevents such things?

It can be overriden. In any case, the NULL check we have there now makes sense.

Ingo

Quoted 8 lines

Well, it duplicates the check for begin == end. Actually, if begin != end _and_ being is NULL, this should be a WARN_ON or BUG_ON, because the kernel would be trying to load a module with incompatible struct module.

Are we supposed to assume that module.c allows loading modules with incompatible struct module at all ? That sounds like we would be trying to fix up things broken by the module loader in the first place.

Then it should be WARN_ON, no need to lock up a box hard, and give the user in X with no serial, no idea why the box just locked up.

-- Steve

#29 lore raw

Re: oops in tracepoint_update_probe_range()

Quoted 6 lines · On Friday 20 March 2009 02:50:30 Steven Rostedt wrote:
Quoted 3 lines

Are we supposed to assume that module.c allows loading modules with incompatible struct module at all ? That sounds like we would be trying to fix up things broken by the module loader in the first place.

Then it should be WARN_ON, no need to lock up a box hard, and give the user in X with no serial, no idea why the box just locked up.

In case anyone wonders, I don't care in general (though I won't protest if
anyone else wants to put checks in their code). You can crash in all kinds of exotic ways using the wrong modules. If you are a kernel dev, modversions will often save you (eg. struct module changing). If not, the kernel version string should change. Otherwise, someone's just messing with you.

Hope that helps,
Rusty.