Skip to contentSkip to conversation

Massive log spam loading modules on ARM after 3.2-rc5 (regression)

DISCUSSION Nick Bowler 3 participants 7 messages Dec 14, 2011 – Dec 15, 2011 1 message not in the archive Arm (32-bit)

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

[Copying LKML because linux-arm-kernel is bouncing my mails.]

Quoted 99 lines · Nick Bowler

Hi folks,

With latest Linus' master, I'm faced with massive log spam on our ARM Versatile Express platform when using modules. Here's an example:

  # modprobe sp805-wdt
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  sp805-wdt mb:wdt: registration successful
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009

The message only seem to get printed at module load time. Other than the noise, things _seem_ to be working. Nevertheless, it's a regression introduced after 3.2-rc5 by the following commit:

de66a979012dbc66b1ec0125795a3f79ee667b8a is the first bad commit commit de66a979012dbc66b1ec0125795a3f79ee667b8a
Author: Uwe Kleine-König <[email protected]>
Date: Mon Dec 5 09:39:59 2011 +0100

ARM: 7187/1: fix unwinding for XIP kernels

The linker places the unwind tables in readonly sections. So when using an XIP kernel these are located in ROM and cannot be modified. For that reason the current approach to convert the relative offsets in the unwind index to absolute addresses early in the boot process doesn't work with XIP.

The offsets in the unwind index section are signed 31 bit numbers and the structs are sorted by this offset. So it first has offsets between 0x40000000 and 0x7fffffff (i.e. the negative offsets) and then offsets between 0x00000000 and 0x3fffffff. When seperating these two blocks the numbers are sorted even when interpreting the offsets as unsigned longs.

So determine the first non-negative entry once and track that using the new origin pointer. The actual bisection can then use a plain unsigned long comparison. The only thing that makes the new bisection more complicated is that the offsets are relative to their position in the index section, so the key to search needs to be adapted accordingly in each step.

Moreover several consts are added to catch future writes and rename the member "addr" of struct unwind_idx to "addr_offset" to better match the new semantic. (This has the additional benefit of breaking eventual users at compile time to make them aware of the change.)

In my tests the new algorithm was a tad faster than the original and has the additional upside of not needing the initial conversion and so saves some boot time and it's possible to unwind even earlier.

:040000 040000 9c4363228515808d71dac84a4a16e0aa0cf4ceaa 65b80d4e51fabf0a3142a880be795f4b38f4d9fe M arch

I am *not* using XIP. Reverting this commit resolves the issue.

Let me know if you need any more info,

Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
#2 lore raw
In reply to a message not in the archive
Quoted 89 lines · Nick Bowler

Hi folks,

With latest Linus' master, I'm faced with massive log spam on our ARM Versatile Express platform when using modules. Here's an example:

  # modprobe sp805-wdt
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  sp805-wdt mb:wdt: registration successful
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009

The message only seem to get printed at module load time. Other than the noise, things _seem_ to be working. Nevertheless, it's a regression introduced after 3.2-rc5 by the following commit:

de66a979012dbc66b1ec0125795a3f79ee667b8a is the first bad commit commit de66a979012dbc66b1ec0125795a3f79ee667b8a
Author: Uwe Kleine-K?nig <[email protected]>
Date: Mon Dec 5 09:39:59 2011 +0100

ARM: 7187/1: fix unwinding for XIP kernels

The linker places the unwind tables in readonly sections. So when using an XIP kernel these are located in ROM and cannot be modified. For that reason the current approach to convert the relative offsets in the unwind index to absolute addresses early in the boot process doesn't work with XIP.

The offsets in the unwind index section are signed 31 bit numbers and the structs are sorted by this offset. So it first has offsets between 0x40000000 and 0x7fffffff (i.e. the negative offsets) and then offsets between 0x00000000 and 0x3fffffff. When seperating these two blocks the numbers are sorted even when interpreting the offsets as unsigned longs.

So determine the first non-negative entry once and track that using the new origin pointer. The actual bisection can then use a plain unsigned long comparison. The only thing that makes the new bisection more complicated is that the offsets are relative to their position in the index section, so the key to search needs to be adapted accordingly in each step.

Moreover several consts are added to catch future writes and rename the member "addr" of struct unwind_idx to "addr_offset" to better match the new semantic. (This has the additional benefit of breaking eventual users at compile time to make them aware of the change.)

In my tests the new algorithm was a tad faster than the original and has the additional upside of not needing the initial conversion and so saves some boot time and it's possible to unwind even earlier.

Quoted 8 lines

:040000 040000 9c4363228515808d71dac84a4a16e0aa0cf4ceaa 65b80d4e51fabf0a3142a880be795f4b38f4d9fe M arch

I am *not* using XIP. Reverting this commit resolves the issue.

I'd say revert that change for now, I'll take a look. (IMHO this is merge window material even though it has "fix" in the title. The addressed problem is only relevant on XIP (which isn't used much AFAIK) and is broken since 2.6.30-rc1~636^2~52^2~18 when unwinding was introduced.)

Quoted 1 line

Let me know if you need any more info,

I know where to look, I'll address you via private mail if I need some more data.

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#3 lore raw
Quoted 45 lines · Uwe Kleine-K?nig
Quoted 44 lines · Nick Bowler

The message only seem to get printed at module load time. Other than the noise, things _seem_ to be working. Nevertheless, it's a regression introduced after 3.2-rc5 by the following commit:

de66a979012dbc66b1ec0125795a3f79ee667b8a is the first bad commit commit de66a979012dbc66b1ec0125795a3f79ee667b8a
Author: Uwe Kleine-K?nig <[email protected]>
Date: Mon Dec 5 09:39:59 2011 +0100

ARM: 7187/1: fix unwinding for XIP kernels

The linker places the unwind tables in readonly sections. So when using an XIP kernel these are located in ROM and cannot be modified. For that reason the current approach to convert the relative offsets in the unwind index to absolute addresses early in the boot process doesn't work with XIP.

The offsets in the unwind index section are signed 31 bit numbers and the structs are sorted by this offset. So it first has offsets between 0x40000000 and 0x7fffffff (i.e. the negative offsets) and then offsets between 0x00000000 and 0x3fffffff. When seperating these two blocks the numbers are sorted even when interpreting the offsets as unsigned longs.

So determine the first non-negative entry once and track that using the new origin pointer. The actual bisection can then use a plain unsigned long comparison. The only thing that makes the new bisection more complicated is that the offsets are relative to their position in the index section, so the key to search needs to be adapted accordingly in each step.

Moreover several consts are added to catch future writes and rename the member "addr" of struct unwind_idx to "addr_offset" to better match the new semantic. (This has the additional benefit of breaking eventual users at compile time to make them aware of the change.)

In my tests the new algorithm was a tad faster than the original and has the additional upside of not needing the initial conversion and so saves some boot time and it's possible to unwind even earlier.

Quoted 8 lines
Quoted 3 lines

:040000 040000 9c4363228515808d71dac84a4a16e0aa0cf4ceaa 65b80d4e51fabf0a3142a880be795f4b38f4d9fe M arch

I am *not* using XIP. Reverting this commit resolves the issue.

I'd say revert that change for now, I'll take a look. (IMHO this is merge window material even though it has "fix" in the title. The addressed problem is only relevant on XIP (which isn't used much AFAIK) and is broken since 2.6.30-rc1~636^2~52^2~18 when unwinding was introduced.)

For christ sake Uwe, stop being an idiot. If that's the case then YOU NEED TO SAY which kernel you want the patch applied to, and not leave it up to chance. Nothing in that says "this should not go into -rc".

And wtf wasn't it tested with modules?

Congratulations, your incompetence will now have consequences. I'm *not* going to apply any patch from you in future for -rc without there being _several_ tested-by's from various people. So you're going to find it harder to get patches in.

#4 lore raw
In reply to a message not in the archive
Quoted 94 lines · Nick Bowler

Hi folks,

With latest Linus' master, I'm faced with massive log spam on our ARM Versatile Express platform when using modules. Here's an example:

  # modprobe sp805-wdt
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  sp805-wdt mb:wdt: registration successful
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009
  unwind: Unknown symbol address 7f802009
  unwind: Index not found 7f802009

The message only seem to get printed at module load time. Other than the noise, things _seem_ to be working. Nevertheless, it's a regression introduced after 3.2-rc5 by the following commit:

de66a979012dbc66b1ec0125795a3f79ee667b8a is the first bad commit commit de66a979012dbc66b1ec0125795a3f79ee667b8a
Author: Uwe Kleine-K?nig <[email protected]>
Date: Mon Dec 5 09:39:59 2011 +0100

ARM: 7187/1: fix unwinding for XIP kernels

The linker places the unwind tables in readonly sections. So when using an XIP kernel these are located in ROM and cannot be modified. For that reason the current approach to convert the relative offsets in the unwind index to absolute addresses early in the boot process doesn't work with XIP.

The offsets in the unwind index section are signed 31 bit numbers and the structs are sorted by this offset. So it first has offsets between 0x40000000 and 0x7fffffff (i.e. the negative offsets) and then offsets between 0x00000000 and 0x3fffffff. When seperating these two blocks the numbers are sorted even when interpreting the offsets as unsigned longs.

So determine the first non-negative entry once and track that using the new origin pointer. The actual bisection can then use a plain unsigned long comparison. The only thing that makes the new bisection more complicated is that the offsets are relative to their position in the index section, so the key to search needs to be adapted accordingly in each step.

Moreover several consts are added to catch future writes and rename the member "addr" of struct unwind_idx to "addr_offset" to better match the new semantic. (This has the additional benefit of breaking eventual users at compile time to make them aware of the change.)

In my tests the new algorithm was a tad faster than the original and has the additional upside of not needing the initial conversion and so saves some boot time and it's possible to unwind even earlier.

Quoted 5 lines

:040000 040000 9c4363228515808d71dac84a4a16e0aa0cf4ceaa 65b80d4e51fabf0a3142a880be795f4b38f4d9fe M arch

I am *not* using XIP. Reverting this commit resolves the issue.

Let me know if you need any more info,

Can you provide me your .config please? I cannot reproduce your problems with modprobe on an i.MX28 based tx28 machine.

Russell, maybe this makes you a bit less annoyed?

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#5 lore raw
Quoted 47 lines · Uwe Kleine-K?nig
Quoted 46 lines · Nick Bowler

The message only seem to get printed at module load time. Other than the noise, things _seem_ to be working. Nevertheless, it's a regression introduced after 3.2-rc5 by the following commit:

de66a979012dbc66b1ec0125795a3f79ee667b8a is the first bad commit commit de66a979012dbc66b1ec0125795a3f79ee667b8a
Author: Uwe Kleine-K?nig <[email protected]>
Date: Mon Dec 5 09:39:59 2011 +0100

ARM: 7187/1: fix unwinding for XIP kernels

The linker places the unwind tables in readonly sections. So when using an XIP kernel these are located in ROM and cannot be modified. For that reason the current approach to convert the relative offsets in the unwind index to absolute addresses early in the boot process doesn't work with XIP.

The offsets in the unwind index section are signed 31 bit numbers and the structs are sorted by this offset. So it first has offsets between 0x40000000 and 0x7fffffff (i.e. the negative offsets) and then offsets between 0x00000000 and 0x3fffffff. When seperating these two blocks the numbers are sorted even when interpreting the offsets as unsigned longs.

So determine the first non-negative entry once and track that using the new origin pointer. The actual bisection can then use a plain unsigned long comparison. The only thing that makes the new bisection more complicated is that the offsets are relative to their position in the index section, so the key to search needs to be adapted accordingly in each step.

Moreover several consts are added to catch future writes and rename the member "addr" of struct unwind_idx to "addr_offset" to better match the new semantic. (This has the additional benefit of breaking eventual users at compile time to make them aware of the change.)

In my tests the new algorithm was a tad faster than the original and has the additional upside of not needing the initial conversion and so saves some boot time and it's possible to unwind even earlier.

:040000 040000 9c4363228515808d71dac84a4a16e0aa0cf4ceaa 65b80d4e51fabf0a3142a880be795f4b38f4d9fe M arch

Quoted 7 lines
Quoted 3 lines

I am *not* using XIP. Reverting this commit resolves the issue.

Let me know if you need any more info,

Can you provide me your .config please? I cannot reproduce your problems with modprobe on an i.MX28 based tx28 machine.

Russell, maybe this makes you a bit less annoyed?

Why should it?

1. Your change is totally unaffected by whether XIP is being used or not, so this is irrelevant.
2. You plainly said in your previous message (and I quote) "IMHO this is merge window material even though it has "fix" in the title.".

The fact of the matter is that you _never_ said that you didn't consider your patch -rc material - and that alone is what I'm complaining at you about. It is _YOUR_ responsibility to indicate where a patch should be applied. You failed to do that.

Don't expect others to have a magical crystal ball to be able to see into your mind to find out where you intended your patch to be applied. It doesn't work like that. You have to state that - especially if it is a 'fix' patch but is _not_ supposed to go into -rc.

#6 lore raw

Hello Russell,

Quoted 6 lines · Russell King - ARM Linux
Quoted 4 lines

Can you provide me your .config please? I cannot reproduce your problems with modprobe on an i.MX28 based tx28 machine.

Russell, maybe this makes you a bit less annoyed?

Why should it?

If you consider my statement to relate to your "And wtf wasn't it tested with modules?" it might convince you that my testing included modules, too.

Assuming that the fix I send to Nick solves his problem, this problem only affects the last function before and first function after the .idx section and that only in approx. one out of two cases. (And additionally for me the unwinder isn't used on modprobe.) So I still think my testing was OK.

Quoted 9 lines

1. Your change is totally unaffected by whether XIP is being used or not, so this is irrelevant.
2. You plainly said in your previous message (and I quote) "IMHO this is merge window material even though it has "fix" in the title.".

The fact of the matter is that you _never_ said that you didn't consider your patch -rc material - and that alone is what I'm complaining at you about. It is _YOUR_ responsibility to indicate where a patch should be applied. You failed to do that.

I don't think that it makes sense to continue arguing here. If you have to name someone to Linus who is responsible for this regression, feel free to say my name. (Ah, I see he's on Cc: ...)

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#7 lore raw
In reply to a message not in the archive

Hi Nick,

I found a problem, can you please try the following patch:

arch/arm/kernel/unwind.c +2 -2
index 3f03fe0..00df012 100644
@@ -160,12 +160,12 @@ static const struct unwind_idx *unwind_find_origin(
 		const struct unwind_idx *start, const struct unwind_idx *stop)
 {
 	pr_debug("%s(%p, %p)\n", __func__, start, stop);
-	while (start < stop - 1) {
+	while (start < stop) {
 		const struct unwind_idx *mid = start + ((stop - start) >> 1);
 
 		if (mid->addr_offset >= 0x40000000)
 			/* negative offset */
-			start = mid;
+			start = mid + 1;
 		else
 			/* positive offset */
 			stop = mid;

I'll think a bit more about that during lunch and come up with a proper commit log if it works for you.

Thanks
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |