Skip to contentSkip to conversation

dell_rbtn - kernel panic at boot...

DISCUSSION Valdis Kletnieks 5 participants 13 messages May 22, 2015 – May 29, 2015 x86 platform drivers

lore
Share & tools
lore
#0 lore raw

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

Dell Latitude E6530, most current released BIOS.

[    1.760739] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[    1.760778] IP: [<ffffffffa79460cc>] klist_next+0x1c/0x1d0
[    1.760804] PGD 0 
[    1.760816] Oops: 0000 [#1] PREEMPT SMP 
[    1.760842] Modules linked in:
[    1.761268] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.1.0-rc4-next-20150521 #55
[    1.761295] Hardware name: Dell Inc. Latitude E6530/07Y85M, BIOS A16 09/24/2014
[    1.761320] task: ffff88022ca10010 ti: ffff88022ca14000 task.ti: ffff88022ca14000
[    1.761346] RIP: 0010:[<ffffffffa79460cc>]  [<ffffffffa79460cc>] klist_next+0x1c/0x1d0
[    1.761377] RSP: 0000:ffff88022ca17d58  EFLAGS: 00010292
[    1.761396] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffa7633580
[    1.761420] RDX: 0000000000000000 RSI: ffff88022ca17db0 RDI: 0000000000000040
[    1.761445] RBP: ffff88022ca17d98 R08: 0000000000000000 R09: 00000000000a204a
[    1.761469] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88022ca17db0
[    1.761471] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB)
[    1.761473] sd 0:0:0:0: [sda] 4096-byte physical blocks
[    1.761533] R13: 0000000000000000 R14: ffffffffa7ff466d R15: 0000000000000000
[    1.761558] FS:  0000000000000000(0000) GS:ffff88022dc80000(0000) knlGS:0000000000000000
[    1.761585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.761605] CR2: 0000000000000090 CR3: 0000000027e0b000 CR4: 00000000001406e0
[    1.761628] Stack:
[    1.761633] sd 0:0:0:0: [sda] Write Protect is off
[    1.761635] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.761671]  ffff88022ca17dd8 ffffffffa73b8929 0000000000000000
[    1.761694] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.761725]  0000000000000000
[    1.761743]  ffff88022ca17dfc ffffffffa7633580 ffffffffa7ff466d 0000000000000000
[    1.761782]  ffff88022ca17de8 ffffffffa74a202c ffffffffa7f07f10 0000000000000040
[    1.761846] Call Trace:
[    1.761871]  [<ffffffffa73b8929>] ? ___preempt_schedule+0x12/0x14
[    1.761894]  [<ffffffffa7633580>] ? rbtn_rfkill_set_block+0x10/0x10
[    1.761916]  [<ffffffffa7ff466d>] ? find_tokens+0xa2/0xa2
[    1.761938]  [<ffffffffa74a202c>] driver_for_each_device+0x6c/0xb0
[    1.761960]  [<ffffffffa7633752>] dell_rbtn_notifier_register+0x42/0xc0
[    1.761984]  [<ffffffffa7ff4981>] dell_init+0x314/0x98c
[    1.762003]  [<ffffffffa7ff466d>] ? find_tokens+0xa2/0xa2
[    1.762024]  [<ffffffffa70002b9>] do_one_initcall+0x99/0x210
[    1.762046]  [<ffffffffa7081300>] ? parse_args+0x140/0x3b0
[    1.762068]  [<ffffffffa7fb0237>] kernel_init_freeable+0x1c0/0x261
[    1.762090]  [<ffffffffa7946b90>] ? rest_init+0x140/0x140
[    1.762110]  [<ffffffffa7946b9e>] kernel_init+0xe/0xe0
[    1.762129]  [<ffffffffa795b00f>] ret_from_fork+0x3f/0x70
[    1.762149]  [<ffffffffa7946b90>] ? rest_init+0x140/0x140
[    1.762168] Code: ff ff 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fc 53 48 83 ec 18 48 8b 3f 4d 8b 6c 24 08 <4c> 8b 77 50 e8 2b 3b 01 00 4d 85 ed 0f 84 30 01 00 00 49 8b 45 
[    1.762415] RIP  [<ffffffffa79460cc>] klist_next+0x1c/0x1d0
[    1.762437]  RSP <ffff88022ca17d58>
[    1.762450] CR2: 0000000000000090
[    1.762466] ---[ end trace 24f4c0df0e74c2d0 ]---
[    1.762737] scsi 1:0:0:0: CD-ROM            MATSHITA DVD+-RW UJ8C2    1.02 PQ: 0 ANSI: 5
[    1.765506]  sda: sda1 sda2 sda3
[    1.766304] sd 0:0:0:0: [sda] Attached SCSI disk
[    1.905686] 
[    1.905686] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
[    1.909713] Kernel Offset: 0x26000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
#1 lore raw
Quoted 2 lines · Valdis Kletnieks

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Thanks.

Signature · 1 line
Dmitry
#2 lore raw
Quoted 6 lines · On Saturday 23 May 2015 00:53:16 Dmitry Torokhov wrote:

On Thu, May 21, 2015 at 7:06 PM, Valdis Kletnieks

Quoted 2 lines · <[email protected]> wrote:

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

Quoted 7 lines

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Thanks.

pr_debug() messages could be useful... but no idea if we can get them.

Is there any way to fix that dependency race condition? Could driver_attach() function call help?

Signature · 2 lines
Pali Rohár
[email protected]
#3 lore raw
Quoted 13 lines · On Saturday 23 May 2015 03:05:36 Pali Rohár wrote:
Quoted 12 lines · On Saturday 23 May 2015 00:53:16 Dmitry Torokhov wrote:

On Thu, May 21, 2015 at 7:06 PM, Valdis Kletnieks

Quoted 2 lines · <[email protected]> wrote:

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Quoted 7 lines
Quoted 1 line

Thanks.

pr_debug() messages could be useful... but no idea if we can get them.

Is there any way to fix that dependency race condition? Could driver_attach() function call help?

Or what about using late_initcall() instead module_init() for driver dell-laptop.ko? By default module_init() for static linked modules is macro for device_initcall() which is called before late_initcall(), right?

Signature · 2 lines
Pali Rohár
[email protected]
#4 lore raw
In reply to Pali Rohár
Quoted 13 lines · Pali Rohár
Quoted 12 lines · On Saturday 23 May 2015 00:53:16 Dmitry Torokhov wrote:

On Thu, May 21, 2015 at 7:06 PM, Valdis Kletnieks

Quoted 2 lines · <[email protected]> wrote:

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Quoted 6 lines
Quoted 1 line

Thanks.

pr_debug() messages could be useful... but no idea if we can get them.

Is there any way to fix that dependency race condition? Could driver_attach() function call help?

I believe you can avoid this by moving dell-rbtn earlier in the Makefile than dell-laptop - but this is fragile and a hack to resolve a dependency problem.

I suppose this is why thinkpad-acpi.c is huge, and why having separate drivers talk to eachother is discouraged.

dell-laptop seems to be the base, while dell-rbtn is the more specific of the two - which makes dell-laptop calling a dell-rbtn function which requires dell-rbtn to have been initialized prior a poor approach.

Greg, Matthew, I'm tempted to recommend this 434 line driver be rolled into dell-laptop.c. Any strong opinions?

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#5 lore raw
Quoted 2 lines · Darren Hart

Greg, Matthew, I'm tempted to recommend this 434 line driver be rolled into dell-laptop.c. Any strong opinions?

Mrm. It's slightly conceptually nasty in that one's an ACPI driver and one's calling a Dell custom interface, but I think merging them is probably the last bad answer.

Signature · 1 line
Matthew Garrett | [email protected]
#6 lore raw
Quoted 7 lines · On Monday 25 May 2015 07:01:21 Matthew Garrett wrote:
Quoted 2 lines · Darren Hart

Greg, Matthew, I'm tempted to recommend this 434 line driver be rolled into dell-laptop.c. Any strong opinions?

Mrm. It's slightly conceptually nasty in that one's an ACPI driver and one's calling a Dell custom interface, but I think merging them is probably the last bad answer.

I think merging does not fix our problem. dell laptop rfkill driver needs to be initialized after dell-rbtn acpi driver register itself.

And dell-laptop and dell-rbtn are two different devices (one dell smbios and one acpi) and it for me it sounds like bad idea too...

Signature · 2 lines
Pali Rohár
[email protected]
#7 lore raw
Quoted 11 lines · Pali Rohár
Quoted 7 lines · On Monday 25 May 2015 07:01:21 Matthew Garrett wrote:
Quoted 2 lines · Darren Hart

Greg, Matthew, I'm tempted to recommend this 434 line driver be rolled into dell-laptop.c. Any strong opinions?

Mrm. It's slightly conceptually nasty in that one's an ACPI driver and one's calling a Dell custom interface, but I think merging them is probably the last bad answer.

I think merging does not fix our problem. dell laptop rfkill driver needs to be initialized after dell-rbtn acpi driver register itself.

If they were the same driver, you could control this ordering.

Quoted 3 lines

And dell-laptop and dell-rbtn are two different devices (one dell smbios and one acpi) and it for me it sounds like bad idea too...

We all agree it's a bad idea - the point Mathew and I made was it may be the "least bad" idea (all the others may be worse).

I'm looking into this, but I don't have an easy answer for you. This one is going to take some research on your part to get to the right answer.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#8 lore raw
Quoted 6 lines · On Tuesday 26 May 2015 21:16:58 Darren Hart wrote:
Quoted 5 lines · Pali Rohár
Quoted 4 lines · On Monday 25 May 2015 07:01:21 Matthew Garrett wrote:
Quoted 2 lines · Darren Hart

Greg, Matthew, I'm tempted to recommend this 434 line driver be rolled into dell-laptop.c. Any strong opinions?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Mrm. It's slightly conceptually nasty in that one's an ACPI driver and one's calling a Dell custom interface, but I think merging them is probably the last bad answer.

I think merging does not fix our problem. dell laptop rfkill driver needs to be initialized after dell-rbtn acpi driver register itself.

If they were the same driver, you could control this ordering.

Yes, I see, you are right. I can call acpi driver register function and after that initializing dell laptop rfkill code.

Quoted 10 lines
Quoted 3 lines

And dell-laptop and dell-rbtn are two different devices (one dell smbios and one acpi) and it for me it sounds like bad idea too...

We all agree it's a bad idea - the point Mathew and I made was it may be the "least bad" idea (all the others may be worse).

I'm looking into this, but I don't have an easy answer for you. This one is going to take some research on your part to get to the right answer.

I still think that changing module_init() could work... Do you know who can help us with those _ini*() macros (and ideally answer how to do that)?

Signature · 2 lines
Pali Rohár
[email protected]
#9 lore raw
Quoted 27 lines · Pali Rohár
Quoted 6 lines · On Tuesday 26 May 2015 21:16:58 Darren Hart wrote:
Quoted 5 lines · Pali Rohár
Quoted 4 lines · On Monday 25 May 2015 07:01:21 Matthew Garrett wrote:
Quoted 2 lines · Darren Hart

Greg, Matthew, I'm tempted to recommend this 434 line driver be rolled into dell-laptop.c. Any strong opinions?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Mrm. It's slightly conceptually nasty in that one's an ACPI driver and one's calling a Dell custom interface, but I think merging them is probably the last bad answer.

I think merging does not fix our problem. dell laptop rfkill driver needs to be initialized after dell-rbtn acpi driver register itself.

If they were the same driver, you could control this ordering.

Yes, I see, you are right. I can call acpi driver register function and after that initializing dell laptop rfkill code.

Quoted 7 lines
Quoted 3 lines

And dell-laptop and dell-rbtn are two different devices (one dell smbios and one acpi) and it for me it sounds like bad idea too...

We all agree it's a bad idea - the point Mathew and I made was it may be the "least bad" idea (all the others may be worse).

Quoted 6 lines
Quoted 3 lines

I'm looking into this, but I don't have an easy answer for you. This one is going to take some research on your part to get to the right answer.

I still think that changing module_init() could work... Do you know who can help us with those _ini*() macros (and ideally answer how to do that)?

You sent the patches implementing that, I'd suggest providing complete details on what you tested to add confidence to this working.

Greg, on Cc, is likely the best one to say if this is a reasonable approach, or an abuse of the *_init APIs. I suspect he'll say it's the a hack to workaround a fundamentally flawed design.

I'd suggest spending some time thinking about how this could be written such that the individual drivers do not talk back and forth to eachother, but instead talk to a subsystem (rfkill?) in a way that can tolerate the ordering issue.

If you simply cannot reasonably avoid the ordering issue, then I suspect the least bad approach is to merge the drivers.

Greg - from a general driver development best practices perspective, would you disagree with anything I've said here?

Quick summary to save Greg the search: dell-laptop calls a function in dell-rbtn. If both are built-in, if dell-rbtn hasn't completed init yet, the dell-laptop init will crash.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#10 lore raw
In reply to Darren Hart
Quoted 16 lines · On Sunday 24 May 2015 21:44:32 Darren Hart wrote:
Quoted 15 lines · Pali Rohár
Quoted 13 lines · On Saturday 23 May 2015 00:53:16 Dmitry Torokhov wrote:

On Thu, May 21, 2015 at 7:06 PM, Valdis Kletnieks

Quoted 2 lines · <[email protected]> wrote:

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Thanks.

Quoted 8 lines
Quoted 4 lines

pr_debug() messages could be useful... but no idea if we can get them.

Is there any way to fix that dependency race condition? Could driver_attach() function call help?

I believe you can avoid this by moving dell-rbtn earlier in the Makefile than dell-laptop - but this is fragile and a hack to resolve a dependency problem.

And what about that late_initcall() instead module_init() in dell-laptop? Will it fix this problem?

Signature · 2 lines
Pali Rohár
[email protected]
#11 lore raw
Quoted 22 lines · Pali Rohár
Quoted 21 lines · On Sunday 24 May 2015 21:44:32 Darren Hart wrote:
Quoted 19 lines · Pali Rohár
Quoted 6 lines · On Saturday 23 May 2015 00:53:16 Dmitry Torokhov wrote:

On Thu, May 21, 2015 at 7:06 PM, Valdis Kletnieks

Quoted 2 lines · <[email protected]> wrote:

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

Quoted 7 lines

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Thanks.

pr_debug() messages could be useful... but no idea if we can get them.

Is there any way to fix that dependency race condition? Could driver_attach() function call help?

Quoted 7 lines
Quoted 3 lines

I believe you can avoid this by moving dell-rbtn earlier in the Makefile than dell-laptop - but this is fragile and a hack to resolve a dependency problem.

And what about that late_initcall() instead module_init() in dell-laptop? Will it fix this problem?

No, because late_initcall() for modules is module_init(). See include/linux/init.h.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#12 lore raw
Quoted 27 lines · Darren Hart
Quoted 26 lines · Pali Rohár
Quoted 24 lines · On Sunday 24 May 2015 21:44:32 Darren Hart wrote:
Quoted 13 lines · Pali Rohár
Quoted 12 lines · On Saturday 23 May 2015 00:53:16 Dmitry Torokhov wrote:

On Thu, May 21, 2015 at 7:06 PM, Valdis Kletnieks

Quoted 2 lines · <[email protected]> wrote:

So after I made both config variables =y, the resulting kernel built, but died a glorious death at boot.

I guess if both are built-in then, according to link order, dell-laptop starts first, before dell-rbtn, and dies in dell_rbtn_notifier_register() in call to driver_for_each_device(&rbtn_driver.drv, ...) because rbtn_driver has not been registered yet and thus half-initlalized.

Quoted 6 lines
Quoted 1 line

Thanks.

pr_debug() messages could be useful... but no idea if we can get them.

Is there any way to fix that dependency race condition? Could driver_attach() function call help?

I believe you can avoid this by moving dell-rbtn earlier in the Makefile than dell-laptop - but this is fragile and a hack to resolve a dependency problem.

Quoted 6 lines
Quoted 3 lines

And what about that late_initcall() instead module_init() in dell-laptop? Will it fix this problem?

No, because late_initcall() for modules is module_init(). See include/linux/init.h.

Apologies, in this context we're concerned about built-in, not module.

This might function as desired.

module_init() is defined as device_initcall (level 6) for built-in late_initcall() is level 7

There is precedent for this under drivers/ - although not in anything that stood out to me as a good exemplar. See:

b233020 Input: gpio_keys - move to late_initcall

for an example with a similar purpose.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center