Skip to contentSkip to conversation

lockdep and debug objects together are broken?

DISCUSSION Nick Piggin 4 participants 8 messages Jan 20, 2009 – Jan 21, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

I've had a problem frustrating my testing because lockdep was silently turning itself off... I patched out the code to disable lockdep after the first error, and it started showing up weird errors. kernel/fork.c:990 seemed to be the first to trigger (hard irqs disabled) from a call_usermodehelper call. Later, migration thread was reported to try to unlock rq->lock although it was holding no locks. Then init was reported to return to userspace without releasing an objectdebug hash lock.

All that went away and everything seemed to work properly with debug objects configured out.

I didn't get too far in trying to debug the problem. But it should be easy enough to reproduce (if not, I can post traces or test patches).

Thanks,
Nick

#1 lore raw
Quoted 10 lines · Nick Piggin

Hi,

I've had a problem frustrating my testing because lockdep was silently turning itself off... I patched out the code to disable lockdep after the first error, and it started showing up weird errors. kernel/fork.c:990 seemed to be the first to trigger (hard irqs disabled) from a call_usermodehelper call. Later, migration thread was reported to try to unlock rq->lock although it was holding no locks. Then init was reported to return to userspace without releasing an objectdebug hash lock.

Quoted 5 lines

All that went away and everything seemed to work properly with debug objects configured out.

I didn't get too far in trying to debug the problem. But it should be easy enough to reproduce (if not, I can post traces or test patches).

I just built a kernel with lockdep and debugobjects enabled, and everything seemed fine. I think you should post your kernel version, config, and the lockdep patch (if needed -- it didn't seem to turn itself off here).

Vegard

Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036
#2 lore raw
Quoted 14 lines · Vegard Nossum
Quoted 13 lines · Nick Piggin

Hi,

I've had a problem frustrating my testing because lockdep was silently turning itself off... I patched out the code to disable lockdep after the first error, and it started showing up weird errors. kernel/fork.c:990 seemed to be the first to trigger (hard irqs disabled) from a call_usermodehelper call. Later, migration thread was reported to try to unlock rq->lock although it was holding no locks. Then init was reported to return to userspace without releasing an objectdebug hash lock.

All that went away and everything seemed to work properly with debug objects configured out.

Quoted 7 lines
Quoted 2 lines

I didn't get too far in trying to debug the problem. But it should be easy enough to reproduce (if not, I can post traces or test patches).

I just built a kernel with lockdep and debugobjects enabled, and everything seemed fine. I think you should post your kernel version, config, and the lockdep patch (if needed -- it didn't seem to turn itself off here).

Are you sure? Ie. sysrq+D a still works properly? In that case, you wouldn't need the lockdep patch because it just prevents the feature from being switched off.

I'll have to dig a bit further, then. The annoying thing is that lockdep turns itself off at the drop of a hat (and this particular problem seems to happen without any backtraces), so it invalidates all your lockdep testing if you don't realise it has turned itself off.

Is there a way to re-arm lockdep? That would be neat.

#3 lore raw
Quoted 27 lines · Nick Piggin
Quoted 14 lines · Vegard Nossum
Quoted 13 lines · Nick Piggin

Hi,

I've had a problem frustrating my testing because lockdep was silently turning itself off... I patched out the code to disable lockdep after the first error, and it started showing up weird errors. kernel/fork.c:990 seemed to be the first to trigger (hard irqs disabled) from a call_usermodehelper call. Later, migration thread was reported to try to unlock rq->lock although it was holding no locks. Then init was reported to return to userspace without releasing an objectdebug hash lock.

All that went away and everything seemed to work properly with debug objects configured out.

Quoted 7 lines
Quoted 2 lines

I didn't get too far in trying to debug the problem. But it should be easy enough to reproduce (if not, I can post traces or test patches).

I just built a kernel with lockdep and debugobjects enabled, and everything seemed fine. I think you should post your kernel version, config, and the lockdep patch (if needed -- it didn't seem to turn itself off here).

Are you sure? Ie. sysrq+D a still works properly? In that case, you wouldn't need the lockdep patch because it just prevents the feature from being switched off.

Quoted 7 lines

I'll have to dig a bit further, then. The annoying thing is that lockdep turns itself off at the drop of a hat (and this particular problem seems to happen without any backtraces), so it invalidates all your lockdep testing if you don't realise it has turned itself off.

Is there a way to re-arm lockdep? That would be neat.

Not at the moment, and it looks somewhat complicated. All lock state freezes the moment lockdep disarms itself. That's very much a key design
element: rarely will you see any real lockdep-inflicted crash - even if it has a bug it is self-disabling itself and running for the door very efficiently.

So by the time you'd rearm, there's a lot of tasks with no proper locking state built up. We might be able to re-arm via stop_machine_run perhaps.

Ingo

#4 lore raw
Quoted 2 lines · Ingo Molnar

So by the time you'd rearm, there's a lot of tasks with no proper locking state built up. We might be able to re-arm via stop_machine_run perhaps.

Won't work either, kstopmachine only preempts everybody. We'd require something stronger.

What we need is a point where there's guaranteed no locks held, for regular tasks that would be a trip to userspace and back, but for kernel tasks that's a bit harder -- does the freezer stuff guarantee this?

Supposing we have such a point for all tasks, what you then do is wipe all lock state and rig a trigger to start tracking lock state once you passed through the point.

#5 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Ingo Molnar

So by the time you'd rearm, there's a lot of tasks with no proper locking state built up. We might be able to re-arm via stop_machine_run perhaps.

Won't work either, kstopmachine only preempts everybody. We'd require something stronger.

indeed - mutexes wont be covered.

Quoted 3 lines

What we need is a point where there's guaranteed no locks held, for regular tasks that would be a trip to userspace and back, but for kernel tasks that's a bit harder -- does the freezer stuff guarantee this?

yes, that might be doable.

Sounds very fragile though.

Ingo

#6 lore raw
In reply to Ingo Molnar
Quoted 36 lines · Ingo Molnar
Quoted 33 lines · Nick Piggin
Quoted 14 lines · Vegard Nossum
Quoted 13 lines · Nick Piggin

Hi,

I've had a problem frustrating my testing because lockdep was silently turning itself off... I patched out the code to disable lockdep after the first error, and it started showing up weird errors. kernel/fork.c:990 seemed to be the first to trigger (hard irqs disabled) from a call_usermodehelper call. Later, migration thread was reported to try to unlock rq->lock although it was holding no locks. Then init was reported to return to userspace without releasing an objectdebug hash lock.

All that went away and everything seemed to work properly with debug objects configured out.

Quoted 7 lines
Quoted 2 lines

I didn't get too far in trying to debug the problem. But it should be easy enough to reproduce (if not, I can post traces or test patches).

I just built a kernel with lockdep and debugobjects enabled, and everything seemed fine. I think you should post your kernel version, config, and the lockdep patch (if needed -- it didn't seem to turn itself off here).

Are you sure? Ie. sysrq+D a still works properly? In that case, you wouldn't need the lockdep patch because it just prevents the feature from being switched off.

I'll have to dig a bit further, then. The annoying thing is that lockdep turns itself off at the drop of a hat (and this particular problem seems to happen without any backtraces), so it invalidates all your lockdep testing if you don't realise it has turned itself off.

Quoted 7 lines
Quoted 1 line

Is there a way to re-arm lockdep? That would be neat.

Not at the moment, and it looks somewhat complicated. All lock state freezes the moment lockdep disarms itself. That's very much a key design
element: rarely will you see any real lockdep-inflicted crash - even if it has a bug it is self-disabling itself and running for the door very efficiently.

Lockdep isn't exactly for production systems though, is it? If you want to debug some problem but you have other code (that you don't have knowledge to debug) is switching it off...

Also, I'd guess that most bugs in lockdep would probably fall pretty neatly into either the "pretty harmless" or "completely take down the system" categories ;)

#7 lore raw
Quoted 45 lines · Nick Piggin
Quoted 36 lines · Ingo Molnar
Quoted 33 lines · Nick Piggin
Quoted 14 lines · Vegard Nossum
Quoted 13 lines · Nick Piggin

Hi,

I've had a problem frustrating my testing because lockdep was silently turning itself off... I patched out the code to disable lockdep after the first error, and it started showing up weird errors. kernel/fork.c:990 seemed to be the first to trigger (hard irqs disabled) from a call_usermodehelper call. Later, migration thread was reported to try to unlock rq->lock although it was holding no locks. Then init was reported to return to userspace without releasing an objectdebug hash lock.

All that went away and everything seemed to work properly with debug objects configured out.

Quoted 7 lines
Quoted 2 lines

I didn't get too far in trying to debug the problem. But it should be easy enough to reproduce (if not, I can post traces or test patches).

I just built a kernel with lockdep and debugobjects enabled, and everything seemed fine. I think you should post your kernel version, config, and the lockdep patch (if needed -- it didn't seem to turn itself off here).

Are you sure? Ie. sysrq+D a still works properly? In that case, you wouldn't need the lockdep patch because it just prevents the feature from being switched off.

I'll have to dig a bit further, then. The annoying thing is that lockdep turns itself off at the drop of a hat (and this particular problem seems to happen without any backtraces), so it invalidates all your lockdep testing if you don't realise it has turned itself off.

Quoted 7 lines
Quoted 1 line

Is there a way to re-arm lockdep? That would be neat.

Not at the moment, and it looks somewhat complicated. All lock state freezes the moment lockdep disarms itself. That's very much a key design
element: rarely will you see any real lockdep-inflicted crash - even if it has a bug it is self-disabling itself and running for the door very efficiently.

Quoted 7 lines

Lockdep isn't exactly for production systems though, is it? If you want to debug some problem but you have other code (that you don't have knowledge to debug) is switching it off...

Also, I'd guess that most bugs in lockdep would probably fall pretty neatly into either the "pretty harmless" or "completely take down the system" categories ;)

i think lockdep could be expanded into production use via code patching techniques.

So in that sense the rearm bit could be useful - it would give us a lockdep variant that would run for the first 5 minutes of uptime (where 90% of all lockdep reports trigger: lockdep maps the dependencies very quickly) - and could turn itself off after that, and patch out / disable its callbacks.

The memory footprint would still remain, but that is not nearly as much of a problem for production systems as runtime overhead.

Ingo