Skip to contentSkip to conversation

Lock dependency based tree report in perf lock

DISCUSSION Frederic Weisbecker 2 participants 7 messages Jan 29, 2010 – Feb 1, 2010

lore
Share & tools
lore
#0 lore raw

Hi,

Looking at the report layout we have with perf lock, it gives a cool overview of per lock waittime, acquire time, by avg/max, that's cool.

Now to complete the overview through another dimension we could have another mode on top of a lock dependency based tree:

-- 125 ms -- lock1

             |
             -- 100 ms -- lock2
             |              |
             |               -- 99 ms -- lock 3
             |              |
             |               -- 16 us -- lock 4
             |                            |
             |                            ----- [...]
             |
             --  25 ms -- lock 5

Having such view can tell you which lock is delaying another one. We could have this view by average or maximum of acquired time (waittime here would be pointless).

And we can also report the outer locking noise time for these reports.
Taking the above example:

-- 100 ms/84 us -- lock2

                   |
                   -- 99 ms -- lock 3
                   |
                   -- 16 us -- lock 4
                               |
                               ----- [...]

100 ms is the time lock2 has been acquired. Lock 3 has been acquired 99 ms and lock 4 during 16 us, which means the outer locking noise (the code executed outside children locks) is of 84 us.

This noise can give a good rough glance of the influence childs locks can have on a parent. This is somehow the child weight, this all can even be expressed using percentages, we could even reproduce the absolute/relative percentage we currently have with the callchains in perf report (-g graph for absolute, -g fractal for relative).

Or we can have a ghost child for each locks that represent the outer child locks noise:

-- parent percentage -- lock2

                        |
                        -- 99 %   -- lock 3
                        |
                        -- 0.84 % -- noise // could have its own color for quick distinguishing
                        |
                        -- 0.16 % -- lock 4
                                     |
                                     ----- [...]

May be just one tricky thing.
Say we have lock3 and lock4 that depend on lock2, it doesn't always means lock4 will always be taken after lock3, it doesn't even mean lock4 will ever be taken after lock3.

So we shouldn't have one branch per dependency but one branch per practical scenario encountered.

So imagine we have the following dependency:

lock2
|
--- lock3
|
--- lock4
|
--- lock5

But we can actually have only the following scenarios:

lock2
|
-- lock3
|
-- lock4

lock2
|
-- lock4

lock2
|
-- lock5

Then we need three branches:

-- lock2 percentage -- lock2

                        |
                        -- 60 %
                        |   |
                        |   -- 90 % -- lock 3
                        |   |
                        |    -- 9 %  -- noise
                        |   |
                        |    -- 1 %  -- lock 4
                        |
                        -- 20 %
                        |   |
                        |    -- 99 % -- lock4
                        |   |
                        |   --  1 %  -- noise
                        |
                        -- 20 %
                            |
                             -- 14 % -- lock5
                            |
                             -- 86 % -- noise

Hopefully we could have such expandable tree using ncurses, but we can probably start like callchains in perf report.

There might also be a problem with the accuracy. The max time lock2 is acquired won't always match the path where lock3 and lock4 had their max time too, but this should give, most of the time, a view close to the reality, especially once we enter non-sleepable lock branches.

Anyway, that's just an idea, not trivial I must admit.

#1 lore raw
Quoted 3 lines · Frederic Weisbecker

Anyway, that's just an idea, not trivial I must admit.

lockdep actually collects all this information, so writing it out isn't too hard.

#2 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

Anyway, that's just an idea, not trivial I must admit.

lockdep actually collects all this information, so writing it out isn't too hard.

Lockdep collects the theorical dependencies but not the practical scenarios.

Say B and C depend on A, you'll get:

A
/ \
B C

But nothing can tell you that if A is taken, B and C will always be taken. You may have different scenarios based on this dependency, which is not something that lockdep logs, right?

#3 lore raw
Quoted 16 lines · Frederic Weisbecker
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

Anyway, that's just an idea, not trivial I must admit.

lockdep actually collects all this information, so writing it out isn't too hard.

Lockdep collects the theorical dependencies but not the practical scenarios.

Say B and C depend on A, you'll get:

Quoted 7 lines

A
/ \
B C

But nothing can tell you that if A is taken, B and C will always be taken. You may have different scenarios based on this dependency, which is not something that lockdep logs, right?

Right. But we keep track of the full held lock stack, which is what was requested.

#4 lore raw
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Frederic Weisbecker
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

Anyway, that's just an idea, not trivial I must admit.

lockdep actually collects all this information, so writing it out isn't too hard.

Lockdep collects the theorical dependencies but not the practical scenarios.

Say B and C depend on A, you'll get:

A
/ \
B C

Quoted 6 lines
Quoted 3 lines

But nothing can tell you that if A is taken, B and C will always be taken. You may have different scenarios based on this dependency, which is not something that lockdep logs, right?

Right. But we keep track of the full held lock stack, which is what was requested.

Ah, I see what you mean. Yeah we have that, but it is a runtime check feature, and the representation is linear. We should not add further tracepoint based on this to get the possible scenarios of locking, we should rather deduce them from the current tracepoints we have.

Would be a bad idea to add even more overhead.

#5 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

Anyway, that's just an idea, not trivial I must admit.

lockdep actually collects all this information, so writing it out isn't too hard.

Hmm, I'm discovering the /proc/lock_stat file this evening, did not know it exist :)

Still, a tree representation can bring another dimension.

#6 lore raw
Quoted 6 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

Anyway, that's just an idea, not trivial I must admit.

Quoted 8 lines
Quoted 2 lines

lockdep actually collects all this information, so writing it out isn't too hard.

Hmm, I'm discovering the /proc/lock_stat file this evening, did not know it exist :)

Still, a tree representation can bring another dimension.

current->held_locks[i]->instance, i < current->lock_depth