Skip to contentSkip to conversation

Lttng add-context perf commands gives me error

DISCUSSION Hani Nemati 3 participants 7 messages Nov 17, 2015 – Nov 22, 2015 7 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Hi,
I am trying to count last level cache misses by using lttng add-context command. I use add-context to add perf:cpu:LLC-load-misses to my channel but I see an error. There is no error for other perf events. Just two perf events including perf:cpu:LLC-load-misses and  perf:cpu:LLC-store-misses give me the following error:
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

Here is result of "uname -a" on my computer:
Linux nemati 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Here is the command that I used:
#lttng enable-channel -k test
#Kernel channel test enabled for session hh #lttng add-context -k -t perf:cpu:LLC-load-misses -c test
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

I also tried it on other machines.  <<<<<<<    /-/ /\ /\/ / /\/ E /\/\ /\ T /    >>>>>>>>

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#3 lore raw
In reply to a message not in the archive
Quoted 13 lines · Hani Nemati

Hi,
I am trying to count last level cache misses by using lttng add-context command. I use add-context to add perf:cpu:LLC-load-misses to my channel but I see an error. There is no error for other perf events. Just two perf events including perf:cpu:LLC-load-misses and perf:cpu:LLC-store-misses give me the following error:
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

Here is result of "uname -a" on my computer:
Linux nemati 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Quoted 8 lines

Here is the command that I used:
#lttng enable-channel -k test
#Kernel channel test enabled for session hh #lttng add-context -k -t perf:cpu:LLC-load-misses -c test
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

I also tried it on other machines.

It is possible that your architecture does not expose this counter to perf. Did you try using this PMC with the perf tool ?

Thanks,

Mathieu

Quoted 6 lines

<<<<<<< /-/ /\ /\/ / /\/ E /\/\ /\ T / >>>>>>>>

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#5 lore raw
In reply to a message not in the archive

There is no PERF symbolic name for LLC-load-misses but I can use raw identifier to find out Number of code read, data read that miss the LLC. Here is the command that I used.
#perf stat -e r0109 sleep 2
 Performance counter stats for 'sleep 2':

               117      r0109

Do you have any idea why I don't have the symbolic name? Is there any way to give raw identifier instead of perf:cpu:LLC-load-misses?

On Wednesday, November 18, 2015 9:45 AM, Mathieu Desnoyers <[email protected]> wrote:

----- On Nov 17, 2015, at 4:46 PM, Hani Nemati <[email protected]> wrote:

Hi,
I am trying to count last level cache misses by using lttng add-context command. I use add-context to add perf:cpu:LLC-load-misses to my channel but I see an error. There is no error for other perf events. Just two perf events including perf:cpu:LLC-load-misses and  perf:cpu:LLC-store-misses give me the following error:
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

Here is result of "uname -a" on my computer:
Linux nemati 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Here is the command that I used:
#lttng enable-channel -k test
#Kernel channel test enabled for session hh #lttng add-context -k -t perf:cpu:LLC-load-misses -c test
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

I also tried it on other machines.

It is possible that your architecture does not expose this counter to perf. Did you try using this PMC with the perf tool ?

Thanks,

Mathieu

<<<<<<<    /-/ /\ /\/ / /\/ E /\/\ /\ T /    >>>>>>>>
_______________________________________________
lttng-dev mailing list [email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#7 lore raw
In reply to a message not in the archive

Hi Mathieu,
LCC is last level cache and it is shared between CPU cores. Also, I was running several VMs in background. That's why I used sleep command. Based on Intel's manual r0109 is raw identifier for LCC-load-misses, The question is how can I fix the problem since Lttng is using perf symbolic name.

Thanks,
Hani

On Wednesday, November 18, 2015 3:19 PM, Hani Nemati <[email protected]> wrote:

There is no PERF symbolic name for LLC-load-misses but I can use raw identifier to find out Number of code read, data read that miss the LLC. Here is the command that I used.

#perf stat -e r0109 sleep 2
 Performance counter stats for 'sleep 2':

               117      r0109

Do you have any idea why I don't have the symbolic name? Is there any way to give raw identifier instead of perf:cpu:LLC-load-misses?

On Wednesday, November 18, 2015 9:45 AM, Mathieu Desnoyers <[email protected]> wrote:

----- On Nov 17, 2015, at 4:46 PM, Hani Nemati <[email protected]> wrote:

Hi,
I am trying to count last level cache misses by using lttng add-context command. I use add-context to add perf:cpu:LLC-load-misses to my channel but I see an error. There is no error for other perf events. Just two perf events including perf:cpu:LLC-load-misses and  perf:cpu:LLC-store-misses give me the following error:
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

Here is result of "uname -a" on my computer:
Linux nemati 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Here is the command that I used:
#lttng enable-channel -k test
#Kernel channel test enabled for session hh #lttng add-context -k -t perf:cpu:LLC-load-misses -c test
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

I also tried it on other machines.

It is possible that your architecture does not expose this counter to perf. Did you try using this PMC with the perf tool ?

Thanks,

Mathieu

<<<<<<<    /-/ /\ /\/ / /\/ E /\/\ /\ T /    >>>>>>>>
_______________________________________________
lttng-dev mailing list [email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#9 lore raw
In reply to a message not in the archive

Hi,

Passing raw pmc identifiers is not supported at the moment. This is a feature we would like to add to lttng next year.

Thanks,

Mathieu

Quoted 72 lines · Hani Nemati

Hi Mathieu ,
LCC is last level cache and it is shared between CPU cores. Also, I was running several VMs in background. That's why I used sleep command.
Based on Intel's manual r0109 is raw identifier for LCC-load-misses, The question is how can I fix the problem since Lttng is using perf symbolic name.

Thanks,
Hani

On Wednesday, November 18, 2015 3:19 PM, Hani Nemati <[email protected]> wrote:

There is no PERF symbolic name for LLC-load-misses but I can use raw identifier to find out Number of code read, data read that miss the LLC. Here is the command that I used.

#perf stat -e r0109 sleep 2
Performance counter stats for 'sleep 2':

117 r0109

Do you have any idea why I don't have the symbolic name? Is there any way to give raw identifier instead of perf:cpu:LLC-load-misses?

On Wednesday, November 18, 2015 9:45 AM, Mathieu Desnoyers <[email protected]> wrote:

Quoted 13 lines · Hani Nemati

Hi,
I am trying to count last level cache misses by using lttng add-context command. I use add-context to add perf:cpu:LLC-load-misses to my channel but I see an error. There is no error for other perf events. Just two perf events including perf:cpu:LLC-load-misses and perf:cpu:LLC-store-misses give me the following error:
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

Here is result of "uname -a" on my computer:
Linux nemati 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Quoted 8 lines

Here is the command that I used:
#lttng enable-channel -k test
#Kernel channel test enabled for session hh #lttng add-context -k -t perf:cpu:LLC-load-misses -c test
Error: perf:cpu:LLC-load-misses: Add kernel context failed
Warning: Some command(s) went wrong

I also tried it on other machines.

It is possible that your architecture does not expose this counter to perf. Did you try using this PMC with the perf tool ?

Thanks,

Mathieu

Quoted 8 lines

<<<<<<<
/-/ /\ /\/ / /\/ E /\/\ /\ T
/ >>>>>>>>

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 3 lines
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#11 lore raw
In reply to a message not in the archive

I’m using the liburcu system (qsbr flavor) with its list structures as well as a variant of a bonsai tree.

In my unit testing code I have a race between the rcu callback invocation thread (currently just the default one) invoking a callback it was given and the main thread tearing down data that callback depends on.

In looking at the call_rcu_thread() function, it looks like that thread sleeps for no more than ~10ms. However, even if I guarantee 20ms elapses between the last time call_rcu() is called and my teardown, I am still seeing the callback thread processing "old" call_rcu() requests. This appears correlated with the callback thread having thousands of free's to process.

In my real application I happened to have set things up so that everything associated with rcu-managed allocation is freed with call_rcu() so I'm not seeing a problem there (so far). However, I suspect the same issue could crop up with abnormal exit paths.

How do I reliably determine that all rcu callbacks registered prior to some point in time have been processed?

Thanks,

David Boles

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#13 lore raw

Re: liburcu qsbr call_rcu quiescence

In reply to a message not in the archive

Hi David,

Please create a new discussion thread when posting a question next time. It will help others following the lttng-dev archives. You will find the answer to your question below,

Quoted 13 lines · ----- On Nov 20, 2015, at 5:33 PM, David Boles (dboles) [email protected] wrote:

I’m using the liburcu system (qsbr flavor) with its list structures as well as a variant of a bonsai tree.

In my unit testing code I have a race between the rcu callback invocation thread (currently just the default one) invoking a callback it was given and the main thread tearing down data that callback depends on.

In looking at the call_rcu_thread() function, it looks like that thread sleeps for no more than ~10ms. However, even if I guarantee 20ms elapses between the last time call_rcu() is called and my teardown, I am still seeing the callback thread processing "old" call_rcu() requests. This appears correlated with the callback thread having thousands of free's to process.

Quoted 7 lines

In my real application I happened to have set things up so that everything associated with rcu-managed allocation is freed with call_rcu() so I'm not seeing a problem there (so far). However, I suspect the same issue could crop up with abnormal exit paths.

How do I reliably determine that all rcu callbacks registered prior to some point in time have been processed?

There's an API for that! :-)

See rcu_barrier(). It has been introduced in liburcu 0.8.

Ref:
https://github.com/urcu/userspace-rcu/blob/master/doc/rcu-api.md

"void rcu_barrier(void);

Wait for all call_rcu() work initiated prior to rcu_barrier() by any thread on the system to have completed before rcu_barrier() returns. rcu_barrier() should never be called from a call_rcu() thread. This function can be used, for instance, to ensure that all memory reclaim involving a shared object has completed before allowing dlclose() of this shared object to complete."

Thanks,

Mathieu

Quoted 9 lines

Thanks,

David Boles

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 8 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev