LTTng system call tracing on ARM
A message that is not in the archive (sent off-list or not yet received): 20130410084053.GB25958@hal
A message that is not in the archive (sent off-list or not yet received): CA+umk=pX-4t74XQqsP+vLsDGF1atWzUi75gmSyoVGsDkqQcB7g@mail.gmail.com
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel hacking/patching. I could send you some links if you're interested in that.
-PL
Quoted 9 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
Quoted 32 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel hacking/patching. I could send you some links if you're interested in that.
-PL
Quoted 24 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules seems
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only with kernel 3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint support is backported.
Quoted 3 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel hacking/patching. I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 26 lines
-PL
Quoted 23 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules seems to support system call tracing on ARM already since
"8f4f80e LTTng Modules ARM syscall instrumentation".But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only with kernel 3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 20130410130246.GA12801@hal
Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :
Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.html
Earlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933
I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.
Good luck!
-PL
Quoted 28 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 5 lines
-PL
Quoted 2 lines · Jan Glauber
Hi,
Quoted 24 lines
Quoted 1 line
Quoted 1 line
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): CA+umk=pPAvGqkC2pPfr6sA3sPtZroKoPegJ6gET2eBYSPp4Fhw@mail.gmail.com
Quoted 11 lines · Pierre-Luc St-Charles
Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :
Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.htmlEarlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933
Quoted 7 lines
I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.
Good luck!
Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running and ftrace already works.
Now I'm facing something hilarious, on loading the lttng-tracer module I get:
Error: could not insert module lttng-tracer.ko: Cannot allocate memory
[15912.259399] alloc_vmap_area: 2 callbacks suppressed [15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size. [15912.273895] vmalloc: allocation failure: 15991653 bytes [15912.279602] insmod: page allocation failure: order:0, mode:0xd0
Never saw vmalloc fail on loading a kernel module before. Hmmm. Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.ko
Wow. 16 megs for a kernel module ?!? readelf says its all in .rodata.
Doing an objdump on that
Disassembly of section .rodata:
00000000 <sc_table>:
...
1714: 00f0a788 rscseq sl, r0, r8, lsl #15
1718: 00f0a7a0 rscseq sl, r0, r0, lsr #15
171c: 00000004 andeq r0, r0, r4
...
f00054: 00f0a860 rscseq sl, r0, r0, ror #16
f00058: 00f0a878 rscseq sl, r0, r8, ror r8
f0005c: 00000002 andeq r0, r0, r2Now whats that? A big hole in the system call table?
Digging through the various defines and headers of lttng-modules I've found:
instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#define OVERRIDE_TABLE_32_sys_set_tls TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
Weird ARM apparently expanded the system call area above 0xf0000. I don't understand why this override logic is needed but removing that define results in a kernel module with a much more sane size.
LTTng system call tracing works with that change for me.
Any idea on how the override problem can be solved with the magic system calls?
--Jan
Quoted 58 lines
-PL
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 20130410160233.GA17568@hal
Hi Jan,
This seems related to bug #472 [1]. CCing Mathieu.
Christian
[1] - http://bugs.lttng.org/issues/472
Quoted 133 lines · Jan Glauber
Quoted 11 lines · Pierre-Luc St-Charles
Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :
Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.htmlEarlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933Quoted 7 lines
I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.
Good luck!
Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running and ftrace already works.
Now I'm facing something hilarious, on loading the lttng-tracer module I get:
Error: could not insert module lttng-tracer.ko: Cannot allocate memory
[15912.259399] alloc_vmap_area: 2 callbacks suppressed [15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size. [15912.273895] vmalloc: allocation failure: 15991653 bytes [15912.279602] insmod: page allocation failure: order:0, mode:0xd0Never saw vmalloc fail on loading a kernel module before. Hmmm. Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.koWow. 16 megs for a kernel module ?!? readelf says its all in .rodata.
Doing an objdump on that
Disassembly of section .rodata:00000000 <
sc_table>:
...1714: 00f0a788 rscseq sl, r0, r8, lsl #15 1718: 00f0a7a0 rscseq sl, r0, r0, lsr #15 171c: 00000004 andeq r0, r0, r4 ... f00054: 00f0a860 rscseq sl, r0, r0, ror #16 f00058: 00f0a878 rscseq sl, r0, r8, ror r8 f0005c: 00000002 andeq r0, r0, r2Now whats that? A big hole in the system call table?
Digging through the various defines and headers of lttng-modules I've found:
instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#defineOVERRIDE_TABLE_32_sys_set_tlsTRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)Weird ARM apparently expanded the system call area above
0xf0000. I don't understand why this override logic is needed but removing that define results in a kernel module with a much more sane size.LTTng system call tracing works with that change for me.
Any idea on how the override problem can be solved with the magic system calls?
--Jan
Quoted 4 lines
-PL
Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 60 lines · * Jan Glauber ([email protected]) wrote:
Quoted 11 lines · Pierre-Luc St-Charles
Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :
Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.htmlEarlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933Quoted 7 lines
I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.
Good luck!
Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running and ftrace already works.
Now I'm facing something hilarious, on loading the lttng-tracer module I get:
Error: could not insert module lttng-tracer.ko: Cannot allocate memory
[15912.259399] alloc_vmap_area: 2 callbacks suppressed [15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size. [15912.273895] vmalloc: allocation failure: 15991653 bytes [15912.279602] insmod: page allocation failure: order:0, mode:0xd0Never saw vmalloc fail on loading a kernel module before. Hmmm. Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.koWow. 16 megs for a kernel module ?!? readelf says its all in .rodata.
Doing an objdump on that
Disassembly of section .rodata:00000000 <
sc_table>:
...1714: 00f0a788 rscseq sl, r0, r8, lsl #15 1718: 00f0a7a0 rscseq sl, r0, r0, lsr #15 171c: 00000004 andeq r0, r0, r4 ... f00054: 00f0a860 rscseq sl, r0, r0, ror #16 f00058: 00f0a878 rscseq sl, r0, r8, ror r8 f0005c: 00000002 andeq r0, r0, r2Now whats that? A big hole in the system call table?
Digging through the various defines and headers of lttng-modules I've found:
instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#defineOVERRIDE_TABLE_32_sys_set_tlsTRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
Quoted 7 lines
Weird ARM apparently expanded the system call area above
0xf0000. I don't understand why this override logic is needed but removing that define results in a kernel module with a much more sane size.LTTng system call tracing works with that change for me.
Any idea on how the override problem can be solved with the magic system calls?
Thanks for the detailed report !!
Please try with this commit:
commit 11fa478f18241fedee86f7dc7820a91c629c9e7e
Author: Mathieu Desnoyers <[email protected]> Date: Wed Apr 10 12:14:38 2013 -0400
Fix: remove ARM set_tls system call override
We'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.
Removing this instrumentation for now. sys_set_tls will appear as sys_unknown.
Fixes #472
CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html
instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0 -16
@@ -2,7 +2,6 @@#define OVERRIDE_TABLE_32_sys_arm_fadvise64_64#define OVERRIDE_TABLE_32_sys_sync_file_range2 -#define OVERRIDE_TABLE_32_sys_set_tls#ifndef CREATE_SYSCALL_TABLE
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2, TP_printk() ) -SC_TRACE_EVENT (sys_set_tls, - TP_PROTO(unsigned int tid, unsigned long tls), -TP_ARGS (tid, tls), -TP_STRUCT__entry ( - __field(unsigned int, tid) - __field_hex(unsigned int, tls)), -TP_fast_assign ( - tp_assign(tid, tid) -tp_assign (tls, tls)), -TP_printk () -) -#else /* CREATE_SYSCALL_TABLE */ #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6) TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)#define OVERRIDE_TABLE_32_sys_sync_file_range2 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4) -#define OVERRIDE_TABLE_32_sys_set_tls -TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2) -#endif /* CREATE_SYSCALL_TABLE */
Quoted 66 lines
--Jan
Quoted 4 lines
-PL
Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): 20130410162012.GA6525@Krystal
Quoted 4 lines · Mathieu Desnoyers
Thanks for the detailed report !!
Please try with this commit:
Yes, that patch works. But I'm still curious if you can explain what the override was for ;-
--Jan
Quoted 133 lines
commit
11fa478f18241fedee86f7dc7820a91c629c9e7eAuthor: Mathieu Desnoyers <[email protected]> Date: Wed Apr 10 12:14:38 2013 -0400Fix: remove ARM
set_tlssystem call overrideWe'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.
Removing this instrumentation for now.
sys_set_tlswill appear assys_unknown.Fixes #472
CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.htmlSigned-off-by Mathieu Desnoyers <[email protected]>instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0
-16@@ -2,7 +2,6 @@#define OVERRIDE_TABLE_32_sys_arm_fadvise64_64#define OVERRIDE_TABLE_32_sys_sync_file_range2 -#define OVERRIDE_TABLE_32_sys_set_tls#ifndef CREATE_SYSCALL_TABLE @@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2, TP_printk() ) -SC_TRACE_EVENT (sys_set_tls, - TP_PROTO(unsigned int tid, unsigned long tls), -TP_ARGS (tid, tls), -TP_STRUCT__entry ( - __field(unsigned int, tid) - __field_hex(unsigned int, tls)), -TP_fast_assign ( - tp_assign(tid, tid) -tp_assign (tls, tls)), -TP_printk () -) -#else /* CREATE_SYSCALL_TABLE */ #define OVVERRIDE_TABLE_32_sys_mmap@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6) TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)#define OVERRIDE_TABLE_32_sys_sync_file_range2 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4) -#define OVERRIDE_TABLE_32_sys_set_tls -TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2) -#endif /* CREATE_SYSCALL_TABLE */ Quoted 68 lines
--Jan
Quoted 4 lines
-PL
Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-devSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): 20130410163055.GA18278@hal
Quoted 8 lines · * Jan Glauber ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers
Thanks for the detailed report !!
Please try with this commit:
Yes, that patch works. But I'm still curious if you can explain what the override was for ;-
The "override" allows defining how to serialize the information for specific system calls. It can either replace the behavior of the "semi-automated" description of the system call (brought into lttng-modules by means of the system call extractor script), or just provide the serialization description for a system call that was not available to the system call extractor.
In this case, sys_set_tls was not available for automated extraction, so the override entry has been added to it would not appear as a "sys_unknown" system call, but would rather show "sys_set_tls" along with the well-printed system call arguments.
You may want to have a look at:
lttng-modules/instrumentation/syscalls/README
for details.
If we want to handle the set_tls special-case (and same for other ARM-specific syscalls), we'll probably want to do that with a arm-specific table that will contain those system calls.
Thanks,
Mathieu
Quoted 136 lines
--Jan
Quoted 133 lines
commit
11fa478f18241fedee86f7dc7820a91c629c9e7eAuthor: Mathieu Desnoyers <[email protected]> Date: Wed Apr 10 12:14:38 2013 -0400Fix: remove ARM
set_tlssystem call overrideWe'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.
Removing this instrumentation for now.
sys_set_tlswill appear assys_unknown.Fixes #472
CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.htmlSigned-off-by Mathieu Desnoyers <[email protected]>instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0
-16@@ -2,7 +2,6 @@#define OVERRIDE_TABLE_32_sys_arm_fadvise64_64#define OVERRIDE_TABLE_32_sys_sync_file_range2 -#define OVERRIDE_TABLE_32_sys_set_tls#ifndef CREATE_SYSCALL_TABLE @@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2, TP_printk() ) -SC_TRACE_EVENT (sys_set_tls, - TP_PROTO(unsigned int tid, unsigned long tls), -TP_ARGS (tid, tls), -TP_STRUCT__entry ( - __field(unsigned int, tid) - __field_hex(unsigned int, tls)), -TP_fast_assign ( - tp_assign(tid, tid) -tp_assign (tls, tls)), -TP_printk () -) -#else /* CREATE_SYSCALL_TABLE */ #define OVVERRIDE_TABLE_32_sys_mmap@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6) TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)#define OVERRIDE_TABLE_32_sys_sync_file_range2 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4) -#define OVERRIDE_TABLE_32_sys_set_tls -TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2) -#endif /* CREATE_SYSCALL_TABLE */ Quoted 68 lines
--Jan
Quoted 4 lines
-PL
Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-devSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): 20130410163950.GA6879@Krystal
Quoted 17 lines · Mathieu Desnoyers
Quoted 8 lines · * Jan Glauber ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers
Thanks for the detailed report !!
Please try with this commit:
Yes, that patch works. But I'm still curious if you can explain what the override was for ;-
The "override" allows defining how to serialize the information for specific system calls. It can either replace the behavior of the "semi-automated" description of the system call (brought into lttng-modules by means of the system call extractor script), or just provide the serialization description for a system call that was not available to the system call extractor.
Quoted 4 lines
In this case,
sys_set_tlswas not available for automated extraction, so the override entry has been added to it would not appear as a "sys_unknown" system call, but would rather show "sys_set_tls" along with the well-printed system call arguments.
OK, thanks for explaining. So the version in instrumentation/syscalls/headers doesn't mean much, its just the starting point from where the extractor script picked up the descriptions.
So we could append recent system calls without generating new headers for lets say 3.4.
Quoted 9 lines
You may want to have a look at:
lttng-modules/instrumentation/syscalls/README
for details.
If we want to handle the
set_tlsspecial-case (and same for other ARM-specific syscalls), we'll probably want to do that with a arm-specific table that will contain those system calls.
Yes, a seperate table would solve the issue. I don't know if ARM is the only arch doing such tricks to the system call table, so it might be worth checkingt that.
Thanks,
Jan
Quoted 146 lines
Thanks,
Mathieu
Quoted 136 lines
--Jan
Quoted 133 lines
commit
11fa478f18241fedee86f7dc7820a91c629c9e7eAuthor: Mathieu Desnoyers <[email protected]> Date: Wed Apr 10 12:14:38 2013 -0400Fix: remove ARM
set_tlssystem call overrideWe'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.
Removing this instrumentation for now.
sys_set_tlswill appear assys_unknown.Fixes #472
CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.htmlSigned-off-by Mathieu Desnoyers <[email protected]>instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0
-16@@ -2,7 +2,6 @@#define OVERRIDE_TABLE_32_sys_arm_fadvise64_64#define OVERRIDE_TABLE_32_sys_sync_file_range2 -#define OVERRIDE_TABLE_32_sys_set_tls#ifndef CREATE_SYSCALL_TABLE @@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2, TP_printk() ) -SC_TRACE_EVENT (sys_set_tls, - TP_PROTO(unsigned int tid, unsigned long tls), -TP_ARGS (tid, tls), -TP_STRUCT__entry ( - __field(unsigned int, tid) - __field_hex(unsigned int, tls)), -TP_fast_assign ( - tp_assign(tid, tid) -tp_assign (tls, tls)), -TP_printk () -) -#else /* CREATE_SYSCALL_TABLE */ #define OVVERRIDE_TABLE_32_sys_mmap@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6) TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)#define OVERRIDE_TABLE_32_sys_sync_file_range2 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4) -#define OVERRIDE_TABLE_32_sys_set_tls -TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2) -#endif /* CREATE_SYSCALL_TABLE */ Quoted 68 lines
--Jan
Quoted 4 lines
-PL
Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-devSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.comSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): 20130411082033.GA2314@hal
Quoted 23 lines · * Jan Glauber ([email protected]) wrote:
Quoted 17 lines · Mathieu Desnoyers
Quoted 8 lines · * Jan Glauber ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers
Thanks for the detailed report !!
Please try with this commit:
Yes, that patch works. But I'm still curious if you can explain what the override was for ;-
The "override" allows defining how to serialize the information for specific system calls. It can either replace the behavior of the "semi-automated" description of the system call (brought into lttng-modules by means of the system call extractor script), or just provide the serialization description for a system call that was not available to the system call extractor.
Quoted 4 lines
In this case,
sys_set_tlswas not available for automated extraction, so the override entry has been added to it would not appear as a "sys_unknown" system call, but would rather show "sys_set_tls" along with the well-printed system call arguments.
Quoted 6 lines
OK, thanks for explaining. So the version in instrumentation/syscalls/headers doesn't mean much, its just the starting point from where the extractor script picked up the descriptions.
So we could append recent system calls without generating new headers for lets say 3.4.
Yes, this is one option.
The other option, if we want to upgrade a system call table from e.g. 2.6.38 to 3.4, is to just replace the automatically generated header by headers generated by extracting system calls from a 3.4 kernel. This will keep the overrides in places, because they are located into a separate override file.
Therefore, all work done by hand in addition to the automatically generated file does not get wasted between updates.
Quoted 7 lines
Quoted 7 lines
You may want to have a look at:
lttng-modules/instrumentation/syscalls/README
for details.
Quoted 7 lines
Quoted 3 lines
If we want to handle the
set_tlsspecial-case (and same for other ARM-specific syscalls), we'll probably want to do that with a arm-specific table that will contain those system calls.Yes, a seperate table would solve the issue. I don't know if ARM is the only arch doing such tricks to the system call table, so it might be worth checkingt that.
Indeed. This would probably be 2.3 material. I'm happy with showing set_tls as sys_unknown for the moment.
Thanks!
Mathieu
Quoted 150 lines
Thanks,
JanQuoted 146 lines
Thanks,
Mathieu
Quoted 136 lines
--Jan
Quoted 133 lines
commit
11fa478f18241fedee86f7dc7820a91c629c9e7eAuthor: Mathieu Desnoyers <[email protected]> Date: Wed Apr 10 12:14:38 2013 -0400Fix: remove ARM
set_tlssystem call overrideWe'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.
Removing this instrumentation for now.
sys_set_tlswill appear assys_unknown.Fixes #472
CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.htmlSigned-off-by Mathieu Desnoyers <[email protected]>instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0
-16@@ -2,7 +2,6 @@#define OVERRIDE_TABLE_32_sys_arm_fadvise64_64#define OVERRIDE_TABLE_32_sys_sync_file_range2 -#define OVERRIDE_TABLE_32_sys_set_tls#ifndef CREATE_SYSCALL_TABLE @@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2, TP_printk() ) -SC_TRACE_EVENT (sys_set_tls, - TP_PROTO(unsigned int tid, unsigned long tls), -TP_ARGS (tid, tls), -TP_STRUCT__entry ( - __field(unsigned int, tid) - __field_hex(unsigned int, tls)), -TP_fast_assign ( - tp_assign(tid, tid) -tp_assign (tls, tls)), -TP_printk () -) -#else /* CREATE_SYSCALL_TABLE */ #define OVVERRIDE_TABLE_32_sys_mmap@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6) TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)#define OVERRIDE_TABLE_32_sys_sync_file_range2 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4) -#define OVERRIDE_TABLE_32_sys_set_tls -TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2) -#endif /* CREATE_SYSCALL_TABLE */ Quoted 68 lines
--Jan
Quoted 4 lines
-PL
Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC
Hey Jan,
I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.
OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.Quoted 2 lines
There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel
hacking/patching.
Quoted 1 line
I could send you some links if you're interested in that.
Yes, sure!
--Jan
Quoted 6 lines
-PL
Quoted 3 lines · Jan Glauber
Hi,
I want to use LTTng for system call tracing on ARM. Now lttng-modules
seems
Quoted 5 lines
Quoted 5 lines
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".
But I wonder how that worked since
lttng-syscalls.cis only build underCONFIG_HAVE_SYSCALL_TRACEPOINTSand that was added to ARM only withkernel
Quoted 15 lines
Quoted 15 lines
3.6
(much after than the lttng-modules commit).Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?thanks,
Jan_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-devSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.comSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com