Skip to contentSkip to conversation

LTTng system call tracing on ARM

DISCUSSION Pierre-Luc St-Charles 4 participants 10 messages Apr 10, 2013 – Apr 11, 2013 9 messages not in the archive

lore
Share & tools
lore
#2 lore raw
In reply to a message not in the archive

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel hacking/patching. I could send you some links if you're interested in that.

-PL

Quoted 9 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

Quoted 32 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel hacking/patching. I could send you some links if you're interested in that.

-PL

Quoted 24 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules seems
to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with kernel 3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#3 lore raw
In reply to a message not in the archive
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint support is backported.

Quoted 3 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel hacking/patching. I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 26 lines

-PL

Quoted 23 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules seems to support system call tracing on ARM already since
"8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with kernel 3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#5 lore raw
In reply to a message not in the archive

Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :

Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.html

Earlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933

I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.

Good luck!

-PL

Quoted 28 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 5 lines

-PL

Quoted 2 lines · Jan Glauber

Hi,

Quoted 24 lines
Quoted 1 line
Quoted 1 line

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#7 lore raw
In reply to a message not in the archive
Quoted 11 lines · Pierre-Luc St-Charles

Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :

Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.html

Earlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933

Quoted 7 lines

I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.

Good luck!

Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running and ftrace already works.

Now I'm facing something hilarious, on loading the lttng-tracer module I get:

Error: could not insert module lttng-tracer.ko: Cannot allocate memory

[15912.259399] alloc_vmap_area: 2 callbacks suppressed
[15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size.
[15912.273895] vmalloc: allocation failure: 15991653 bytes
[15912.279602] insmod: page allocation failure: order:0, mode:0xd0

Never saw vmalloc fail on loading a kernel module before. Hmmm. Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.ko

Wow. 16 megs for a kernel module ?!? readelf says its all in .rodata.

Doing an objdump on that
Disassembly of section .rodata:

00000000 <sc_table>:
...

    1714:       00f0a788        rscseq  sl, r0, r8, lsl #15
    1718:       00f0a7a0        rscseq  sl, r0, r0, lsr #15
    171c:       00000004        andeq   r0, r0, r4
        ...
  f00054:       00f0a860        rscseq  sl, r0, r0, ror #16
  f00058:       00f0a878        rscseq  sl, r0, r8, ror r8
  f0005c:       00000002        andeq   r0, r0, r2

Now whats that? A big hole in the system call table?

Digging through the various defines and headers of lttng-modules I've found:

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#define OVERRIDE_TABLE_32_sys_set_tls TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)

Weird ARM apparently expanded the system call area above 0xf0000. I don't understand why this override logic is needed but removing that define results in a kernel module with a much more sane size.

LTTng system call tracing works with that change for me.

Any idea on how the override problem can be solved with the magic system calls?

--Jan

Quoted 58 lines

-PL

Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#9 lore raw
In reply to a message not in the archive

Hi Jan,

This seems related to bug #472 [1]. CCing Mathieu.

Christian

[1] - http://bugs.lttng.org/issues/472

Quoted 133 lines · Jan Glauber
Quoted 11 lines · Pierre-Luc St-Charles

Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :

Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.html

Earlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933

Quoted 7 lines

I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.

Good luck!

Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running and ftrace already works.

Now I'm facing something hilarious, on loading the lttng-tracer module I get:

Error: could not insert module lttng-tracer.ko: Cannot allocate memory

[15912.259399] alloc_vmap_area: 2 callbacks suppressed
[15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size.
[15912.273895] vmalloc: allocation failure: 15991653 bytes
[15912.279602] insmod: page allocation failure: order:0, mode:0xd0

Never saw vmalloc fail on loading a kernel module before. Hmmm. Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.ko

Wow. 16 megs for a kernel module ?!? readelf says its all in .rodata.

Doing an objdump on that
Disassembly of section .rodata:

00000000 <sc_table>:
...

    1714:       00f0a788        rscseq  sl, r0, r8, lsl #15
    1718:       00f0a7a0        rscseq  sl, r0, r0, lsr #15
    171c:       00000004        andeq   r0, r0, r4
        ...
  f00054:       00f0a860        rscseq  sl, r0, r0, ror #16
  f00058:       00f0a878        rscseq  sl, r0, r8, ror r8
  f0005c:       00000002        andeq   r0, r0, r2

Now whats that? A big hole in the system call table?

Digging through the various defines and headers of lttng-modules I've found:

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#define OVERRIDE_TABLE_32_sys_set_tls TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)

Weird ARM apparently expanded the system call area above 0xf0000. I don't understand why this override logic is needed but removing that define results in a kernel module with a much more sane size.

LTTng system call tracing works with that change for me.

Any idea on how the override problem can be solved with the magic system calls?

--Jan

Quoted 4 lines

-PL

Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#10 lore raw
In reply to a message not in the archive
Quoted 60 lines · * Jan Glauber ([email protected]) wrote:
Quoted 11 lines · Pierre-Luc St-Charles

Here are the links I stashed a while back about previous patching attempts to add syscall tracepoints to the ARM kernel :

Most promising:
http://www.spinics.net/lists/arm-kernel/msg166419.html

Earlier draft(s):
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/086974.html
https://lkml.org/lkml/2011/12/1/131
http://comments.gmane.org/gmane.linux.ports.arm.kernel/141933

Quoted 7 lines

I believe the first link shows the most refined patch there is out there, but it might take some minor tinkering to apply it to a different kernel version. I briefly tried to apply it to the 3.0.31 kernel, but it's a bit out of my 'tinkering' range, and I never finished it.

Good luck!

Thanks, fortunately I'm targeting 3.4. It wasn't too hard, got the kernel running and ftrace already works.

Now I'm facing something hilarious, on loading the lttng-tracer module I get:

Error: could not insert module lttng-tracer.ko: Cannot allocate memory

[15912.259399] alloc_vmap_area: 2 callbacks suppressed
[15912.264953] vmap allocation for size 15998976 failed: use vmalloc=<size> to increase size.
[15912.273895] vmalloc: allocation failure: 15991653 bytes
[15912.279602] insmod: page allocation failure: order:0, mode:0xd0

Never saw vmalloc fail on loading a kernel module before. Hmmm. Looking at the module reveals:
-rw-rw-r-- 1 jang jang 16036281 Apr 10 17:48 lttng-tracer.ko

Wow. 16 megs for a kernel module ?!? readelf says its all in .rodata.

Doing an objdump on that
Disassembly of section .rodata:

00000000 <sc_table>:
...

    1714:       00f0a788        rscseq  sl, r0, r8, lsl #15
    1718:       00f0a7a0        rscseq  sl, r0, r0, lsr #15
    171c:       00000004        andeq   r0, r0, r4
        ...
  f00054:       00f0a860        rscseq  sl, r0, r0, ror #16
  f00058:       00f0a878        rscseq  sl, r0, r8, ror r8
  f0005c:       00000002        andeq   r0, r0, r2

Now whats that? A big hole in the system call table?

Digging through the various defines and headers of lttng-modules I've found:

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h:
...
#define OVERRIDE_TABLE_32_sys_set_tls TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)

Quoted 7 lines

Weird ARM apparently expanded the system call area above 0xf0000. I don't understand why this override logic is needed but removing that define results in a kernel module with a much more sane size.

LTTng system call tracing works with that change for me.

Any idea on how the override problem can be solved with the magic system calls?

Thanks for the detailed report !!

Please try with this commit:

commit 11fa478f18241fedee86f7dc7820a91c629c9e7e

Author: Mathieu Desnoyers <[email protected]>
Date:   Wed Apr 10 12:14:38 2013 -0400

Fix: remove ARM set_tls system call override

We'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.

Removing this instrumentation for now. sys_set_tls will appear as sys_unknown.

Fixes #472

CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0 -16
index 93b8674..895370f 100644
@@ -2,7 +2,6 @@
 
 #define OVERRIDE_TABLE_32_sys_arm_fadvise64_64
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
-#define OVERRIDE_TABLE_32_sys_set_tls
 
 #ifndef CREATE_SYSCALL_TABLE
 
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2,
 	TP_printk()
 )
 
-SC_TRACE_EVENT(sys_set_tls,
-	TP_PROTO(unsigned int tid, unsigned long tls),
-	TP_ARGS(tid, tls),
-	TP_STRUCT__entry(
-		__field(unsigned int, tid)
-		__field_hex(unsigned int, tls)),
-	TP_fast_assign(
-		tp_assign(tid, tid)
-		tp_assign(tls, tls)),
-	TP_printk()
-)
-
 #else	/* CREATE_SYSCALL_TABLE */
 
 #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6)
 TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4)
-#define OVERRIDE_TABLE_32_sys_set_tls
-TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
-
 
 #endif /* CREATE_SYSCALL_TABLE */
 
Quoted 66 lines

--Jan

Quoted 4 lines

-PL

Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#12 lore raw
In reply to a message not in the archive
Quoted 4 lines · Mathieu Desnoyers

Thanks for the detailed report !!

Please try with this commit:

Yes, that patch works. But I'm still curious if you can explain what the override was for ;-

--Jan

Quoted 133 lines

commit 11fa478f18241fedee86f7dc7820a91c629c9e7e

Author: Mathieu Desnoyers <[email protected]>
Date:   Wed Apr 10 12:14:38 2013 -0400

Fix: remove ARM set_tls system call override

We'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.

Removing this instrumentation for now. sys_set_tls will appear as sys_unknown.

Fixes #472

CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0 -16
index 93b8674..895370f 100644
@@ -2,7 +2,6 @@
 
 #define OVERRIDE_TABLE_32_sys_arm_fadvise64_64
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
-#define OVERRIDE_TABLE_32_sys_set_tls
 
 #ifndef CREATE_SYSCALL_TABLE
 
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2,
 	TP_printk()
 )
 
-SC_TRACE_EVENT(sys_set_tls,
-	TP_PROTO(unsigned int tid, unsigned long tls),
-	TP_ARGS(tid, tls),
-	TP_STRUCT__entry(
-		__field(unsigned int, tid)
-		__field_hex(unsigned int, tls)),
-	TP_fast_assign(
-		tp_assign(tid, tid)
-		tp_assign(tls, tls)),
-	TP_printk()
-)
-
 #else	/* CREATE_SYSCALL_TABLE */
 
 #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6)
 TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4)
-#define OVERRIDE_TABLE_32_sys_set_tls
-TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
-
 
 #endif /* CREATE_SYSCALL_TABLE */
 
Quoted 68 lines

--Jan

Quoted 4 lines

-PL

Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#14 lore raw
In reply to a message not in the archive
Quoted 8 lines · * Jan Glauber ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers

Thanks for the detailed report !!

Please try with this commit:

Yes, that patch works. But I'm still curious if you can explain what the override was for ;-

The "override" allows defining how to serialize the information for specific system calls. It can either replace the behavior of the "semi-automated" description of the system call (brought into lttng-modules by means of the system call extractor script), or just provide the serialization description for a system call that was not available to the system call extractor.

In this case, sys_set_tls was not available for automated extraction, so the override entry has been added to it would not appear as a "sys_unknown" system call, but would rather show "sys_set_tls" along with the well-printed system call arguments.

You may want to have a look at:

lttng-modules/instrumentation/syscalls/README

for details.

If we want to handle the set_tls special-case (and same for other ARM-specific syscalls), we'll probably want to do that with a arm-specific table that will contain those system calls.

Thanks,

Mathieu

Quoted 136 lines

--Jan

Quoted 133 lines

commit 11fa478f18241fedee86f7dc7820a91c629c9e7e

Author: Mathieu Desnoyers <[email protected]>
Date:   Wed Apr 10 12:14:38 2013 -0400

Fix: remove ARM set_tls system call override

We'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.

Removing this instrumentation for now. sys_set_tls will appear as sys_unknown.

Fixes #472

CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0 -16
index 93b8674..895370f 100644
@@ -2,7 +2,6 @@
 
 #define OVERRIDE_TABLE_32_sys_arm_fadvise64_64
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
-#define OVERRIDE_TABLE_32_sys_set_tls
 
 #ifndef CREATE_SYSCALL_TABLE
 
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2,
 	TP_printk()
 )
 
-SC_TRACE_EVENT(sys_set_tls,
-	TP_PROTO(unsigned int tid, unsigned long tls),
-	TP_ARGS(tid, tls),
-	TP_STRUCT__entry(
-		__field(unsigned int, tid)
-		__field_hex(unsigned int, tls)),
-	TP_fast_assign(
-		tp_assign(tid, tid)
-		tp_assign(tls, tls)),
-	TP_printk()
-)
-
 #else	/* CREATE_SYSCALL_TABLE */
 
 #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6)
 TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4)
-#define OVERRIDE_TABLE_32_sys_set_tls
-TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
-
 
 #endif /* CREATE_SYSCALL_TABLE */
 
Quoted 68 lines

--Jan

Quoted 4 lines

-PL

Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#16 lore raw
In reply to a message not in the archive
Quoted 17 lines · Mathieu Desnoyers
Quoted 8 lines · * Jan Glauber ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers

Thanks for the detailed report !!

Please try with this commit:

Yes, that patch works. But I'm still curious if you can explain what the override was for ;-

The "override" allows defining how to serialize the information for specific system calls. It can either replace the behavior of the "semi-automated" description of the system call (brought into lttng-modules by means of the system call extractor script), or just provide the serialization description for a system call that was not available to the system call extractor.

Quoted 4 lines

In this case, sys_set_tls was not available for automated extraction, so the override entry has been added to it would not appear as a "sys_unknown" system call, but would rather show "sys_set_tls" along with the well-printed system call arguments.

OK, thanks for explaining. So the version in instrumentation/syscalls/headers doesn't mean much, its just the starting point from where the extractor script picked up the descriptions.

So we could append recent system calls without generating new headers for lets say 3.4.

Quoted 9 lines

You may want to have a look at:

lttng-modules/instrumentation/syscalls/README

for details.

If we want to handle the set_tls special-case (and same for other ARM-specific syscalls), we'll probably want to do that with a arm-specific table that will contain those system calls.

Yes, a seperate table would solve the issue. I don't know if ARM is the only arch doing such tricks to the system call table, so it might be worth checkingt that.

Thanks,
Jan

Quoted 146 lines

Thanks,

Mathieu

Quoted 136 lines

--Jan

Quoted 133 lines

commit 11fa478f18241fedee86f7dc7820a91c629c9e7e

Author: Mathieu Desnoyers <[email protected]>
Date:   Wed Apr 10 12:14:38 2013 -0400

Fix: remove ARM set_tls system call override

We'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.

Removing this instrumentation for now. sys_set_tls will appear as sys_unknown.

Fixes #472

CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0 -16
index 93b8674..895370f 100644
@@ -2,7 +2,6 @@
 
 #define OVERRIDE_TABLE_32_sys_arm_fadvise64_64
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
-#define OVERRIDE_TABLE_32_sys_set_tls
 
 #ifndef CREATE_SYSCALL_TABLE
 
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2,
 	TP_printk()
 )
 
-SC_TRACE_EVENT(sys_set_tls,
-	TP_PROTO(unsigned int tid, unsigned long tls),
-	TP_ARGS(tid, tls),
-	TP_STRUCT__entry(
-		__field(unsigned int, tid)
-		__field_hex(unsigned int, tls)),
-	TP_fast_assign(
-		tp_assign(tid, tid)
-		tp_assign(tls, tls)),
-	TP_printk()
-)
-
 #else	/* CREATE_SYSCALL_TABLE */
 
 #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6)
 TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4)
-#define OVERRIDE_TABLE_32_sys_set_tls
-TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
-
 
 #endif /* CREATE_SYSCALL_TABLE */
 
Quoted 68 lines

--Jan

Quoted 4 lines

-PL

Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#18 lore raw
In reply to a message not in the archive
Quoted 23 lines · * Jan Glauber ([email protected]) wrote:
Quoted 17 lines · Mathieu Desnoyers
Quoted 8 lines · * Jan Glauber ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers

Thanks for the detailed report !!

Please try with this commit:

Yes, that patch works. But I'm still curious if you can explain what the override was for ;-

The "override" allows defining how to serialize the information for specific system calls. It can either replace the behavior of the "semi-automated" description of the system call (brought into lttng-modules by means of the system call extractor script), or just provide the serialization description for a system call that was not available to the system call extractor.

Quoted 4 lines

In this case, sys_set_tls was not available for automated extraction, so the override entry has been added to it would not appear as a "sys_unknown" system call, but would rather show "sys_set_tls" along with the well-printed system call arguments.

Quoted 6 lines

OK, thanks for explaining. So the version in instrumentation/syscalls/headers doesn't mean much, its just the starting point from where the extractor script picked up the descriptions.

So we could append recent system calls without generating new headers for lets say 3.4.

Yes, this is one option.

The other option, if we want to upgrade a system call table from e.g. 2.6.38 to 3.4, is to just replace the automatically generated header by headers generated by extracting system calls from a 3.4 kernel. This will keep the overrides in places, because they are located into a separate override file.

Therefore, all work done by hand in addition to the automatically generated file does not get wasted between updates.

Quoted 7 lines
Quoted 7 lines

You may want to have a look at:

lttng-modules/instrumentation/syscalls/README

for details.

Quoted 7 lines
Quoted 3 lines

If we want to handle the set_tls special-case (and same for other ARM-specific syscalls), we'll probably want to do that with a arm-specific table that will contain those system calls.

Yes, a seperate table would solve the issue. I don't know if ARM is the only arch doing such tricks to the system call table, so it might be worth checkingt that.

Indeed. This would probably be 2.3 material. I'm happy with showing set_tls as sys_unknown for the moment.

Thanks!

Mathieu

Quoted 150 lines

Thanks,
Jan

Quoted 146 lines

Thanks,

Mathieu

Quoted 136 lines

--Jan

Quoted 133 lines

commit 11fa478f18241fedee86f7dc7820a91c629c9e7e

Author: Mathieu Desnoyers <[email protected]>
Date:   Wed Apr 10 12:14:38 2013 -0400

Fix: remove ARM set_tls system call override

We'll need to find a better way to instrument ARM-specific system calls located at a far offset from the standard systems calls. A 16MB lttng-modules kernel module is really not acceptable.

Removing this instrumentation for now. sys_set_tls will appear as sys_unknown.

Fixes #472

CC: Ryan Kyser <[email protected]>
Ref: http://lists.lttng.org/pipermail/lttng-dev/2013-April/019990.html

instrumentation/syscalls/headers/arm-32-syscalls-2.6.38_integers_override.h +0 -16
index 93b8674..895370f 100644
@@ -2,7 +2,6 @@
 
 #define OVERRIDE_TABLE_32_sys_arm_fadvise64_64
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
-#define OVERRIDE_TABLE_32_sys_set_tls
 
 #ifndef CREATE_SYSCALL_TABLE
 
@@ -38,18 +37,6 @@ SC_TRACE_EVENT(sys_sync_file_range2,
 	TP_printk()
 )
 
-SC_TRACE_EVENT(sys_set_tls,
-	TP_PROTO(unsigned int tid, unsigned long tls),
-	TP_ARGS(tid, tls),
-	TP_STRUCT__entry(
-		__field(unsigned int, tid)
-		__field_hex(unsigned int, tls)),
-	TP_fast_assign(
-		tp_assign(tid, tid)
-		tp_assign(tls, tls)),
-	TP_printk()
-)
-
 #else	/* CREATE_SYSCALL_TABLE */
 
 #define OVVERRIDE_TABLE_32_sys_mmap
@@ -59,9 +46,6 @@ TRACE_SYSCALL_TABLE(sys_mmap, sys_mmap, 90, 6)
 TRACE_SYSCALL_TABLE(sys_arm_fadvise64_64, sys_arm_fadvise64_64, 270, 4)
 #define OVERRIDE_TABLE_32_sys_sync_file_range2
 TRACE_SYSCALL_TABLE(sys_sync_file_range2, sys_sync_file_range2, 341, 4)
-#define OVERRIDE_TABLE_32_sys_set_tls
-TRACE_SYSCALL_TABLE(sys_set_tls, sys_set_tls, 0xf0005, 2)
-
 
 #endif /* CREATE_SYSCALL_TABLE */
 
Quoted 68 lines

--Jan

Quoted 4 lines

-PL

Quoted 54 lines
Quoted 52 lines · Jan Glauber
Quoted 8 lines · PLSTC

Hey Jan,

I cannot speak on behalf of everyone here, but during our attempt to port LTTng to Android, we also noticed that the kernels we were using (3.0.x) were nowhere near the requirements for syscall tracepoints support. I believe such support was added on x86/64 way earlier (early 3.x) than on ARM, which is why it was included in LTTng's modules a while ago. Simply put, the ARM kernel is late.

OK, so for ARM kernel version 3.6 is the minimum unless the syscall tracepoint
support is backported.

Quoted 2 lines

There are a few actuals ways to 'enable' syscall tracepoints support on early ARM kernels, but they all including a bit of kernel

hacking/patching.

Quoted 1 line

I could send you some links if you're interested in that.

Yes, sure!

--Jan

Quoted 6 lines

-PL

Quoted 3 lines · Jan Glauber

Hi,

I want to use LTTng for system call tracing on ARM. Now lttng-modules

seems

Quoted 5 lines
Quoted 5 lines

to support system call tracing on ARM already since "8f4f80e LTTng Modules ARM syscall instrumentation".

But I wonder how that worked since lttng-syscalls.c is only build under CONFIG_HAVE_SYSCALL_TRACEPOINTS and that was added to ARM only with

kernel

Quoted 15 lines
Quoted 15 lines

3.6
(much after than the lttng-modules commit).

Am I missing something? Is system call tracing working on ARM with the upstream
LTTng version?

thanks,
Jan

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com