Error: opening metadata failed (LTTng on OpenWRT)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Robin,
Can you add the commands you use to create the session?
Thanks,
Jérémie
Quoted 74 lines · Robin Michielsen
Hi all,
I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions (2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:
· MIPS platform running Linux 3.18.29
· Package versions:
o lttng-modules-2.10.4
o lttng-tools-2.10.1
o lttng-ust-2.10.0
· lsmod shows that the following modules have been loaded:
o
lttng_clocko
lttng_lib_ring_buffero
lttng_ring_buffer_client_discardo
lttng_statedumpo
lttng_tracer· Commands to create and start a session:
o lttng create
o lttng enable-event --kernel --syscall open,close
o lttng start
When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in
trace_kernel_destroy_metadata()at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (inthread_manage_clients()at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.Does anybody know what may cause this issue, or have a hint on to check?
Kind regards,
Robin
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 7 lines
Jérémie Galarneau EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): CA+jJMxuwsDjeWXTrypiu-2zXc8B2Akhbn--wyq9isoxo0wY=1w@mail.gmail.com
Hi Jérémie,
Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?
• lttng create
• lttng enable-event --kernel --syscall open,close
• lttng start
Kind regards,
Robin
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)
Hi Robin,
Can you add the commands you use to create the session?
Thanks,
Jérémie
Quoted 71 lines · Robin Michielsen
Hi all,
I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:
· MIPS platform running Linux 3.18.29
· Package versions:
o lttng-modules-2.10.4
o lttng-tools-2.10.1
o lttng-ust-2.10.0
· lsmod shows that the following modules have been loaded:
o
lttng_clocko
lttng_lib_ring_buffero
lttng_ring_buffer_client_discardo
lttng_statedumpo
lttng_tracer· Commands to create and start a session:
o lttng create
o lttng enable-event --kernel --syscall open,close
o lttng start
When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in
trace_kernel_destroy_metadata()at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (inthread_manage_clients()at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.Does anybody know what may cause this issue, or have a hint on to check?
Kind regards,
Robin
Quoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Robin Michielsen
Hi Jérémie,
Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?
• lttng create • lttng enable-event --kernel --syscall open,close • lttng start
Yes, sorry. I missed it on the first read through.
Jérémie
Quoted 102 lines
Kind regards,
Robin-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)Hi Robin,
Can you add the commands you use to create the session?
Thanks,
JérémieQuoted 71 lines · Robin Michielsen
Hi all,
I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:
· MIPS platform running Linux 3.18.29
· Package versions:
o lttng-modules-2.10.4
o lttng-tools-2.10.1
o lttng-ust-2.10.0
· lsmod shows that the following modules have been loaded:
o
lttng_clocko
lttng_lib_ring_buffero
lttng_ring_buffer_client_discardo
lttng_statedumpo
lttng_tracer· Commands to create and start a session:
o lttng create
o lttng enable-event --kernel --syscall open,close
o lttng start
When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in
trace_kernel_destroy_metadata()at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (inthread_manage_clients()at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.Does anybody know what may cause this issue, or have a hint on to check?
Kind regards,
Robin
Quoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
Signature · 7 lines
Jérémie Galarneau EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): CA+jJMxsfhj7JKu8MDdLqVb0eELmTq8vORLmyccNQzYpk8yubWw@mail.gmail.com
Hi all,
Any suggestions from anyone on what to check and/or how to debug the issue that I described in my mail from last week?
Kind regards,
Robin
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 22:14
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)
Quoted 8 lines · Robin Michielsen
Hi Jérémie,
Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?
• lttng create • lttng enable-event --kernel --syscall open,close • lttng start
Yes, sorry. I missed it on the first read through.
Jérémie
Quoted 99 lines
Kind regards,
Robin-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)Hi Robin,
Can you add the commands you use to create the session?
Thanks,
JérémieQuoted 71 lines · Robin Michielsen
Hi all,
I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:
· MIPS platform running Linux 3.18.29
· Package versions:
o lttng-modules-2.10.4
o lttng-tools-2.10.1
o lttng-ust-2.10.0
· lsmod shows that the following modules have been loaded:
o
lttng_clocko
lttng_lib_ring_buffero
lttng_ring_buffer_client_discardo
lttng_statedumpo
lttng_tracer· Commands to create and start a session:
o lttng create
o lttng enable-event --kernel --syscall open,close
o lttng start
When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in
trace_kernel_destroy_metadata()at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (inthread_manage_clients()at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.Does anybody know what may cause this issue, or have a hint on to check?
Kind regards,
Robin
Quoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 5 lines
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 3 lines · Robin Michielsen
Hi all,
Any suggestions from anyone on what to check and/or how to debug the issue that I described in my mail from last week?
From lttng-sessiond.log:
lttng-ring-buffer-client-discard is already loaded
Error: Unable to load required module lttng-ring-buffer-client-discard
Is there any particular reason why some modules are already loaded ?
Did you perform a "depmod -a" on the target?
Cheers
Quoted 141 lines
Kind regards,
Robin-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 22:14
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)Quoted 8 lines · Robin Michielsen
Hi Jérémie,
Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?
• lttng create • lttng enable-event --kernel --syscall open,close • lttng startYes, sorry. I missed it on the first read through.
Jérémie
Quoted 99 lines
Kind regards,
Robin-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)Hi Robin,
Can you add the commands you use to create the session?
Thanks,
JérémieQuoted 71 lines · Robin Michielsen
Hi all,
I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:
· MIPS platform running Linux 3.18.29
· Package versions:
o lttng-modules-2.10.4
o lttng-tools-2.10.1
o lttng-ust-2.10.0
· lsmod shows that the following modules have been loaded:
o
lttng_clocko
lttng_lib_ring_buffero
lttng_ring_buffer_client_discardo
lttng_statedumpo
lttng_tracer· Commands to create and start a session:
o lttng create
o lttng enable-event --kernel --syscall open,close
o lttng start
When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in
trace_kernel_destroy_metadata()at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (inthread_manage_clients()at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.Does anybody know what may cause this issue, or have a hint on to check?
Kind regards,
Robin
Quoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-devQuoted 5 lines
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 6 lines
Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 20171122185514.mvxrlkspqbwunwlt@psrcode-TP-X230
Hi Jonathan,
The lttng-sessiond.log that I provided was actually from a second time that I started 'lttng-sessiond' (did not realize that there is a difference), which resulted in the already loaded error. When I captured the output from a fresh start of 'lttng-sessiond' I noticed lots of modprobe errors with modules that could not be loaded.
Turns out that the modprobe on my OpenWrt system can only handle modules with a name of max 32 characters (!). The first mandatory core module ("lttng-ring-buffer-client-discard") is still loaded as its name is exactly 32 chars, the other 5 core modules do not get loaded as their names are longer than 32 chars. I will modify modprobe as a proper fix, but shortening the names of the lttng-*.ko modules and manually loading each core module already showed that this solves the issue.
Thanks for your reply! This helped in pointing me in the right direction.
Kind regards,
Robin
-----Original Message-----
From: Jonathan Rajotte-Julien [mailto:[email protected]]
Sent: Wednesday, 22 November, 2017 19:55
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)
Hi,
Quoted 3 lines · Robin Michielsen
Hi all,
Any suggestions from anyone on what to check and/or how to debug the issue that I described in my mail from last week?
From lttng-sessiond.log:
lttng-ring-buffer-client-discard is already loaded
Error: Unable to load required module lttng-ring-buffer-client-discard
Is there any particular reason why some modules are already loaded ?
Did you perform a "depmod -a" on the target?
Cheers
Quoted 140 lines
Kind regards,
Robin-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 22:14
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)Quoted 8 lines · Robin Michielsen
Hi Jérémie,
Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?
• lttng create • lttng enable-event --kernel --syscall open,close • lttng startYes, sorry. I missed it on the first read through.
Jérémie
Quoted 100 lines
Kind regards,
Robin-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)Hi Robin,
Can you add the commands you use to create the session?
Thanks,
JérémieQuoted 71 lines · Robin Michielsen
Hi all,
I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:
· MIPS platform running Linux 3.18.29
· Package versions:
o lttng-modules-2.10.4
o lttng-tools-2.10.1
o lttng-ust-2.10.0
· lsmod shows that the following modules have been loaded:
o
lttng_clocko
lttng_lib_ring_buffero
lttng_ring_buffer_client_discardo
lttng_statedumpo
lttng_tracer· Commands to create and start a session:
o lttng create
o lttng enable-event --kernel --syscall open,close
o lttng start
When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in
trace_kernel_destroy_metadata()at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (inthread_manage_clients()at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.Does anybody know what may cause this issue, or have a hint on to check?
Kind regards,
Robin
Quoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-devQuoted 5 lines
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
Quoted 4 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 3 lines · Robin Michielsen
Hi Jonathan,
The lttng-sessiond.log that I provided was actually from a second time that I started 'lttng-sessiond' (did not realize that there is a difference), which resulted in the already loaded error. When I captured the output from a fresh start of 'lttng-sessiond' I noticed lots of modprobe errors with modules that could not be loaded.
Normally, lttng-sessiond will unload the module it loaded on start if a clean teardown is initiated (SIGTERM).
Quoted 2 lines
Turns out that the modprobe on my OpenWrt system can only handle modules with a name of max 32 characters (!). The first mandatory core module ("lttng-ring-buffer-client-discard") is still loaded as its name is exactly 32 chars, the other 5 core modules do not get loaded as their names are longer than 32 chars. I will modify modprobe as a proper fix, but shortening the names of the lttng-*.ko modules and manually loading each core module already showed that this solves the issue.
Glad I could help. :P I'm curious could you point me to the modprobe code that openWRT uses?
Cheers!
Quoted 6 lines
Thanks for your reply! This helped in pointing me in the right direction.
Kind regards,
Robin
Signature · 6 lines
Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 20171123185405.ogq622ssm22d2ye3@psrcode-TP-X230
Hi Jonathan,
You can find the modprobe code (kmodloader.c, dated 2015-11-22) atgit://nbd.name/luci2/ubox.git
Kind regards,
Robin
-----Original Message-----
From: Jonathan Rajotte-Julien [mailto:[email protected]]
Sent: Thursday, 23 November, 2017 19:54
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)
Hi,
Quoted 3 lines · Robin Michielsen
Hi Jonathan,
The lttng-sessiond.log that I provided was actually from a second time
that I started 'lttng-sessiond' (did not realize that there is a difference), which resulted in the already loaded error. When I captured the output from a fresh start of 'lttng-sessiond' I noticed lots of modprobe errors with modules that could not be loaded.
Normally, lttng-sessiond will unload the module it loaded on start if a clean teardown is initiated (SIGTERM).
Quoted 2 lines
Turns out that the modprobe on my OpenWrt system can only handle modules
with a name of max 32 characters (!). The first mandatory core module ("lttng-ring-buffer-client-discard") is still loaded as its name is exactly 32 chars, the other 5 core modules do not get loaded as their names are longer than 32 chars. I will modify modprobe as a proper fix, but shortening the names of the lttng-*.ko modules and manually loading each core module already showed that this solves the issue.
Glad I could help. :P I'm curious could you point me to the modprobe code that openWRT uses?
Cheers!
Quoted 6 lines
Thanks for your reply! This helped in pointing me in the right direction.
Kind regards,
Robin
--
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev