Skip to contentSkip to conversation

Error: opening metadata failed (LTTng on OpenWRT)

DISCUSSION Jérémie Galarneau 3 participants 8 messages Nov 14, 2017 – Nov 23, 2017 8 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi Robin,

Can you add the commands you use to create the session?

Thanks,
Jérémie

Quoted 74 lines · Robin Michielsen

Hi all,

I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions (2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?

I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:

· MIPS platform running Linux 3.18.29

· Package versions:

o lttng-modules-2.10.4

o lttng-tools-2.10.1

o lttng-ust-2.10.0

· lsmod shows that the following modules have been loaded:

o lttng_clock

o lttng_lib_ring_buffer

o lttng_ring_buffer_client_discard

o lttng_statedump

o lttng_tracer

· Commands to create and start a session:

o lttng create

o lttng enable-event --kernel --syscall open,close

o lttng start

When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in trace_kernel_destroy_metadata() at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (in thread_manage_clients() at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.

Does anybody know what may cause this issue, or have a hint on to check?

Kind regards,

Robin

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 7 lines
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CA+jJMxuwsDjeWXTrypiu-2zXc8B2Akhbn--wyq9isoxo0wY=1w@mail.gmail.com

#3 lore raw

Hi Jérémie,

Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?

• lttng create
• lttng enable-event --kernel --syscall open,close
• lttng start

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi Robin,

Can you add the commands you use to create the session?

Thanks,
Jérémie

Quoted 71 lines · Robin Michielsen

Hi all,

I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?

I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:

· MIPS platform running Linux 3.18.29

· Package versions:

o lttng-modules-2.10.4

o lttng-tools-2.10.1

o lttng-ust-2.10.0

· lsmod shows that the following modules have been loaded:

o lttng_clock

o lttng_lib_ring_buffer

o lttng_ring_buffer_client_discard

o lttng_statedump

o lttng_tracer

· Commands to create and start a session:

o lttng create

o lttng enable-event --kernel --syscall open,close

o lttng start

When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in trace_kernel_destroy_metadata() at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (in thread_manage_clients() at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.

Does anybody know what may cause this issue, or have a hint on to check?

Kind regards,

Robin

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 8 lines · Robin Michielsen

Hi Jérémie,

Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?

•       lttng create
•       lttng enable-event --kernel --syscall open,close
•       lttng start

Yes, sorry. I missed it on the first read through.

Jérémie

Quoted 102 lines

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi Robin,

Can you add the commands you use to create the session?

Thanks,
Jérémie

Quoted 71 lines · Robin Michielsen

Hi all,

I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?

I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:

· MIPS platform running Linux 3.18.29

· Package versions:

o lttng-modules-2.10.4

o lttng-tools-2.10.1

o lttng-ust-2.10.0

· lsmod shows that the following modules have been loaded:

o lttng_clock

o lttng_lib_ring_buffer

o lttng_ring_buffer_client_discard

o lttng_statedump

o lttng_tracer

· Commands to create and start a session:

o lttng create

o lttng enable-event --kernel --syscall open,close

o lttng start

When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in trace_kernel_destroy_metadata() at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (in thread_manage_clients() at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.

Does anybody know what may cause this issue, or have a hint on to check?

Kind regards,

Robin

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

Signature · 7 lines
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CA+jJMxsfhj7JKu8MDdLqVb0eELmTq8vORLmyccNQzYpk8yubWw@mail.gmail.com

#7 lore raw

Hi all,

Any suggestions from anyone on what to check and/or how to debug the issue that I described in my mail from last week?

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 22:14
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Quoted 8 lines · Robin Michielsen

Hi Jérémie,

Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?

•       lttng create
•       lttng enable-event --kernel --syscall open,close
•       lttng start

Yes, sorry. I missed it on the first read through.

Jérémie

Quoted 99 lines

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi Robin,

Can you add the commands you use to create the session?

Thanks,
Jérémie

Quoted 71 lines · Robin Michielsen

Hi all,

I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?

I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:

· MIPS platform running Linux 3.18.29

· Package versions:

o lttng-modules-2.10.4

o lttng-tools-2.10.1

o lttng-ust-2.10.0

· lsmod shows that the following modules have been loaded:

o lttng_clock

o lttng_lib_ring_buffer

o lttng_ring_buffer_client_discard

o lttng_statedump

o lttng_tracer

· Commands to create and start a session:

o lttng create

o lttng enable-event --kernel --syscall open,close

o lttng start

When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in trace_kernel_destroy_metadata() at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (in thread_manage_clients() at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.

Does anybody know what may cause this issue, or have a hint on to check?

Kind regards,

Robin

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Quoted 5 lines

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw

Hi,

Quoted 3 lines · Robin Michielsen

Hi all,

Any suggestions from anyone on what to check and/or how to debug the issue that I described in my mail from last week?

From lttng-sessiond.log:

lttng-ring-buffer-client-discard is already loaded
Error: Unable to load required module lttng-ring-buffer-client-discard

Is there any particular reason why some modules are already loaded ?

Did you perform a "depmod -a" on the target?

Cheers

Quoted 141 lines

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 22:14
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Quoted 8 lines · Robin Michielsen

Hi Jérémie,

Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?

•       lttng create
•       lttng enable-event --kernel --syscall open,close
•       lttng start

Yes, sorry. I missed it on the first read through.

Jérémie

Quoted 99 lines

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi Robin,

Can you add the commands you use to create the session?

Thanks,
Jérémie

Quoted 71 lines · Robin Michielsen

Hi all,

I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?

I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:

· MIPS platform running Linux 3.18.29

· Package versions:

o lttng-modules-2.10.4

o lttng-tools-2.10.1

o lttng-ust-2.10.0

· lsmod shows that the following modules have been loaded:

o lttng_clock

o lttng_lib_ring_buffer

o lttng_ring_buffer_client_discard

o lttng_statedump

o lttng_tracer

· Commands to create and start a session:

o lttng create

o lttng enable-event --kernel --syscall open,close

o lttng start

When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in trace_kernel_destroy_metadata() at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (in thread_manage_clients() at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.

Does anybody know what may cause this issue, or have a hint on to check?

Kind regards,

Robin

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Quoted 5 lines

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 6 lines
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 20171122185514.mvxrlkspqbwunwlt@psrcode-TP-X230

#11 lore raw

Hi Jonathan,

The lttng-sessiond.log that I provided was actually from a second time that I started 'lttng-sessiond' (did not realize that there is a difference), which resulted in the already loaded error. When I captured the output from a fresh start of 'lttng-sessiond' I noticed lots of modprobe errors with modules that could not be loaded.

Turns out that the modprobe on my OpenWrt system can only handle modules with a name of max 32 characters (!). The first mandatory core module ("lttng-ring-buffer-client-discard") is still loaded as its name is exactly 32 chars, the other 5 core modules do not get loaded as their names are longer than 32 chars. I will modify modprobe as a proper fix, but shortening the names of the lttng-*.ko modules and manually loading each core module already showed that this solves the issue.

Thanks for your reply! This helped in pointing me in the right direction.

Kind regards,
Robin

-----Original Message-----
From: Jonathan Rajotte-Julien [mailto:[email protected]]
Sent: Wednesday, 22 November, 2017 19:55
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi,

Quoted 3 lines · Robin Michielsen

Hi all,

Any suggestions from anyone on what to check and/or how to debug the issue that I described in my mail from last week?

From lttng-sessiond.log:

lttng-ring-buffer-client-discard is already loaded
Error: Unable to load required module lttng-ring-buffer-client-discard

Is there any particular reason why some modules are already loaded ?

Did you perform a "depmod -a" on the target?

Cheers

Quoted 140 lines

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 22:14
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Quoted 8 lines · Robin Michielsen

Hi Jérémie,

Maybe I misunderstood how to create a session, but are the commands that I listed not sufficient for this?

•       lttng create
•       lttng enable-event --kernel --syscall open,close
•       lttng start

Yes, sorry. I missed it on the first read through.

Jérémie

Quoted 100 lines

Kind regards,
Robin

-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Jérémie Galarneau
Sent: Tuesday, 14 November, 2017 17:09
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi Robin,

Can you add the commands you use to create the session?

Thanks,
Jérémie

Quoted 71 lines · Robin Michielsen

Hi all,

I am trying to trace the Linux kernel (system calls) using LTTng on a platform that is running OpenWRT. I have cross compiled the latest versions
(2.10.x) of the lttng-modules, lttng-tools and lttng-ust packages and installed these on my OpenWRT system. Now when I start a kernel trace then the lttng tool returns the error "LTTng transport relay-metadata not found /
Error: Opening metadata failed". Does anybody know what might cause this error, or have a hint on what to check?

I have attached the captured output of ‘lttng-sessiond -vvv --verbose-consumer’. Here is some more context:

· MIPS platform running Linux 3.18.29

· Package versions:

o lttng-modules-2.10.4

o lttng-tools-2.10.1

o lttng-ust-2.10.0

· lsmod shows that the following modules have been loaded:

o lttng_clock

o lttng_lib_ring_buffer

o lttng_ring_buffer_client_discard

o lttng_statedump

o lttng_tracer

· Commands to create and start a session:

o lttng create

o lttng enable-event --kernel --syscall open,close

o lttng start

When ‘lttng start’ is called then lttng-sessiond’s debug output shows “Closing metadata fd -1 (in trace_kernel_destroy_metadata() at trace-kernel.c:569)” followed by “Sending response (size: 20, retcode:
Opening metadata failed (45)) (in thread_manage_clients() at main.c:4641)”. I assume that the error that I’m seeing is because for some reason metadata fd is being closed.

Does anybody know what may cause this issue, or have a hint on to check?

Kind regards,

Robin

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Quoted 5 lines

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

Quoted 4 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Jonathan Rajotte-Julien
EfficiOS

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw

Hi,

Quoted 3 lines · Robin Michielsen

Hi Jonathan,

The lttng-sessiond.log that I provided was actually from a second time that I started 'lttng-sessiond' (did not realize that there is a difference), which resulted in the already loaded error. When I captured the output from a fresh start of 'lttng-sessiond' I noticed lots of modprobe errors with modules that could not be loaded.

Normally, lttng-sessiond will unload the module it loaded on start if a clean teardown is initiated (SIGTERM).

Quoted 2 lines

Turns out that the modprobe on my OpenWrt system can only handle modules with a name of max 32 characters (!). The first mandatory core module ("lttng-ring-buffer-client-discard") is still loaded as its name is exactly 32 chars, the other 5 core modules do not get loaded as their names are longer than 32 chars. I will modify modprobe as a proper fix, but shortening the names of the lttng-*.ko modules and manually loading each core module already showed that this solves the issue.

Glad I could help. :P I'm curious could you point me to the modprobe code that openWRT uses?

Cheers!

Quoted 6 lines

Thanks for your reply! This helped in pointing me in the right direction.

Kind regards,
Robin

Signature · 6 lines
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 20171123185405.ogq622ssm22d2ye3@psrcode-TP-X230

#15 lore raw

Hi Jonathan,

You can find the modprobe code (kmodloader.c, dated 2015-11-22) at
git://nbd.name/luci2/ubox.git

Kind regards,
Robin

-----Original Message-----
From: Jonathan Rajotte-Julien [mailto:[email protected]]
Sent: Thursday, 23 November, 2017 19:54
To: Robin Michielsen
Cc: [email protected]
Subject: Re: [lttng-dev] Error: opening metadata failed (LTTng on OpenWRT)

Hi,

Quoted 3 lines · Robin Michielsen

Hi Jonathan,

The lttng-sessiond.log that I provided was actually from a second time

that I started 'lttng-sessiond' (did not realize that there is a difference), which resulted in the already loaded error. When I captured the output from a fresh start of 'lttng-sessiond' I noticed lots of modprobe errors with modules that could not be loaded.

Normally, lttng-sessiond will unload the module it loaded on start if a clean teardown is initiated (SIGTERM).

Quoted 2 lines

Turns out that the modprobe on my OpenWrt system can only handle modules

with a name of max 32 characters (!). The first mandatory core module ("lttng-ring-buffer-client-discard") is still loaded as its name is exactly 32 chars, the other 5 core modules do not get loaded as their names are longer than 32 chars. I will modify modprobe as a proper fix, but shortening the names of the lttng-*.ko modules and manually loading each core module already showed that this solves the issue.

Glad I could help. :P I'm curious could you point me to the modprobe code that openWRT uses?

Cheers!

Quoted 6 lines

Thanks for your reply! This helped in pointing me in the right direction.

Kind regards,
Robin

--
Jonathan Rajotte-Julien
EfficiOS

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev