Skip to contentSkip to conversation

Extract lttng trace from kernel coredump

DISCUSSION David Goulet 3 participants 14 messages Jan 14, 2014 – Feb 19, 2014 11 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 8 lines · Corey Minyard

I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?

Oh yes we do actually already have a way to do that :)

Please see:
https://git.lttng.org/?p=lttng-tools.git;a=tree;f=extras/core-handler;h=29ded89a71fd36e159d671ca1f405cea394f8683;hb=HEAD

With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.

Cheers!
David

Quoted 9 lines

Thanks,

-corey

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 20140114184243.GC8177@thessa

#3 lore raw
Quoted 9 lines · David Goulet
Quoted 8 lines · Corey Minyard

I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?

Quoted 7 lines

Oh yes we do actually already have a way to do that :)

Please see:
https://git.lttng.org/?p=lttng-tools.git;a=tree;f=extras/core-handler;h=29ded89a71fd36e159d671ca1f405cea394f8683;hb=HEAD

With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.

I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.

Thanks,

-corey

Quoted 11 lines

Cheers!
David

Quoted 8 lines

Thanks,

-corey

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 15 lines · Corey Minyard
Quoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard

I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?

Oh yes we do actually already have a way to do that :)

Please see:
https://git.lttng.org/?p=lttng-tools.git;a=tree;f=extras/core-handler;h=29ded89a71fd36e159d671ca1f405cea394f8683;hb=HEAD

Quoted 8 lines
Quoted 2 lines

With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.

I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.

Oh I misunderstood, my appology. For that I have no idea how to proceed. Mathieu Desnoyers might be able to answer that since I'm pretty sure it has been at some point in time discussed.

CCing him.

Cheers!
David

Quoted 18 lines

Thanks,

-corey

Quoted 11 lines

Cheers!
David

Quoted 8 lines

Thanks,

-corey

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#6 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 21 lines

From: "Corey Minyard" <[email protected]>
To: "David Goulet" <[email protected]>
Cc: [email protected]
Sent: Tuesday, January 14, 2014 3:39:43 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

Quoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard

I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?

Oh yes we do actually already have a way to do that :)

Please see:
https://git.lttng.org/?p=lttng-tools.git;a=tree;f=extras/core-handler;h=29ded89a71fd36e159d671ca1f405cea394f8683;hb=HEAD

Quoted 8 lines
Quoted 2 lines

With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.

I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.

Something along these lines have been proposed a while back on the ML, but it was against an old LTTng version (0.x):

http://lists.lttng.org/pipermail/lttng-dev/2010-August/014239.html

It might be interesting to adapt it to the newer LTTng 2.x. Anyone interested to look into this ?

Thanks,

Mathieu

Quoted 24 lines

Thanks,

-corey

Quoted 11 lines

Cheers!
David

Quoted 8 lines

Thanks,

-corey

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 29 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 21 lines

From: "Corey Minyard" <[email protected]>
To: "David Goulet" <[email protected]>
Cc: [email protected]
Sent: Tuesday, January 14, 2014 3:39:43 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

Quoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard

I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?

Oh yes we do actually already have a way to do that :)

Please see:
https://git.lttng.org/?p=lttng-tools.git;a=tree;f=extras/core-handler;h=29ded89a71fd36e159d671ca1f405cea394f8683;hb=HEAD

Quoted 7 lines
Quoted 2 lines

With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.

I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.

Quoted 7 lines

Something along these lines have been proposed a while back on the ML, but it was against an old LTTng version (0.x):

http://lists.lttng.org/pipermail/lttng-dev/2010-August/014239.html

It might be interesting to adapt it to the newer LTTng 2.x. Anyone interested to look into this ?

That was done by a colleague of mine. However, things appear to have changed so much since then that it doesn't seem simple to extract a trace from memory. It could be done, but I'm currently looking at adding a separate transport that writes to a simpler per-core circular data structure and doesn't actually ever generate any output. Since it doesn't generate any output, it can be much simpler.

But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.

And I'm still learning about the internals of LTT.

Thanks,

-corey

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

----- Original Message -----

Quoted 45 lines

From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Saturday, January 18, 2014 1:29:52 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

Quoted 29 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 21 lines

From: "Corey Minyard" <[email protected]>
To: "David Goulet" <[email protected]>
Cc: [email protected]
Sent: Tuesday, January 14, 2014 3:39:43 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

Quoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard

I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?

Oh yes we do actually already have a way to do that :)

Please see:
https://git.lttng.org/?p=lttng-tools.git;a=tree;f=extras/core-handler;h=29ded89a71fd36e159d671ca1f405cea394f8683;hb=HEAD

Quoted 7 lines
Quoted 2 lines

With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.

I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.

Quoted 8 lines

Something along these lines have been proposed a while back on the ML, but it was against an old LTTng version (0.x):

http://lists.lttng.org/pipermail/lttng-dev/2010-August/014239.html

It might be interesting to adapt it to the newer LTTng 2.x. Anyone interested
to look into this ?

Quoted 6 lines

That was done by a colleague of mine. However, things appear to have changed so much since then that it doesn't seem simple to extract a trace from memory. It could be done, but I'm currently looking at adding a separate transport that writes to a simpler per-core circular data structure and doesn't actually ever generate any output. Since it doesn't generate any output, it can be much simpler.

You can use the flight recorder mode in recent LTTng for this (2.3 and newer). It simply writes to memory, without any output. I understand that you want to create a contiguous ring buffer memory layout. However, you have to be aware that this will probably be done using either

a) statically allocated memory at boot time (not very flexible),
b) vmalloc() (very flexible, but can triggers minor page faults, which can interact badly with page fault instrumentation. vmalloc() space is often limited by a kernel boot time parameter, and is putting quite important limitations on systems with 32-bit address spaces).

Quoted 5 lines

But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.

And I'm still learning about the internals of LTT.

One option would be to modify the tool to understand the LTTng 2.x buffer layout by stitching pages together by software using the LTTng libringbuffer "subbuffer table". You can think of it as a 2-level page table, but one level indexes the sub-buffers, and the next level indexes the pages within a sub-buffer.

A good way to understand its layout is to look at:

lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c

lib_ring_buffer_backend_allocate()

lib/ringbuffer/backend_types.h

struct lib_ring_buffer_backend
struct lib_ring_buffer_backend_subbuffer
struct lib_ring_buffer_backend_pages
struct lib_ring_buffer_backend_page

In your case, you never care about the bufb->buf_rsb (read-side owned subbuffer), because you always ever just write into it. buf_rsb is only useful when taking snapshots.

bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the actual sub-buffers and memory pages associated to each buffer.

You'll notice that the "id" field within struct lib_ring_buffer_backend_subbuffer is actually made of a mask of many fields. In order to understand how to use it, see

lib/ringbuffer/backend_types.h

where we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.

So you'll need to use the structures presented above to make sense of the memory layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.

The algorithm you want to end up doing (offline, on a vmcore) is pretty much the same as grabbing an online snapshot (iterate from the consumer position up to the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ). You will need an extra trick to handle the sub-buffer that was being written to at the time of the crash, by using the

lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field

which is designed to track the contiguously committed data within the currently written buffer. This can be used at any point in time (whenever a crash occurs) to populate the last sub-buffer's content size, packet size, see:

lttng-ring-buffer-client.h: client_buffer_end()

and find out how much of the last sub-buffer needs to be copied into the output CTF trace.

Thanks,

Mathieu

Quoted 5 lines

Thanks,

-corey

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 14 lines · Mathieu Desnoyers

You can use the flight recorder mode in recent LTTng for this (2.3 and newer). It simply writes to memory, without any output. I understand that you want to create a contiguous ring buffer memory layout. However, you have to be aware that this will probably be done using either

a) statically allocated memory at boot time (not very flexible),
b) vmalloc() (very flexible, but can triggers minor page faults, which can interact badly with page fault instrumentation. vmalloc() space is often limited by a kernel boot time parameter, and is putting quite important limitations on systems with 32-bit address spaces).

Quoted 3 lines

But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.

Quoted 6 lines
Quoted 1 line

And I'm still learning about the internals of LTT.

One option would be to modify the tool to understand the LTTng 2.x buffer layout by stitching pages together by software using the LTTng libringbuffer "subbuffer table". You can think of it as a 2-level page table, but one level indexes the sub-buffers, and the next level indexes the pages within a sub-buffer.

I'm finally back to this. I discovered that /proc/vmcore did not map vmalloc-ed memory, so I had to come up with something to handle that before I could continue to work on this.

This information was very useful, and snapshot mode is definitely the way to go. I just want to make sure I understand this before I go on. I have some specific question:

consumed is where the data starts and offset is where the data ends. So just go through the subbufs through a double index. Calculate the start/end location by taking the consumed/offset value, dividing that by the size of a subbuffer, looking up that value in backend.buf_wsb, getting the index from the id there, then indexing into the backend.array with the index. Once you have the subbuffer, you mod the location by the size of a subbuffer and that's the subbuffer offset. Divide the subbuffer offset by a page size to get the page index in the subbuffer, and mod by a page size to get the offset into the page.

Starting from the consumed position, dump data from pages until you hit subbufer->data_size, then move to the next subbuffer. On the last subbuffer, you have to fill in the header and dump up to the offset.

I think I'm missing something, though, because the data size of the last subbuffer doesn't match the offset location in that subbuffer. It's a pretty good distance away.

Thanks,

-corey

Quoted 61 lines

A good way to understand its layout is to look at:

lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c

lib_ring_buffer_backend_allocate()

lib/ringbuffer/backend_types.h

struct lib_ring_buffer_backend
struct lib_ring_buffer_backend_subbuffer
struct lib_ring_buffer_backend_pages
struct lib_ring_buffer_backend_page

In your case, you never care about the bufb->buf_rsb (read-side owned subbuffer), because you always ever just write into it. buf_rsb is only useful when taking snapshots.

bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the actual sub-buffers and memory pages associated to each buffer.

You'll notice that the "id" field within struct lib_ring_buffer_backend_subbuffer is actually made of a mask of many fields. In order to understand how to use it, see

lib/ringbuffer/backend_types.h

where we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.

So you'll need to use the structures presented above to make sense of the memory layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.

The algorithm you want to end up doing (offline, on a vmcore) is pretty much the same as grabbing an online snapshot (iterate from the consumer position up to the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ). You will need an extra trick to handle the sub-buffer that was being written to at the time of the crash, by using the

lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field

which is designed to track the contiguously committed data within the currently written buffer. This can be used at any point in time (whenever a crash occurs) to populate the last sub-buffer's content size, packet size, see:

lttng-ring-buffer-client.h: client_buffer_end()

and find out how much of the last sub-buffer needs to be copied into the output CTF trace.

Thanks,

Mathieu

Quoted 4 lines

Thanks,

-corey

#13 lore raw
In reply to a message not in the archive

I have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.

I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.

Thanks,

-corey

# gdb macros for extracting an LTT trace from a kernel coredump
# Copyright (C) 2014  Montavista Software <[email protected]>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA

# They layout of the LTT buffers in memory is rather complex. But here's

# the data structures involved:
#
# sessions - a list of running sessions of type struct lttng_session.
# A session consists of a some generic information and a list of
# channels for the session.
#
# session->chan - a list of channels of type struct lttng_channel.  The
# main information of interest is in chan.backend.
#
# This code parses the sessions and channels until it finds the first
# active non-metadata session/channel running in snapshot
# (relay-overwrite-mmap) mode.
#
# The chan.backend has some useful information about data sizes and a
# percpu array of ring buffers.  Each CPU logs to its ring buffer and
# the downstream tools pull all the data together.  The dump process
# creates a separate file for each CPU, just like the normal tools do.
#
# chan.backend->buf - A percpu array of structures of type struct
# lib_ring_buffer.  This is more or less the heart of the data
# structure.  Like channels, lib_ring_buffer has a backend field that
# holds most of the useful data.
#
# buf->backend.array - An array of pointers to subbuffers.  A
# subbuffer itself is an array of pointers to pages, and each page in
# order holds data.  LTT elements do not span subbuffers, so the
# actual size of the data in the subbuffer may be smaller than the
# full subbuffer size.  subbuf->data_size holds the actual data size in
# the subbuffer.
#
# buf->backend.buf_wsb - The subbuffers may not be in order.  This array
# holds the order of the subbuffers.
#
# chan->backend.num_subbuf - The number of subbuffers.
#
# chan->backend.subbuf_size - The size of each subbuffer, in bytes.
#
# So at this point, you can trace down any index by taking the index,
# dividing it by num_subbuf to get the subbuffer, taking that and
# doing a modulo on num_subbuf(*), indexing into buf->backend.buf_wsb
# to get the actual subbuffer number, then indexing into
# buf->backend.array to get a pointer to the subbuffer.  Then take the
# original index, modulo it by the number of subbuffers to get a
# subbuffer offset.  Dividing the subbuffer offset by the number of
# pages in a subbuffer gives the page in the subbuffer, and modulo the
# subbuffer offset by the page size to get the offset into the page.
# The page data structure has a virtual pointer so the data can be
# directly accessed.
#
# (*) the offset and consumed values do not wrap at the total output size,
# so you must modulo the subbuffer index to get the proper value.
#
# buf->offset - The index of the end of the data.
# buf->consumed - The index to the start of data.
#
# So the basic algorithm is to index down to buf->consumed and start dumping
# pages until buf->offset is reached.
#
# The command is invoked as:
#    dumpltt <lttdir> <outputdir>
# where lttdir is the place where the ltt modules are compiled and outputdir
# is the place where you want to put the trace output.
#
# The lttdir is required because this code loads the symbol tables
# from the ltt modules so it can access the various data structures in
# ltt.  It will not work without that.
#
# Note that the metadata for the dump is not available from the
# kernel.  You must take a snapshot on the target after you start ltt
# and use the metadata from the snapshot for your operation.

import os

def calc_offset(struct, member):
"""Calculate the offset in the struct to the given member.""" return gdb.parse_and_eval("((int)&((%s *)0).%s)" % (struct, member))

def fetch_container(ptr, struct, lmember):

    """Fetch a pointer to the given structure given a pointer to one of
    its members.  This is generally used to get the container of a list
    element."""
    offset = calc_offset(struct, lmember)
    s = gdb.parse_and_eval("(void *) (((unsigned long) %s) - %s)" %
                          (ptr, offset))
    return "((%s *) %s)" % (struct, s)

def get_ptr(ptr, struct):
"""Return a string that can be used to reference the given type (struct) at the given address (ptr)."""
s = gdb.parse_and_eval("(void *) %s" % ptr)
return "((%s *) %s)" % (struct, s)

def get_str(p):

    """Get a string value from the given target address"""
    s = gdb.parse_and_eval("(char *) %s" % p)
    return str(s).split('"')[1]

def align(v, alignment):
"""Align the given integer to the given alignment""" return (v + (alignment - 1)) & ~(alignment - 1)

def get_file_size(fname):

    """Return the size of the given file"""
    f = open(fname, "a")
    size = f.tell()
    f.close()
    return size

def value_to_binary(tmpdir, typestr, v):
"""Convert an expression to a binary value. The size depends on the typestr given. Return a string with the raw binary value in target-endian order"""
fn = "%s/dummy" % tmpdir
gdb.execute("append value %s ((%s) %ld)" % (fn, typestr, long(v))) f = open(fn, "r")
str = f.read()
f.close()
os.unlink(fn)
return str

def get_cpu_list():
"""Return a list of online CPUs on the target."""
try:
ulongsize = gdb.parse_and_eval("sizeof(unsigned long)") nr_cpumask = gdb.parse_and_eval("sizeof(*cpu_online_mask)") nr_cpumask = int(nr_cpumask) / int(ulongsize)
cpu = 0;
cpus = []
for i in range(0, nr_cpumask):
mask = gdb.parse_and_eval("cpu_online_mask->bits[%d]" % i) for j in range(0, ulongsize * 8):
if ((mask & 1) != 0):
cpus.append(cpu)
mask = mask >> 1
cpu = cpu + 1
except e:
cpus = [ 0 ]
return cpus

def get_percpu_ptr(p, cpu, typename):

    """Convert a percpu pointer 'p' on the given cpu to a pointer to the
    given type."""
    try:
        offset = gdb.parse_and_eval("__per_cpu_offset[%d]" % cpu)
    except:
        offset = "0";
    v = gdb.parse_and_eval("(void *) %s + %s" % (p, offset))
    return "((%s *) %s)" % (typename, v)

def find_module(modname):

    """Return a pointer to the given module name's data structure"""
    offset = gdb.parse_and_eval("((int)&((struct module *)0).list)")
    head = gdb.parse_and_eval("modules").address
    head = head.cast(gdb.lookup_type("unsigned long"))
    current = gdb.parse_and_eval("((struct list_head *) %s)->next" % head)
    while (current != head):
        modp = gdb.parse_and_eval("((struct module *) (((char *) %s) - %s))"
                                  % (current, offset))
        modp = "((struct module *) %s)" % modp
        cname = get_str("%s->name" % modp)
        if (cname == modname):
            return modp
        current = gdb.parse_and_eval("((struct list_head *) %s)->next" %
                                     current)
    return None        

class dumpltt(gdb.Command):
"""Dump an LTT coredump from the kernel image.""" def __init__(self):
super(dumpltt, self).__init__("dumpltt", gdb.COMMAND_USER)

def find_channel(self, head):
"""Find a channel in the list that is a per-cpu channel that is in snapshot mode (relay-overwrite-mmap) and return the first one found or None if not found."""
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % head) while (p != head):
chan = fetch_container(p, "struct lttng_channel", "list") enabled = gdb.parse_and_eval("%s->enabled" % chan)
if (int(enabled)):
chantype = gdb.parse_and_eval("%s->channel_type" % chan) if (chantype == self.PER_CPU_CHANNEL):
backend = gdb.parse_and_eval("&(%s->chan->backend)" % chan) backend = "((struct channel_backend *) %s)" % str(backend) tpname = get_str("%s->name" % backend)
if (tpname == "relay-overwrite-mmap"):
return backend
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return None

def find_session_channel(self):
"""Search the channels in the given session for one in snapshot mode."""
head = gdb.parse_and_eval("&sessions")
p = gdb.parse_and_eval("sessions.next")
while (p != head):
session = fetch_container(p, "struct lttng_session", "list") print "session: " + session
active = gdb.parse_and_eval("%s->active" % session) if (not int(active)):
continue
chanlist = gdb.parse_and_eval("&(%s->chan)" % session) chan = self.find_channel(chanlist)
if (chan is not None):
return chan
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return None

def add_alignment (self, cpu, size):
"""Dump 'size' bytes to the per-CPU output file.""" f = open("%s/channel0_%d" % (self.dumpdir, cpu), "ab") f.write("".ljust(size, chr(0)))
f.close()
return

def dump_page(self, cpu, page, start, size):
"""Dump 'size' bytes from the page starting at offset 'start' to the per-CPU output file."""
addr = long(gdb.parse_and_eval("%s->virt" % page)) + start end = addr + size - start
gdb.execute("append memory %s/channel0_%d %ld %ld" %
(self.dumpdir, cpu, addr, end))
return

def dump_subbuffer(self, cpu, rbbep, start, size):
"""Dump 'size' bytes from offset 'start' in the given subbuffer 'rbbep' to the per-CPU output file."""
start_page = start / self.page_size
start_offset = start % self.page_size
written = 0
for i in range(start_page, self.num_pages_per_subbuf):
bepage = get_ptr("&(%s->p[%d])" % (rbbep, i),
"struct lib_ring_buffer_backend_page")
if (size > self.page_size - start_offset):
self.dump_page(cpu, bepage, start_offset,
self.page_size - start_offset)
size -= self.page_size - start_offset written += self.page_size - start_offset start_offset = 0
else:
self.dump_page(cpu, bepage, start_offset, size) written += size
break
return written

def handle_subbuffer(self, cpu, buf, rbbe, cidx, coffset, length):
"""Extract necessary information from the subbuffer and dump it to a file.

buf is the ring buffer.
rbbe is a pointer to the buf->backend.buf_wsb array. cidx is the subbuffer index (for indexing into cidx). coffset is the offset within the subbuffer to start at. length is the number of bytes to dump from the subbuffer.

Note that each subbuffer has a header, and it may not be completely filled out. So add the subbuffer size and the size padded to a page to the header after writing it out.
"""
sid = long(gdb.parse_and_eval("%s[%d].id" % (rbbe, cidx))) sbidx = sid & self.idmask
rbbep = get_ptr("%s->backend.array[%d]" % (buf, sbidx),
"struct lib_ring_buffer_backend_pages")
size = int(gdb.parse_and_eval("%s->data_size" % rbbep)) last_subbuf_start = get_file_size("%s/channel0_%d"
% (self.dumpdir, cpu))
if (length > 0):
size = length
written_size = self.dump_subbuffer(cpu, rbbep, coffset, size); padded_size = align(written_size, self.page_size) self.add_alignment(cpu, padded_size - written_size)

print " subbuf size = %d (%d padding) %d" % (padded_size,
padded_size - written_size, size)

# Now fix the values in the header
f = open("%s/channel0_%d" % (self.dumpdir, cpu), "r+") f.seek(last_subbuf_start + 40, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", written_size * 8) f.write(val)
f.seek(last_subbuf_start + 48, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", padded_size * 8) f.write(val)
f.close()

def handle_buffer(self, cpu, buf):
"""Dump the given ring buffer to the given cpu's output file.""" print "Handling cpu %d buffer %s" % (cpu, buf)
offset = int(gdb.parse_and_eval("%s->offset.v" % buf)) oidx = (offset / self.subbuf_size) % self.num_subbuf ooffset = offset % self.subbuf_size
consumed = int(gdb.parse_and_eval("%s->consumed.counter" % buf)) cidx = (consumed / self.subbuf_size) % self.num_subbuf coffset = consumed % self.subbuf_size
rbbe = get_ptr("%s->backend.buf_wsb" % buf,
"struct lib_ring_buffer_backend_subbuffer")
subbuf_num = 1;
while (oidx != cidx):
print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, 0) cidx = (cidx + 1) % self.num_subbuf
coffset = 0
subbuf_num += 1

print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, ooffset - coffset) return

def invoke(self, args, from_tty):
argv = gdb.string_to_argv(args)
if (len(argv) != 2):
raise gdb.GdbError("dumpltt <lttdir> <dumpdir>")

lttdir = argv[0]
self.dumpdir = argv[1]
os.system("rm -rf %s/channel*" % self.dumpdir) os.system("rm -rf %s/dummy" % self.dumpdir)
modp = find_module("lttng_tracer")
if (modp is None):
raise gdb.GdbError("Unable to find lttng_tracer module, are you "
"sure lttng is loaded in this kernel dump?")
gdb.execute("add-symbol-file %s/lttng-tracer.o %s -s .data %s" %
(lttdir,
gdb.parse_and_eval("(unsigned long) (%s->module_core_rx)"
% modp),
gdb.parse_and_eval("(unsigned long) (%s->module_core_rw)"
% modp)))

self.PER_CPU_CHANNEL = gdb.parse_and_eval("PER_CPU_CHANNEL")

backend = self.find_session_channel()
if (backend is None):
raise gdb.GdbError("Unable to find operating session, are you sure "
"lttng was running with --snapshot?")
print "Backend: " + backend
self.num_subbuf = gdb.parse_and_eval("%s->num_subbuf" % backend); self.subbuf_size = gdb.parse_and_eval("%s->subbuf_size" % backend); p = get_percpu_ptr("%s->buf" % backend, 0, "struct lib_ring_buffer"); self.num_pages_per_subbuf = gdb.parse_and_eval(
"%s->backend.num_pages_per_subbuf" % p);
self.page_size = self.subbuf_size / self.num_pages_per_subbuf self.idshift = int(gdb.parse_and_eval("sizeof(long)")) * 8 / 2 self.idmask = (1 << self.idshift) - 1
cpus = get_cpu_list()
for cpu in cpus:
p = get_percpu_ptr("%s->buf" % backend, cpu,
"struct lib_ring_buffer");
self.handle_buffer(cpu, p)
dumpltt()

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw

----- Original Message -----

Quoted 28 lines

From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Thursday, February 13, 2014 2:31:13 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

Quoted 14 lines · Mathieu Desnoyers

You can use the flight recorder mode in recent LTTng for this (2.3 and newer). It simply writes to memory, without any output. I understand that you want to create a contiguous ring buffer memory layout. However, you have to be aware that this will probably be done using either

a) statically allocated memory at boot time (not very flexible),
b) vmalloc() (very flexible, but can triggers minor page faults, which can interact badly with page fault instrumentation. vmalloc() space is often limited by a kernel boot time parameter, and is putting quite important limitations on systems with 32-bit address spaces).

Quoted 3 lines

But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.

Quoted 6 lines
Quoted 1 line

And I'm still learning about the internals of LTT.

One option would be to modify the tool to understand the LTTng 2.x buffer layout by stitching pages together by software using the LTTng libringbuffer "subbuffer table". You can think of it as a 2-level page table, but one level indexes the sub-buffers, and the next level indexes the pages within a sub-buffer.

Quoted 7 lines

I'm finally back to this. I discovered that /proc/vmcore did not map vmalloc-ed memory, so I had to come up with something to handle that before I could continue to work on this.

This information was very useful, and snapshot mode is definitely the way to go. I just want to make sure I understand this before I go on. I have some specific question:

Sorry for the lag... ;)

I'll reply to your questions below just to make sure we're in sync.

Quoted 2 lines

consumed is where the data starts and offset is where the data ends.

Yes, approximately. This is right for consumed (this is where consumable data starts), but "offset" is either:
- where the data ends, or
- slightly beyond where the last contiguously committed data end.

It means we have to take extra care for buffers where the amount of commit_seq count is not a multiple of the sub-buffer size.

Quoted 3 lines

So
just go through the subbufs through a double index. Calculate the start/end location by taking the consumed/offset value,

First apply a module buffer size, otherwise the free running counters divided by subbuffer size will go beyond the backend.buf_wsb array.

Quoted 2 lines

dividing that by
the size of a subbuffer,

Yes,

Quoted 3 lines

looking up that value in backend.buf_wsb,
getting the index from the id there, then indexing into the backend.array with the index.

Yes.

Quoted 2 lines

Once you have the subbuffer, you mod the
location by the size of a subbuffer and that's the subbuffer offset.

Exactly.

Quoted 2 lines

Divide the subbuffer offset by a page size to get the page index in the subbuffer, and mod by a page size to get the offset into the page.

Yes.

Quoted 4 lines

Starting from the consumed position, dump data from pages until you hit subbufer->data_size, then move to the next subbuffer. On the last subbuffer, you have to fill in the header and dump up to the offset.

Not quite exactly. It's better to do:

for each subbuffer between consumed and offset (inclusive)
- if commit_seq is multiple of subbuffer size

    dump subbuffer->data_size

- if commit_seq is not a multiple of subbuffer size, dump commit_seq % subbuf size bytes of data, filling up the header accordingly.

Basically, all data that was "being written" (between commit_seq % subbuf size and write offset % subbuf size) cannot be read, because it likely contains holes and/or incomplete data.

Quoted 4 lines

I think I'm missing something, though, because the data size of the last subbuffer doesn't match the offset location in that subbuffer. It's a pretty good distance away.

Does my explanation above help clear things out ?

Thanks,

Mathieu

Quoted 81 lines

Thanks,

-corey

Quoted 74 lines

A good way to understand its layout is to look at:

lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c

lib_ring_buffer_backend_allocate()

lib/ringbuffer/backend_types.h

struct lib_ring_buffer_backend
struct lib_ring_buffer_backend_subbuffer
struct lib_ring_buffer_backend_pages
struct lib_ring_buffer_backend_page

In your case, you never care about the bufb->buf_rsb (read-side owned subbuffer),
because you always ever just write into it. buf_rsb is only useful when taking
snapshots.

bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the
actual sub-buffers and memory pages associated to each buffer.

You'll notice that the "id" field within struct lib_ring_buffer_backend_subbuffer
is actually made of a mask of many fields. In order to understand how to use it,
see

lib/ringbuffer/backend_types.h

where we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.

So you'll need to use the structures presented above to make sense of the memory
layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.

The algorithm you want to end up doing (offline, on a vmcore) is pretty much
the same as grabbing an online snapshot (iterate from the consumer position up to
the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
You will need an extra trick to handle the sub-buffer that was being written to
at the time of the crash, by using the

lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field

which is designed to track the contiguously committed data within the currently
written buffer. This can be used at any point in time (whenever a crash occurs) to
populate the last sub-buffer's content size, packet size, see:

lttng-ring-buffer-client.h: client_buffer_end()

and find out how much of the last sub-buffer needs to be copied into the output
CTF trace.

Thanks,

Mathieu

Quoted 4 lines

Thanks,

-corey

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#17 lore raw
Quoted 12 lines · Mathieu Desnoyers
Quoted 3 lines

Starting from the consumed position, dump data from pages until you hit subbufer->data_size, then move to the next subbuffer. On the last subbuffer, you have to fill in the header and dump up to the offset.

Not quite exactly. It's better to do:

for each subbuffer between consumed and offset (inclusive)
- if commit_seq is multiple of subbuffer size

    dump subbuffer->data_size

- if commit_seq is not a multiple of subbuffer size, dump commit_seq % subbuf size bytes of data, filling up the header accordingly.

Quoted 8 lines

Basically, all data that was "being written" (between commit_seq % subbuf size and write offset % subbuf size) cannot be read, because it likely contains holes and/or incomplete data.

Quoted 3 lines

I think I'm missing something, though, because the data size of the last subbuffer doesn't match the offset location in that subbuffer. It's a pretty good distance away.

Does my explanation above help clear things out ?

Yes, I've modified the code a bit to use the hot commit data if necessary, and to dump the metadata from the metadata cache. A new version is attached.

I also modified to use the flushed value if present. The probabilities of that situation causing a problem are pretty low, but this will avoid any issue. It looks correct to me.

-corey

Quoted 83 lines

Thanks,

Mathieu

Quoted 79 lines

Thanks,

-corey

Quoted 74 lines

A good way to understand its layout is to look at:

lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c

lib_ring_buffer_backend_allocate()

lib/ringbuffer/backend_types.h

struct lib_ring_buffer_backend
struct lib_ring_buffer_backend_subbuffer
struct lib_ring_buffer_backend_pages
struct lib_ring_buffer_backend_page

In your case, you never care about the bufb->buf_rsb (read-side owned subbuffer),
because you always ever just write into it. buf_rsb is only useful when taking
snapshots.

bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the
actual sub-buffers and memory pages associated to each buffer.

You'll notice that the "id" field within struct lib_ring_buffer_backend_subbuffer
is actually made of a mask of many fields. In order to understand how to use it,
see

lib/ringbuffer/backend_types.h

where we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.

So you'll need to use the structures presented above to make sense of the memory
layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.

The algorithm you want to end up doing (offline, on a vmcore) is pretty much
the same as grabbing an online snapshot (iterate from the consumer position up to
the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
You will need an extra trick to handle the sub-buffer that was being written to
at the time of the crash, by using the

lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field

which is designed to track the contiguously committed data within the currently
written buffer. This can be used at any point in time (whenever a crash occurs) to
populate the last sub-buffer's content size, packet size, see:

lttng-ring-buffer-client.h: client_buffer_end()

and find out how much of the last sub-buffer needs to be copied into the output
CTF trace.

Thanks,

Mathieu

Quoted 4 lines

Thanks,

-corey

# gdb macros for extracting an LTT trace from a kernel coredump
# Copyright (C) 2014  Montavista Software <[email protected]>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA

# The layout of the LTT buffers in memory is rather complex. But here's

# the data structures involved:
#
# sessions - a list of running sessions of type struct lttng_session.
# A session consists of a some generic information and a list of
# channels for the session.
#
# session->chan - a list of channels of type struct lttng_channel.  The
# main information of interest is in chan.backend.
#
# This code parses the sessions and channels until it finds the first
# active non-metadata session/channel running in snapshot
# (relay-overwrite-mmap) mode.
#
# The chan.backend has some useful information about data sizes and a
# percpu array of ring buffers.  Each CPU logs to its ring buffer and
# the downstream tools pull all the data together.  The dump process
# creates a separate file for each CPU, just like the normal tools do.
#
# chan.backend->buf - A percpu array of structures of type struct
# lib_ring_buffer.  This is more or less the heart of the data
# structure.  Like channels, lib_ring_buffer has a backend field that
# holds most of the useful data.
#
# buf->backend.array - An array of pointers to subbuffers.  A
# subbuffer itself is an array of pointers to pages, and each page in
# order holds data.  LTT elements do not span subbuffers, so the
# actual size of the data in the subbuffer may be smaller than the
# full subbuffer size.  subbuf->data_size holds the actual data size in
# the subbuffer.
#
# buf->backend.buf_wsb - The subbuffers may not be in order.  This array
# holds the order of the subbuffers.
#
# chan->backend.num_subbuf - The number of subbuffers.
#
# chan->backend.subbuf_size - The size of each subbuffer, in bytes.
#
# So at this point, you can trace down any index by taking the index,
# dividing it by num_subbuf to get the subbuffer, taking that and
# doing a modulo on num_subbuf(*), indexing into buf->backend.buf_wsb
# to get the actual subbuffer number, then indexing into
# buf->backend.array to get a pointer to the subbuffer.  Then take the
# original index, modulo it by the number of subbuffers to get a
# subbuffer offset.  Dividing the subbuffer offset by the number of
# pages in a subbuffer gives the page in the subbuffer, and modulo the
# subbuffer offset by the page size to get the offset into the page.
# The page data structure has a virtual pointer so the data can be
# directly accessed.
#
# (*) the offset and consumed values do not wrap at the total output size,
# so you must modulo the subbuffer index to get the proper value.
#
# buf->offset - The index of the end of the data.
# buf->consumed - The index to the start of data.
#
# So the basic algorithm is to index down to buf->consumed and start dumping
# pages until buf->offset is reached.
#
# The command is invoked as:
#    dumpltt <lttdir> <outputdir>
# where lttdir is the place where the ltt modules are compiled and outputdir
# is the place where you want to put the trace output.
#
# The lttdir is required because this code loads the symbol tables
# from the ltt modules so it can access the various data structures in
# ltt.  It will not work without that.

import os

def calc_offset(struct, member):
"""Calculate the offset in the struct to the given member.""" return gdb.parse_and_eval("((int)&((%s *)0).%s)" % (struct, member))

def fetch_container(ptr, struct, lmember):

    """Fetch a pointer to the given structure given a pointer to one of
    its members.  This is generally used to get the container of a list
    element."""
    offset = calc_offset(struct, lmember)
    s = gdb.parse_and_eval("(void *) (((unsigned long) %s) - %s)" %
                          (ptr, offset))
    return "((%s *) %s)" % (struct, s)

def get_ptr(ptr, struct):
"""Return a string that can be used to reference the given type (struct) at the given address (ptr)."""
s = gdb.parse_and_eval("(void *) %s" % ptr)
return "((%s *) %s)" % (struct, s)

def get_str(p):

    """Get a string value from the given target address"""
    s = gdb.parse_and_eval("(char *) %s" % p)
    return str(s).split('"')[1]

def align(v, alignment):
"""Align the given integer to the given alignment""" return (v + (alignment - 1)) & ~(alignment - 1)

def get_file_size(fname):

    """Return the size of the given file"""
    f = open(fname, "a")
    size = f.tell()
    f.close()
    return size

def value_to_binary(tmpdir, typestr, v):
"""Convert an expression to a binary value. The size depends on the typestr given. Return a string with the raw binary value in target-endian order"""
fn = "%s/dummy" % tmpdir
gdb.execute("append value %s ((%s) %ld)" % (fn, typestr, long(v))) f = open(fn, "r")
str = f.read()
f.close()
os.unlink(fn)
return str

def get_cpu_list():
"""Return a list of online CPUs on the target."""
try:
ulongsize = gdb.parse_and_eval("sizeof(unsigned long)") nr_cpumask = gdb.parse_and_eval("sizeof(*cpu_online_mask)") nr_cpumask = int(nr_cpumask) / int(ulongsize)
cpu = 0;
cpus = []
for i in range(0, nr_cpumask):
mask = gdb.parse_and_eval("cpu_online_mask->bits[%d]" % i) for j in range(0, ulongsize * 8):
if ((mask & 1) != 0):
cpus.append(cpu)
mask = mask >> 1
cpu = cpu + 1
except e:
cpus = [ 0 ]
return cpus

def get_percpu_ptr(p, cpu, typename):

    """Convert a percpu pointer 'p' on the given cpu to a pointer to the
    given type."""
    try:
        offset = gdb.parse_and_eval("__per_cpu_offset[%d]" % cpu)
    except:
        offset = "0";
    v = gdb.parse_and_eval("(void *) %s + %s" % (p, offset))
    return "((%s *) %s)" % (typename, v)

def get_member_ptr(p, membtype, member):
"""Given a pointer to a structure, get the given member as a pointer to another structure."""
v = gdb.parse_and_eval("%s->%s" % (p, member))
return "((%s *) %s)" % (membtype, str(v))

def get_member_long(p, member):

    """Given a pointer to a structure, get the given member that is an
    integer."""
    v = gdb.parse_and_eval("(long) (%s->%s)" % (p, member))
    return long(v)

def find_module(modname):

    """Return a pointer to the given module name's data structure"""
    offset = gdb.parse_and_eval("((int)&((struct module *)0).list)")
    head = gdb.parse_and_eval("modules").address
    head = head.cast(gdb.lookup_type("unsigned long"))
    current = gdb.parse_and_eval("((struct list_head *) %s)->next" % head)
    while (current != head):
        modp = gdb.parse_and_eval("((struct module *) (((char *) %s) - %s))"
                                  % (current, offset))
        modp = "((struct module *) %s)" % modp
        cname = get_str("%s->name" % modp)
        if (cname == modname):
            return modp
        current = gdb.parse_and_eval("((struct list_head *) %s)->next" %
                                     current)
    return None        

class dumpltt(gdb.Command):
"""Dump an LTT coredump from the kernel image.""" def __init__(self):
super(dumpltt, self).__init__("dumpltt", gdb.COMMAND_USER)

def find_channel(self, head):
"""Find a channel in the list that is a per-cpu channel that is in snapshot mode (relay-overwrite-mmap) and return the first one found or None if not found."""
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % head) while (p != head):
chan = fetch_container(p, "struct lttng_channel", "list") enabled = gdb.parse_and_eval("%s->enabled" % chan)
if (int(enabled)):
chantype = gdb.parse_and_eval("%s->channel_type" % chan) if (chantype == self.PER_CPU_CHANNEL):
backend = gdb.parse_and_eval("&(%s->chan->backend)" % chan) backend = "((struct channel_backend *) %s)" % str(backend) tpname = get_str("%s->name" % backend)
if (tpname == "relay-overwrite-mmap"):
return backend
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return None

def find_session_channel(self):
"""Search the channels in the given session for one in snapshot mode."""
head = gdb.parse_and_eval("&sessions")
p = gdb.parse_and_eval("sessions.next")
while (p != head):
session = fetch_container(p, "struct lttng_session", "list") active = gdb.parse_and_eval("%s->active" % session) if (not int(active)):
continue
chanlist = gdb.parse_and_eval("&(%s->chan)" % session) chan = self.find_channel(chanlist)
if (chan is not None):
return (session, chan)
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return (None, None)

def add_alignment (self, cpu, size):
"""Dump 'size' bytes to the per-CPU output file.""" f = open("%s/channel0_%d" % (self.dumpdir, cpu), "ab") f.write("".ljust(size, chr(0)))
f.close()
return

def dump_page(self, cpu, page, start, size):
"""Dump 'size' bytes from the page starting at offset 'start' to the per-CPU output file."""
addr = long(gdb.parse_and_eval("%s->virt" % page)) + start end = addr + size - start
gdb.execute("append memory %s/channel0_%d %ld %ld" %
(self.dumpdir, cpu, addr, end))
return

def dump_subbuffer(self, cpu, rbbep, start, size):
"""Dump 'size' bytes from offset 'start' in the given subbuffer 'rbbep' to the per-CPU output file."""
start_page = start / self.page_size
start_offset = start % self.page_size
written = 0
for i in range(start_page, self.num_pages_per_subbuf):
bepage = get_ptr("&(%s->p[%d])" % (rbbep, i),
"struct lib_ring_buffer_backend_page")
if (size > self.page_size - start_offset):
self.dump_page(cpu, bepage, start_offset,
self.page_size - start_offset)
size -= self.page_size - start_offset written += self.page_size - start_offset start_offset = 0
else:
self.dump_page(cpu, bepage, start_offset, size) written += size
break
return written

def handle_subbuffer(self, cpu, buf, rbbe, cidx, coffset, length):
"""Extract necessary information from the subbuffer and dump it to a file.

buf is the ring buffer.
rbbe is a pointer to the buf->backend.buf_wsb array. cidx is the subbuffer index (for indexing into cidx). coffset is the offset within the subbuffer to start at. length is the number of bytes to dump from the subbuffer.

Note that each subbuffer has a header, and it may not be completely filled out. So add the subbuffer size and the size padded to a page to the header after writing it out.
"""
sid = long(gdb.parse_and_eval("%s[%d].id" % (rbbe, cidx))) sbidx = sid & self.idmask
rbbep = get_ptr("%s->backend.array[%d]" % (buf, sbidx),
"struct lib_ring_buffer_backend_pages")
size = get_member_long(buf, "commit_hot[%d].seq.v" % (cidx)) if ((size & (self.subbuf_size - 1)) == 0):
# Hot value is a multiple of the subbuffer size, use the # buffer data size.
size = int(gdb.parse_and_eval("%s->data_size" % rbbep))
else:
size = size % self.subbuf_size
last_subbuf_start = get_file_size("%s/channel0_%d"
% (self.dumpdir, cpu))
if (length > 0 and length < size):
size = length
written_size = self.dump_subbuffer(cpu, rbbep, coffset, size); padded_size = align(written_size, self.page_size) self.add_alignment(cpu, padded_size - written_size)

print " subbuf size = %d (pad %d) %d" % (padded_size,
padded_size - written_size, size)

# Now fix the values in the header
f = open("%s/channel0_%d" % (self.dumpdir, cpu), "r+") f.seek(last_subbuf_start + 40, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", written_size * 8) f.write(val)
f.seek(last_subbuf_start + 48, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", padded_size * 8) f.write(val)
f.close()

def handle_buffer(self, cpu, buf):
"""Dump the given ring buffer to the given cpu's output file.""" print "Handling cpu %d buffer %s" % (cpu, buf)
offset = int(gdb.parse_and_eval("%s->offset.v" % buf)) oidx = (offset / self.subbuf_size) % self.num_subbuf ooffset = offset % self.subbuf_size
consumed = int(gdb.parse_and_eval("%s->consumed.counter" % buf)) cidx = (consumed / self.subbuf_size) % self.num_subbuf coffset = consumed % self.subbuf_size
rbbe = get_ptr("%s->backend.buf_wsb" % buf,
"struct lib_ring_buffer_backend_subbuffer")
subbuf_num = 1;
while (oidx != cidx):
print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, 0) cidx = (cidx + 1) % self.num_subbuf
coffset = 0
subbuf_num += 1

print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, ooffset - coffset) return

def dump_metadata(self, session):
f = open("%s/metadata" % (self.dumpdir), "w") f.write("/* CTF 1.8 */\n")
f.close()
mdcache = get_member_ptr(session, "struct lttng_metadata_cache",
"metadata_cache")
data = get_member_long(mdcache, "data")
size = get_member_long(mdcache, "metadata_written") gdb.execute("append memory %s/metadata %ld %ld" %
(self.dumpdir, data, data + size))

def invoke(self, args, from_tty):
argv = gdb.string_to_argv(args)
if (len(argv) != 2):
raise gdb.GdbError("dumpltt <lttdir> <dumpdir>")

lttdir = argv[0]
self.dumpdir = argv[1]
os.system("mkdir -p %s" % self.dumpdir)
os.system("rm -rf %s/channel*" % self.dumpdir) os.system("rm -rf %s/dummy" % self.dumpdir)
modp = find_module("lttng_tracer")
if (modp is None):
raise gdb.GdbError("Unable to find lttng_tracer module, are you "
"sure lttng is loaded in this kernel dump?")
gdb.execute("add-symbol-file %s/lttng-tracer.o %s -s .data %s" %
(lttdir,
gdb.parse_and_eval("(unsigned long) (%s->module_core_rx)"
% modp),
gdb.parse_and_eval("(unsigned long) (%s->module_core_rw)"
% modp)))

self.PER_CPU_CHANNEL = gdb.parse_and_eval("PER_CPU_CHANNEL")

(session, backend) = self.find_session_channel()
if (backend is None):
raise gdb.GdbError("Unable to find operating session, are you sure "
"lttng was running with --snapshot?")
self.dump_metadata(session)
self.num_subbuf = gdb.parse_and_eval("%s->num_subbuf" % backend); self.subbuf_size = gdb.parse_and_eval("%s->subbuf_size" % backend); p = get_percpu_ptr("%s->buf" % backend, 0, "struct lib_ring_buffer"); self.num_pages_per_subbuf = gdb.parse_and_eval(
"%s->backend.num_pages_per_subbuf" % p);
self.page_size = self.subbuf_size / self.num_pages_per_subbuf self.idshift = int(gdb.parse_and_eval("sizeof(long)")) * 8 / 2 self.idmask = (1 << self.idshift) - 1
cpus = get_cpu_list()
for cpu in cpus:
p = get_percpu_ptr("%s->buf" % backend, cpu,
"struct lib_ring_buffer");
self.handle_buffer(cpu, p)
dumpltt()

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#19 lore raw

----- Original Message -----

Quoted 8 lines

From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

I have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.

Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.

Quoted 6 lines

I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.

Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).

You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at

/lttng-events.h

struct lttng_session {
...
struct lttng_metadata_cache *metadata_cache;
...
};

Which has the metadata string:

struct lttng_metadata_cache {

        char *data;                     /* Metadata cache */
        unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
        unsigned int metadata_written;  /* Number of bytes written in metadata cache */
        struct kref refcount;           /* Metadata cache usage */

struct list_head metadata_stream; /* Metadata stream list */
};

You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.

If you happen to core dump while the metadata cache is being resized, you might encounter a corrupted metadata. It should not happen frequently, but it's possible. We could eventually change lttng_metadata_printf() so it makes sure to always keep a readable metadata around (by e.g. using kcalloc rather than krealloc and dealing carefully with cache_alloc vs data fields updates).

Thoughts ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#21 lore raw
Quoted 11 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 8 lines

From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

I have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.

Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.

Yes, I'm going through that and the code I have to get things clean.

Quoted 35 lines
Quoted 5 lines

I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.

Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).

You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at

/lttng-events.h

struct lttng_session {
...
struct lttng_metadata_cache *metadata_cache;
...
};

Which has the metadata string:

struct lttng_metadata_cache {

        char *data;                     /* Metadata cache */
        unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
        unsigned int metadata_written;  /* Number of bytes written in metadata cache */
        struct kref refcount;           /* Metadata cache usage */

struct list_head metadata_stream; /* Metadata stream list */
};

You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.

Quoted 7 lines

If you happen to core dump while the metadata cache is being resized, you might encounter a corrupted metadata. It should not happen frequently, but it's possible. We could eventually change lttng_metadata_printf() so it makes sure to always keep a readable metadata around (by e.g. using kcalloc rather than krealloc and dealing carefully with cache_alloc vs data fields updates).

Thoughts ?

Perfect, that's easy. I'll do that.

Thanks again.

-corey

#22 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 57 lines

From: "Mathieu Desnoyers" <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Tuesday, February 18, 2014 4:02:15 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

----- Original Message -----

Quoted 8 lines

From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

I have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.

Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.

Quoted 6 lines

I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.

Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).

You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at

/lttng-events.h

struct lttng_session {
...
struct lttng_metadata_cache *metadata_cache;
...
};

Which has the metadata string:

struct lttng_metadata_cache {

        char *data;                     /* Metadata cache */
        unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
        unsigned int metadata_written;  /* Number of bytes written in

metadata cache */
struct kref refcount; /* Metadata cache usage */
struct list_head metadata_stream; /* Metadata stream list */
};

You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.

Quoted 8 lines

If you happen to core dump while the metadata cache is being resized, you might
encounter a corrupted metadata. It should not happen frequently, but it's possible.
We could eventually change lttng_metadata_printf() so it makes sure to always keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
dealing carefully with cache_alloc vs data fields updates).

Oops, I meant "kzalloc".

I'm attaching a patch that should take care of making sure it is always in a valid state when a core dump occurs. You should be able to safely read "metadata_flushed" bytes of data at any time. Whatever has been flushed should be parseable by Babeltrace.

Thoughts ?

Thanks,

Mathieu

Quoted 17 lines

Thoughts ?

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

commit 875fbbbe6167b3847cdf4de925722be9dcf685cc

Author: Mathieu Desnoyers <[email protected]>
Date:   Tue Feb 18 16:35:05 2014 -0500

Ensure metadata can be extracted from core dumps at any time

lttng-events.c +65 -18
index 0512a3f..0e67520 100644
@@ -603,7 +603,7 @@ int lttng_metadata_output_channel(struct lttng_metadata_stream *stream,
 	if (stream->metadata_in != stream->metadata_out)
 		return 0;
 
-	len = stream->metadata_cache->metadata_written -
+	len = stream->metadata_cache->metadata_flushed -
 		stream->metadata_in;
 	if (!len)
 		return 0;
@@ -631,17 +631,41 @@ end:
 	return ret;
 }
 
+static
+void lttng_metadata_flush(struct lttng_session *session)
+{
+	struct lttng_metadata_cache *cache = session->metadata_cache;
+	struct lttng_metadata_stream *stream;
+
+	if (cache->metadata_written == cache->metadata_flushed)
+		return;
+
+	/*
+	 * Append to metadata before incrementing "metadata flushed",
+	 * ensuring that core dumps always read up to "metadata flushed"
+	 * bytes of valid metadata.
+	 */
+	barrier();
+	cache->metadata_flushed = cache->metadata_written;
+
+	list_for_each_entry(stream, &cache->metadata_stream, list)
+		wake_up_interruptible(&stream->read_wait);
+}
+
 /*
  * Write the metadata to the metadata cache.
  * Must be called with sessions_mutex held.
+ * Grow the metadata cache with allocation, copy, populating the new
+ * structure, and then free the old structure, to ensure that there is
+ * always a consistent state to grab from kernel core dump.
  */
+static
 int lttng_metadata_printf(struct lttng_session *session,
 			  const char *fmt, ...)
 {
 	char *str;
 	size_t len;
 	va_list ap;
-	struct lttng_metadata_stream *stream;
 
 	WARN_ON_ONCE(!ACCESS_ONCE(session->active));
 
@@ -654,18 +678,33 @@ int lttng_metadata_printf(struct lttng_session *session,
 	len = strlen(str);
 	if (session->metadata_cache->metadata_written + len >
 			session->metadata_cache->cache_alloc) {
-		char *tmp_cache_realloc;
-		unsigned int tmp_cache_alloc_size;
-
-		tmp_cache_alloc_size = max_t(unsigned int,
-				session->metadata_cache->cache_alloc + len,
-				session->metadata_cache->cache_alloc << 1);
-		tmp_cache_realloc = krealloc(session->metadata_cache->data,
-				tmp_cache_alloc_size, GFP_KERNEL);
-		if (!tmp_cache_realloc)
+		char *new_cache, *old_cache;
+		unsigned int new_cache_alloc_size, old_cache_alloc_size;
+
+		old_cache_alloc_size = session->metadata_cache->cache_alloc;
+		new_cache_alloc_size = max_t(unsigned int,
+				old_cache_alloc_size + len,
+				old_cache_alloc_size << 1);
+		new_cache = kzalloc(new_cache_alloc_size, GFP_KERNEL);
+		if (!new_cache)
 			goto err;
-		session->metadata_cache->cache_alloc = tmp_cache_alloc_size;
-		session->metadata_cache->data = tmp_cache_realloc;
+		session->metadata_cache->cache_alloc = new_cache_alloc_size;
+		old_cache = session->metadata_cache->data;
+		memcpy(new_cache, old_cache, old_cache_alloc_size);
+		/*
+		 * Copy the old cache into the new one before storing
+		 * the pointer to new cache into "data", so core dumps
+		 * read valid metadata.
+		 */
+		barrier();
+		session->metadata_cache->data = new_cache;
+		/*
+		 * Store "data" before freeing the old cache, thus
+		 * ensuring kernel core dumps always read up to
+		 * "metadata_flushed" bytes of valid metadata.
+		 */
+		barrier();
+		kfree(old_cache);
 	}
 	memcpy(session->metadata_cache->data +
 			session->metadata_cache->metadata_written,
@@ -673,9 +712,6 @@ int lttng_metadata_printf(struct lttng_session *session,
 	session->metadata_cache->metadata_written += len;
 	kfree(str);
 
-	list_for_each_entry(stream, &session->metadata_cache->metadata_stream, list)
-		wake_up_interruptible(&stream->read_wait);
-
 	return 0;
 
 err:
@@ -906,6 +942,7 @@ int _lttng_event_metadata_statedump(struct lttng_session *session,
 
 	event->metadata_dumped = 1;
 end:
+	lttng_metadata_flush(session);
 	return ret;
 
 }
@@ -958,6 +995,7 @@ int _lttng_channel_metadata_statedump(struct lttng_session *session,
 
 	chan->metadata_dumped = 1;
 end:
+	lttng_metadata_flush(session);
 	return ret;
 }
 
@@ -967,7 +1005,9 @@ end:
 static
 int _lttng_stream_packet_context_declare(struct lttng_session *session)
 {
-	return lttng_metadata_printf(session,
+	int ret;
+
+	ret = lttng_metadata_printf(session,
 		"struct packet_context {\n"
 		"	uint64_clock_monotonic_t timestamp_begin;\n"
 		"	uint64_clock_monotonic_t timestamp_end;\n"
@@ -977,6 +1017,8 @@ int _lttng_stream_packet_context_declare(struct lttng_session *session)
 		"	uint32_t cpu_id;\n"
 		"};\n\n"
 		);
+	lttng_metadata_flush(session);
+	return ret;
 }
 
 /*
@@ -993,7 +1035,9 @@ int _lttng_stream_packet_context_declare(struct lttng_session *session)
 static
 int _lttng_event_header_declare(struct lttng_session *session)
 {
-	return lttng_metadata_printf(session,
+	int ret;
+
+	ret = lttng_metadata_printf(session,
 	"struct event_header_compact {\n"
 	"	enum : uint5_t { compact = 0 ... 30, extended = 31 } id;\n"
 	"	variant <id> {\n"
@@ -1022,6 +1066,8 @@ int _lttng_event_header_declare(struct lttng_session *session)
 	lttng_alignof(uint32_t) * CHAR_BIT,
 	lttng_alignof(uint16_t) * CHAR_BIT
 	);
+	lttng_metadata_flush(session);
+	return ret;
 }
 
  /*
@@ -1209,6 +1255,7 @@ skip_session:
 	}
 	session->metadata_dumped = 1;
 end:
+	lttng_metadata_flush(session);
 	return ret;
 }
 
lttng-events.h +1 -0
index 6b39304..9d43d47 100644
@@ -326,6 +326,7 @@ struct lttng_metadata_cache {
 	char *data;			/* Metadata cache */
 	unsigned int cache_alloc;	/* Metadata allocated size (bytes) */
 	unsigned int metadata_written;	/* Number of bytes written in metadata cache */
+	unsigned int metadata_flushed;	/* Number of bytes flushed in metadata cache */
 	struct kref refcount;		/* Metadata cache usage */
 	struct list_head metadata_stream;	/* Metadata stream list */
 };

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#24 lore raw

----- Original Message -----

Quoted 75 lines

From: "Mathieu Desnoyers" <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Tuesday, February 18, 2014 4:42:06 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

----- Original Message -----

Quoted 58 lines

From: "Mathieu Desnoyers" <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Tuesday, February 18, 2014 4:02:15 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

----- Original Message -----

Quoted 8 lines

From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump

I have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.

Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.

Quoted 6 lines

I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.

Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).

You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at

/lttng-events.h

struct lttng_session {
...
struct lttng_metadata_cache *metadata_cache;
...
};

Which has the metadata string:

struct lttng_metadata_cache {
char *data; /* Metadata cache */
unsigned int cache_alloc; /* Metadata allocated size (bytes)
*/
unsigned int metadata_written; /* Number of bytes written in metadata cache */
struct kref refcount; /* Metadata cache usage */
struct list_head metadata_stream; /* Metadata stream list */
};

You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.

Quoted 9 lines

If you happen to core dump while the metadata cache is being resized, you might
encounter a corrupted metadata. It should not happen frequently, but it's possible.
We could eventually change lttng_metadata_printf() so it makes sure to always
keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
dealing carefully with cache_alloc vs data fields updates).

Quoted 6 lines

Oops, I meant "kzalloc".

I'm attaching a patch that should take care of making sure it is always in a valid state when a core dump occurs. You should be able to safely read "metadata_flushed" bytes of data at any time. Whatever has been flushed should be parseable by Babeltrace.

Updated patch (fixing poll).

Thanks,

Mathieu

Quoted 35 lines

Thoughts ?

Thanks,

Mathieu

Quoted 12 lines

Thoughts ?

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

commit 94d1517096f2bcf01cdfdccbfb4030dd62d8bed4

Author: Mathieu Desnoyers <[email protected]>
Date:   Tue Feb 18 16:35:05 2014 -0500

Ensure metadata can be extracted from core dumps at any time

lttng-abi.c +1 -1
index 261a0ad..bdb4f57 100644
@@ -541,7 +541,7 @@ unsigned int lttng_metadata_ring_buffer_poll(struct file *filp,
 		if (finalized)
 			mask |= POLLHUP;
 
-		if (stream->metadata_cache->metadata_written >
+		if (stream->metadata_cache->metadata_flushed >
 				stream->metadata_out)
 			mask |= POLLIN;
 	}
lttng-events.c +65 -18

160 lines not on this page · raw message

lttng-events.h +1 -0
index 6b39304..9d43d47 100644
@@ -326,6 +326,7 @@ struct lttng_metadata_cache {
 	char *data;			/* Metadata cache */
 	unsigned int cache_alloc;	/* Metadata allocated size (bytes) */
 	unsigned int metadata_written;	/* Number of bytes written in metadata cache */
+	unsigned int metadata_flushed;	/* Number of bytes flushed in metadata cache */
 	struct kref refcount;		/* Metadata cache usage */
 	struct list_head metadata_stream;	/* Metadata stream list */
 };

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev