Extract lttng trace from kernel coredump
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Corey Minyard
I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?
Oh yes we do actually already have a way to do that :)
With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.
Cheers!
David
Quoted 9 lines
Thanks,
-corey
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 20140114184243.GC8177@thessa
Quoted 9 lines · David Goulet
Quoted 8 lines · Corey Minyard
I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?
Quoted 7 lines
Oh yes we do actually already have a way to do that :)
With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.
I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.
Thanks,
-corey
Quoted 11 lines
Cheers!
DavidQuoted 8 lines
Thanks,
-corey
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 15 lines · Corey Minyard
Quoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard
I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?
Oh yes we do actually already have a way to do that :)
Quoted 8 lines
Quoted 2 lines
With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.
I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.
Oh I misunderstood, my appology. For that I have no idea how to proceed. Mathieu Desnoyers might be able to answer that since I'm pretty sure it has been at some point in time discussed.
CCing him.
Cheers!
David
Quoted 18 lines
Thanks,
-corey
Quoted 11 lines
Cheers!
DavidQuoted 8 lines
Thanks,
-corey
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
----- Original Message -----
Quoted 21 lines
From: "Corey Minyard" <[email protected]>
To: "David Goulet" <[email protected]>
Cc: [email protected]
Sent: Tuesday, January 14, 2014 3:39:43 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpQuoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard
I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?
Oh yes we do actually already have a way to do that :)
Quoted 8 lines
Quoted 2 lines
With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.
I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.
Something along these lines have been proposed a while back on the ML, but it was against an old LTTng version (0.x):
http://lists.lttng.org/pipermail/lttng-dev/2010-August/014239.html
It might be interesting to adapt it to the newer LTTng 2.x. Anyone interested to look into this ?
Thanks,
Mathieu
Quoted 24 lines
Thanks,
-corey
Quoted 11 lines
Cheers!
DavidQuoted 8 lines
Thanks,
-corey
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 29 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 21 lines
From: "Corey Minyard" <[email protected]>
To: "David Goulet" <[email protected]>
Cc: [email protected]
Sent: Tuesday, January 14, 2014 3:39:43 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpQuoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard
I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?
Oh yes we do actually already have a way to do that :)
Quoted 7 lines
Quoted 2 lines
With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.
I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.
Quoted 7 lines
Something along these lines have been proposed a while back on the ML, but it was against an old LTTng version (0.x):
http://lists.lttng.org/pipermail/lttng-dev/2010-August/014239.html
It might be interesting to adapt it to the newer LTTng 2.x. Anyone interested to look into this ?
That was done by a colleague of mine. However, things appear to have changed so much since then that it doesn't seem simple to extract a trace from memory. It could be done, but I'm currently looking at adding a separate transport that writes to a simpler per-core circular data structure and doesn't actually ever generate any output. Since it doesn't generate any output, it can be much simpler.
But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.
And I'm still learning about the internals of LTT.
Thanks,
-corey
A message that is not in the archive (sent off-list or not yet received): [email protected]
----- Original Message -----
Quoted 45 lines
From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Saturday, January 18, 2014 1:29:52 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpQuoted 29 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 21 lines
From: "Corey Minyard" <[email protected]>
To: "David Goulet" <[email protected]>
Cc: [email protected]
Sent: Tuesday, January 14, 2014 3:39:43 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpQuoted 14 lines · David Goulet
Quoted 8 lines · Corey Minyard
I'm working on a feature to allow an lttng trace to be extracted from a kernel coredump, and I'm wondering the best way to proceed. I've looked at extracting from the existing data structures, but that would require something to always have a trace file open and it looks more than a little complicated. I was thinking instead of having a different trace channel for configuring this coredump and some globals to point to the various data structures. Would that be the best way to pursue this? Is there a better way, perhaps?
Oh yes we do actually already have a way to do that :)
Quoted 7 lines
Quoted 2 lines
With the snapshot feature, it's quite easy to do that. Hope that helps you with what you are trying to achieve.
I did see that, but it seems to be for an application coredump. I was looking for something that would work for a kernel coredump. So that if the kernel crashes, you can use kdump to extract the kernel core and then extract the LTT buffers from that. The idea is that if the kernel crashes, you can get some idea of what was happening before the crash.
Quoted 8 lines
Something along these lines have been proposed a while back on the ML, but it was against an old LTTng version (0.x):
http://lists.lttng.org/pipermail/lttng-dev/2010-August/014239.html
It might be interesting to adapt it to the newer LTTng 2.x. Anyone interested
to look into this ?
Quoted 6 lines
That was done by a colleague of mine. However, things appear to have changed so much since then that it doesn't seem simple to extract a trace from memory. It could be done, but I'm currently looking at adding a separate transport that writes to a simpler per-core circular data structure and doesn't actually ever generate any output. Since it doesn't generate any output, it can be much simpler.
You can use the flight recorder mode in recent LTTng for this (2.3 and newer). It simply writes to memory, without any output. I understand that you want to create a contiguous ring buffer memory layout. However, you have to be aware that this will probably be done using either
a) statically allocated memory at boot time (not very flexible),
b) vmalloc() (very flexible, but can triggers minor page faults, which can interact badly with page fault instrumentation. vmalloc() space is often limited by a kernel boot time parameter, and is putting quite important limitations on systems with 32-bit address spaces).
Quoted 5 lines
But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.
And I'm still learning about the internals of LTT.
One option would be to modify the tool to understand the LTTng 2.x buffer layout by stitching pages together by software using the LTTng libringbuffer "subbuffer table". You can think of it as a 2-level page table, but one level indexes the sub-buffers, and the next level indexes the pages within a sub-buffer.
A good way to understand its layout is to look at:
lttng-modules (master)lib/ringbuffer/ring_buffer_backend.c
lib_ring_buffer_backend_allocate()
lib/ringbuffer/backend_types.h
struct lib_ring_buffer_backend
struct lib_ring_buffer_backend_subbuffer
struct lib_ring_buffer_backend_pages
struct lib_ring_buffer_backend_page
In your case, you never care about the bufb->buf_rsb (read-side owned subbuffer), because you always ever just write into it. buf_rsb is only useful when taking snapshots.
bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the actual sub-buffers and memory pages associated to each buffer.
You'll notice that the "id" field within struct lib_ring_buffer_backend_subbuffer is actually made of a mask of many fields. In order to understand how to use it, see
lib/ringbuffer/backend_types.h
where we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.
So you'll need to use the structures presented above to make sense of the memory layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.
The algorithm you want to end up doing (offline, on a vmcore) is pretty much the same as grabbing an online snapshot (iterate from the consumer position up to the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ). You will need an extra trick to handle the sub-buffer that was being written to at the time of the crash, by using the
lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field
which is designed to track the contiguously committed data within the currently written buffer. This can be used at any point in time (whenever a crash occurs) to populate the last sub-buffer's content size, packet size, see:
lttng-ring-buffer-client.h: client_buffer_end()
and find out how much of the last sub-buffer needs to be copied into the output CTF trace.
Thanks,
Mathieu
Quoted 5 lines
Thanks,
-corey
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 14 lines · Mathieu Desnoyers
You can use the flight recorder mode in recent LTTng for this (2.3 and newer). It simply writes to memory, without any output. I understand that you want to create a contiguous ring buffer memory layout. However, you have to be aware that this will probably be done using either
a) statically allocated memory at boot time (not very flexible),
b)vmalloc()(very flexible, but can triggers minor page faults, which can interact badly with page fault instrumentation.vmalloc()space is often limited by a kernel boot time parameter, and is putting quite important limitations on systems with 32-bit address spaces).Quoted 3 lines
But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.
Quoted 6 lines
Quoted 1 line
And I'm still learning about the internals of LTT.
One option would be to modify the tool to understand the LTTng 2.x buffer layout by stitching pages together by software using the LTTng libringbuffer "subbuffer table". You can think of it as a 2-level page table, but one level indexes the sub-buffers, and the next level indexes the pages within a sub-buffer.
I'm finally back to this. I discovered that /proc/vmcore did not map vmalloc-ed memory, so I had to come up with something to handle that before I could continue to work on this.
This information was very useful, and snapshot mode is definitely the way to go. I just want to make sure I understand this before I go on. I have some specific question:
consumed is where the data starts and offset is where the data ends. So just go through the subbufs through a double index. Calculate the start/end location by taking the consumed/offset value, dividing that by the size of a subbuffer, looking up that value in backend.buf_wsb, getting the index from the id there, then indexing into the backend.array with the index. Once you have the subbuffer, you mod the location by the size of a subbuffer and that's the subbuffer offset. Divide the subbuffer offset by a page size to get the page index in the subbuffer, and mod by a page size to get the offset into the page.
Starting from the consumed position, dump data from pages until you hit subbufer->data_size, then move to the next subbuffer. On the last subbuffer, you have to fill in the header and dump up to the offset.
I think I'm missing something, though, because the data size of the last subbuffer doesn't match the offset location in that subbuffer. It's a pretty good distance away.
Thanks,
-corey
Quoted 61 lines
A good way to understand its layout is to look at:
lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c
lib_ring_buffer_backend_allocate()
lib/ringbuffer/backend_types.hstruct
lib_ring_buffer_backend
structlib_ring_buffer_backend_subbuffer
structlib_ring_buffer_backend_pages
structlib_ring_buffer_backend_pageIn your case, you never care about the
bufb->buf_rsb(read-side owned subbuffer), because you always ever just write into it.buf_rsbis only useful when taking snapshots.bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the actual sub-buffers and memory pages associated to each buffer.
You'll notice that the "id" field within struct
lib_ring_buffer_backend_subbufferis actually made of a mask of many fields. In order to understand how to use it, see
lib/ringbuffer/backend_types.hwhere we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.
So you'll need to use the structures presented above to make sense of the memory layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.
The algorithm you want to end up doing (offline, on a vmcore) is pretty much the same as grabbing an online snapshot (iterate from the consumer position up to the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ). You will need an extra trick to handle the sub-buffer that was being written to at the time of the crash, by using the
lib/ringbuffer/frontend_types.hstructcommit_counters_hot"seq" fieldwhich is designed to track the contiguously committed data within the currently written buffer. This can be used at any point in time (whenever a crash occurs) to populate the last sub-buffer's content size, packet size, see:
lttng-ring-buffer-client.h:client_buffer_end()and find out how much of the last sub-buffer needs to be copied into the output CTF trace.
Thanks,
Mathieu
Quoted 4 lines
Thanks,
-corey
I have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.
I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.
Thanks,
-corey
# gdb macros for extracting an LTT trace from a kernel coredump # Copyright (C) 2014 Montavista Software <[email protected]> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
# They layout of the LTT buffers in memory is rather complex. But here's
# the data structures involved: # # sessions - a list of running sessions of type struct lttng_session. # A session consists of a some generic information and a list of # channels for the session. # # session->chan - a list of channels of type struct lttng_channel. The # main information of interest is in chan.backend. # # This code parses the sessions and channels until it finds the first # active non-metadata session/channel running in snapshot # (relay-overwrite-mmap) mode. # # The chan.backend has some useful information about data sizes and a # percpu array of ring buffers. Each CPU logs to its ring buffer and # the downstream tools pull all the data together. The dump process # creates a separate file for each CPU, just like the normal tools do. # # chan.backend->buf - A percpu array of structures of type struct # lib_ring_buffer. This is more or less the heart of the data # structure. Like channels, lib_ring_buffer has a backend field that # holds most of the useful data. # # buf->backend.array - An array of pointers to subbuffers. A # subbuffer itself is an array of pointers to pages, and each page in # order holds data. LTT elements do not span subbuffers, so the # actual size of the data in the subbuffer may be smaller than the # full subbuffer size. subbuf->data_size holds the actual data size in # the subbuffer. # # buf->backend.buf_wsb - The subbuffers may not be in order. This array # holds the order of the subbuffers. # # chan->backend.num_subbuf - The number of subbuffers. # # chan->backend.subbuf_size - The size of each subbuffer, in bytes. # # So at this point, you can trace down any index by taking the index, # dividing it by num_subbuf to get the subbuffer, taking that and # doing a modulo on num_subbuf(*), indexing into buf->backend.buf_wsb # to get the actual subbuffer number, then indexing into # buf->backend.array to get a pointer to the subbuffer. Then take the # original index, modulo it by the number of subbuffers to get a # subbuffer offset. Dividing the subbuffer offset by the number of # pages in a subbuffer gives the page in the subbuffer, and modulo the # subbuffer offset by the page size to get the offset into the page. # The page data structure has a virtual pointer so the data can be # directly accessed. # # (*) the offset and consumed values do not wrap at the total output size, # so you must modulo the subbuffer index to get the proper value. # # buf->offset - The index of the end of the data. # buf->consumed - The index to the start of data. # # So the basic algorithm is to index down to buf->consumed and start dumping # pages until buf->offset is reached. # # The command is invoked as: # dumpltt <lttdir> <outputdir> # where lttdir is the place where the ltt modules are compiled and outputdir # is the place where you want to put the trace output. # # The lttdir is required because this code loads the symbol tables # from the ltt modules so it can access the various data structures in # ltt. It will not work without that. # # Note that the metadata for the dump is not available from the # kernel. You must take a snapshot on the target after you start ltt # and use the metadata from the snapshot for your operation.
import os
def calc_offset(struct, member):
"""Calculate the offset in the struct to the given member.""" return gdb.parse_and_eval("((int)&((%s *)0).%s)" % (struct, member))
def fetch_container(ptr, struct, lmember):
"""Fetch a pointer to the given structure given a pointer to one of
its members. This is generally used to get the container of a list
element."""
offset = calc_offset(struct, lmember)
s = gdb.parse_and_eval("(void *) (((unsigned long) %s) - %s)" %
(ptr, offset))
return "((%s *) %s)" % (struct, s)def get_ptr(ptr, struct):
"""Return a string that can be used to reference the given type (struct) at the given address (ptr)."""
s = gdb.parse_and_eval("(void *) %s" % ptr)
return "((%s *) %s)" % (struct, s)
def get_str(p):
"""Get a string value from the given target address"""
s = gdb.parse_and_eval("(char *) %s" % p)
return str(s).split('"')[1]def align(v, alignment):
"""Align the given integer to the given alignment""" return (v + (alignment - 1)) & ~(alignment - 1)
def get_file_size(fname):
"""Return the size of the given file"""
f = open(fname, "a")
size = f.tell()
f.close()
return sizedef value_to_binary(tmpdir, typestr, v):
"""Convert an expression to a binary value. The size depends on the typestr given. Return a string with the raw binary value in target-endian order"""
fn = "%s/dummy" % tmpdir
gdb.execute("append value %s ((%s) %ld)" % (fn, typestr, long(v))) f = open(fn, "r")
str = f.read()
f.close()
os.unlink(fn)
return str
def get_cpu_list():
"""Return a list of online CPUs on the target."""
try:
ulongsize = gdb.parse_and_eval("sizeof(unsigned long)") nr_cpumask = gdb.parse_and_eval("sizeof(*cpu_online_mask)") nr_cpumask = int(nr_cpumask) / int(ulongsize)
cpu = 0;
cpus = []
for i in range(0, nr_cpumask):
mask = gdb.parse_and_eval("cpu_online_mask->bits[%d]" % i) for j in range(0, ulongsize * 8):
if ((mask & 1) != 0):
cpus.append(cpu)
mask = mask >> 1
cpu = cpu + 1
except e:
cpus = [ 0 ]
return cpus
def get_percpu_ptr(p, cpu, typename):
"""Convert a percpu pointer 'p' on the given cpu to a pointer to the
given type."""
try:
offset = gdb.parse_and_eval("__per_cpu_offset[%d]" % cpu)
except:
offset = "0";
v = gdb.parse_and_eval("(void *) %s + %s" % (p, offset))
return "((%s *) %s)" % (typename, v)def find_module(modname):
"""Return a pointer to the given module name's data structure"""
offset = gdb.parse_and_eval("((int)&((struct module *)0).list)")
head = gdb.parse_and_eval("modules").address
head = head.cast(gdb.lookup_type("unsigned long"))
current = gdb.parse_and_eval("((struct list_head *) %s)->next" % head)
while (current != head):
modp = gdb.parse_and_eval("((struct module *) (((char *) %s) - %s))"
% (current, offset))
modp = "((struct module *) %s)" % modp
cname = get_str("%s->name" % modp)
if (cname == modname):
return modp
current = gdb.parse_and_eval("((struct list_head *) %s)->next" %
current)
return None class dumpltt(gdb.Command):
"""Dump an LTT coredump from the kernel image.""" def __init__(self):
super(dumpltt, self).__init__("dumpltt", gdb.COMMAND_USER)
def find_channel(self, head):
"""Find a channel in the list that is a per-cpu channel that is in snapshot mode (relay-overwrite-mmap) and return the first one found or None if not found."""
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % head) while (p != head):
chan = fetch_container(p, "struct lttng_channel", "list") enabled = gdb.parse_and_eval("%s->enabled" % chan)
if (int(enabled)):
chantype = gdb.parse_and_eval("%s->channel_type" % chan) if (chantype == self.PER_CPU_CHANNEL):
backend = gdb.parse_and_eval("&(%s->chan->backend)" % chan) backend = "((struct channel_backend *) %s)" % str(backend) tpname = get_str("%s->name" % backend)
if (tpname == "relay-overwrite-mmap"):
return backend
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return None
def find_session_channel(self):
"""Search the channels in the given session for one in snapshot mode."""
head = gdb.parse_and_eval("&sessions")
p = gdb.parse_and_eval("sessions.next")
while (p != head):
session = fetch_container(p, "struct lttng_session", "list") print "session: " + session
active = gdb.parse_and_eval("%s->active" % session) if (not int(active)):
continue
chanlist = gdb.parse_and_eval("&(%s->chan)" % session) chan = self.find_channel(chanlist)
if (chan is not None):
return chan
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return None
def add_alignment (self, cpu, size):
"""Dump 'size' bytes to the per-CPU output file.""" f = open("%s/channel0_%d" % (self.dumpdir, cpu), "ab") f.write("".ljust(size, chr(0)))
f.close()
return
def dump_page(self, cpu, page, start, size):
"""Dump 'size' bytes from the page starting at offset 'start' to the per-CPU output file."""
addr = long(gdb.parse_and_eval("%s->virt" % page)) + start end = addr + size - start
gdb.execute("append memory %s/channel0_%d %ld %ld" %
(self.dumpdir, cpu, addr, end))
return
def dump_subbuffer(self, cpu, rbbep, start, size):
"""Dump 'size' bytes from offset 'start' in the given subbuffer 'rbbep' to the per-CPU output file."""
start_page = start / self.page_size
start_offset = start % self.page_size
written = 0
for i in range(start_page, self.num_pages_per_subbuf):
bepage = get_ptr("&(%s->p[%d])" % (rbbep, i),
"struct lib_ring_buffer_backend_page")
if (size > self.page_size - start_offset):
self.dump_page(cpu, bepage, start_offset,
self.page_size - start_offset)
size -= self.page_size - start_offset written += self.page_size - start_offset start_offset = 0
else:
self.dump_page(cpu, bepage, start_offset, size) written += size
break
return written
def handle_subbuffer(self, cpu, buf, rbbe, cidx, coffset, length):
"""Extract necessary information from the subbuffer and dump it to a file.
buf is the ring buffer.
rbbe is a pointer to the buf->backend.buf_wsb array. cidx is the subbuffer index (for indexing into cidx). coffset is the offset within the subbuffer to start at. length is the number of bytes to dump from the subbuffer.
Note that each subbuffer has a header, and it may not be completely filled out. So add the subbuffer size and the size padded to a page to the header after writing it out.
"""
sid = long(gdb.parse_and_eval("%s[%d].id" % (rbbe, cidx))) sbidx = sid & self.idmask
rbbep = get_ptr("%s->backend.array[%d]" % (buf, sbidx),
"struct lib_ring_buffer_backend_pages")
size = int(gdb.parse_and_eval("%s->data_size" % rbbep)) last_subbuf_start = get_file_size("%s/channel0_%d"
% (self.dumpdir, cpu))
if (length > 0):
size = length
written_size = self.dump_subbuffer(cpu, rbbep, coffset, size); padded_size = align(written_size, self.page_size) self.add_alignment(cpu, padded_size - written_size)
print " subbuf size = %d (%d padding) %d" % (padded_size,
padded_size - written_size, size)
# Now fix the values in the header
f = open("%s/channel0_%d" % (self.dumpdir, cpu), "r+") f.seek(last_subbuf_start + 40, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", written_size * 8) f.write(val)
f.seek(last_subbuf_start + 48, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", padded_size * 8) f.write(val)
f.close()
def handle_buffer(self, cpu, buf):
"""Dump the given ring buffer to the given cpu's output file.""" print "Handling cpu %d buffer %s" % (cpu, buf)
offset = int(gdb.parse_and_eval("%s->offset.v" % buf)) oidx = (offset / self.subbuf_size) % self.num_subbuf ooffset = offset % self.subbuf_size
consumed = int(gdb.parse_and_eval("%s->consumed.counter" % buf)) cidx = (consumed / self.subbuf_size) % self.num_subbuf coffset = consumed % self.subbuf_size
rbbe = get_ptr("%s->backend.buf_wsb" % buf,
"struct lib_ring_buffer_backend_subbuffer")
subbuf_num = 1;
while (oidx != cidx):
print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, 0) cidx = (cidx + 1) % self.num_subbuf
coffset = 0
subbuf_num += 1
print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, ooffset - coffset) return
def invoke(self, args, from_tty):
argv = gdb.string_to_argv(args)
if (len(argv) != 2):
raise gdb.GdbError("dumpltt <lttdir> <dumpdir>")
lttdir = argv[0]
self.dumpdir = argv[1]
os.system("rm -rf %s/channel*" % self.dumpdir) os.system("rm -rf %s/dummy" % self.dumpdir)
modp = find_module("lttng_tracer")
if (modp is None):
raise gdb.GdbError("Unable to find lttng_tracer module, are you "
"sure lttng is loaded in this kernel dump?")
gdb.execute("add-symbol-file %s/lttng-tracer.o %s -s .data %s" %
(lttdir,
gdb.parse_and_eval("(unsigned long) (%s->module_core_rx)"
% modp),
gdb.parse_and_eval("(unsigned long) (%s->module_core_rw)"
% modp)))
self.PER_CPU_CHANNEL = gdb.parse_and_eval("PER_CPU_CHANNEL")
backend = self.find_session_channel()
if (backend is None):
raise gdb.GdbError("Unable to find operating session, are you sure "
"lttng was running with --snapshot?")
print "Backend: " + backend
self.num_subbuf = gdb.parse_and_eval("%s->num_subbuf" % backend); self.subbuf_size = gdb.parse_and_eval("%s->subbuf_size" % backend); p = get_percpu_ptr("%s->buf" % backend, 0, "struct lib_ring_buffer"); self.num_pages_per_subbuf = gdb.parse_and_eval(
"%s->backend.num_pages_per_subbuf" % p);
self.page_size = self.subbuf_size / self.num_pages_per_subbuf self.idshift = int(gdb.parse_and_eval("sizeof(long)")) * 8 / 2 self.idmask = (1 << self.idshift) - 1
cpus = get_cpu_list()
for cpu in cpus:
p = get_percpu_ptr("%s->buf" % backend, cpu,
"struct lib_ring_buffer");
self.handle_buffer(cpu, p)dumpltt()
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
----- Original Message -----
Quoted 28 lines
From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Thursday, February 13, 2014 2:31:13 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpQuoted 14 lines · Mathieu Desnoyers
You can use the flight recorder mode in recent LTTng for this (2.3 and newer). It simply writes to memory, without any output. I understand that you want to create a contiguous ring buffer memory layout. However, you have to be aware that this will probably be done using either
a) statically allocated memory at boot time (not very flexible),
b)vmalloc()(very flexible, but can triggers minor page faults, which can interact badly with page fault instrumentation.vmalloc()space is often limited by a kernel boot time parameter, and is putting quite important limitations on systems with 32-bit address spaces).Quoted 3 lines
But I am certainly open to suggestions on how to do this, and happy to have anything included back into the mainline.
Quoted 6 lines
Quoted 1 line
And I'm still learning about the internals of LTT.
One option would be to modify the tool to understand the LTTng 2.x buffer layout by stitching pages together by software using the LTTng libringbuffer "subbuffer table". You can think of it as a 2-level page table, but one level indexes the sub-buffers, and the next level indexes the pages within a sub-buffer.
Quoted 7 lines
I'm finally back to this. I discovered that
/proc/vmcoredid not map vmalloc-ed memory, so I had to come up with something to handle that before I could continue to work on this.This information was very useful, and snapshot mode is definitely the way to go. I just want to make sure I understand this before I go on. I have some specific question:
Sorry for the lag... ;)
I'll reply to your questions below just to make sure we're in sync.
Quoted 2 lines
consumed is where the data starts and offset is where the data ends.
Yes, approximately. This is right for consumed (this is where consumable data starts), but "offset" is either:
- where the data ends, or
- slightly beyond where the last contiguously committed data end.
It means we have to take extra care for buffers where the amount of commit_seq count is not a multiple of the sub-buffer size.
Quoted 3 lines
So
just go through the subbufs through a double index. Calculate the start/end location by taking the consumed/offset value,
First apply a module buffer size, otherwise the free running counters divided by subbuffer size will go beyond the backend.buf_wsb array.
Quoted 2 lines
dividing that by
the size of a subbuffer,
Yes,
Quoted 3 lines
looking up that value in backend.buf_wsb,
getting the index from the id there, then indexing into the backend.array with the index.
Yes.
Quoted 2 lines
Once you have the subbuffer, you mod the
location by the size of a subbuffer and that's the subbuffer offset.
Exactly.
Quoted 2 lines
Divide the subbuffer offset by a page size to get the page index in the subbuffer, and mod by a page size to get the offset into the page.
Yes.
Quoted 4 lines
Starting from the consumed position, dump data from pages until you hit
subbufer->data_size, then move to the next subbuffer. On the last subbuffer, you have to fill in the header and dump up to the offset.
Not quite exactly. It's better to do:
for each subbuffer between consumed and offset (inclusive)
- if commit_seq is multiple of subbuffer size
dump subbuffer->data_size
- if commit_seq is not a multiple of subbuffer size, dump commit_seq % subbuf size bytes of data, filling up the header accordingly.
Basically, all data that was "being written" (between commit_seq % subbuf size and write offset % subbuf size) cannot be read, because it likely contains holes and/or incomplete data.
Quoted 4 lines
I think I'm missing something, though, because the data size of the last subbuffer doesn't match the offset location in that subbuffer. It's a pretty good distance away.
Does my explanation above help clear things out ?
Thanks,
Mathieu
Quoted 81 lines
Thanks,
-corey
Quoted 74 lines
A good way to understand its layout is to look at:
lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c
lib_ring_buffer_backend_allocate()
lib/ringbuffer/backend_types.hstruct
lib_ring_buffer_backend
structlib_ring_buffer_backend_subbuffer
structlib_ring_buffer_backend_pages
structlib_ring_buffer_backend_pageIn your case, you never care about the
bufb->buf_rsb(read-side owned subbuffer),
because you always ever just write into it.buf_rsbis only useful when taking
snapshots.bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the
actual sub-buffers and memory pages associated to each buffer.You'll notice that the "id" field within struct
lib_ring_buffer_backend_subbuffer
is actually made of a mask of many fields. In order to understand how to use it,
see
lib/ringbuffer/backend_types.hwhere we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.
So you'll need to use the structures presented above to make sense of the memory
layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.The algorithm you want to end up doing (offline, on a vmcore) is pretty much
the same as grabbing an online snapshot (iterate from the consumer position up to
the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
You will need an extra trick to handle the sub-buffer that was being written to
at the time of the crash, by using the
lib/ringbuffer/frontend_types.hstructcommit_counters_hot"seq" fieldwhich is designed to track the contiguously committed data within the currently
written buffer. This can be used at any point in time (whenever a crash occurs) to
populate the last sub-buffer's content size, packet size, see:
lttng-ring-buffer-client.h:client_buffer_end()and find out how much of the last sub-buffer needs to be copied into the output
CTF trace.Thanks,
Mathieu
Quoted 4 lines
Thanks,
-corey
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Mathieu Desnoyers
Quoted 3 lines
Starting from the consumed position, dump data from pages until you hit
subbufer->data_size, then move to the next subbuffer. On the last subbuffer, you have to fill in the header and dump up to the offset.Not quite exactly. It's better to do:
for each subbuffer between consumed and offset (inclusive)
- ifcommit_seqis multiple of subbuffer sizedump subbuffer->data_size- if
commit_seqis not a multiple of subbuffer size, dumpcommit_seq% subbuf size bytes of data, filling up the header accordingly.
Quoted 8 lines
Basically, all data that was "being written" (between
commit_seq% subbuf size and write offset % subbuf size) cannot be read, because it likely contains holes and/or incomplete data.Quoted 3 lines
I think I'm missing something, though, because the data size of the last subbuffer doesn't match the offset location in that subbuffer. It's a pretty good distance away.
Does my explanation above help clear things out ?
Yes, I've modified the code a bit to use the hot commit data if necessary, and to dump the metadata from the metadata cache. A new version is attached.
I also modified to use the flushed value if present. The probabilities of that situation causing a problem are pretty low, but this will avoid any issue. It looks correct to me.
-corey
Quoted 83 lines
Thanks,
Mathieu
Quoted 79 lines
Thanks,
-corey
Quoted 74 lines
A good way to understand its layout is to look at:
lttng-modules (master)
lib/ringbuffer/ring_buffer_backend.c
lib_ring_buffer_backend_allocate()
lib/ringbuffer/backend_types.hstruct
lib_ring_buffer_backend
structlib_ring_buffer_backend_subbuffer
structlib_ring_buffer_backend_pages
structlib_ring_buffer_backend_pageIn your case, you never care about the
bufb->buf_rsb(read-side owned subbuffer),
because you always ever just write into it.buf_rsbis only useful when taking
snapshots.bufb->buf_wsb[] has the mapping from sub-buffers write-side index within the buffer to the associated index into bufb->array[], which allows getting the
actual sub-buffers and memory pages associated to each buffer.You'll notice that the "id" field within struct
lib_ring_buffer_backend_subbuffer
is actually made of a mask of many fields. In order to understand how to use it,
see
lib/ringbuffer/backend_types.hwhere we provide helpers to get and set the various information elements contained within the "id" field. See subbuffer_id*() functions and comments surrounding them.
So you'll need to use the structures presented above to make sense of the memory
layout of a buffer, and reorganize it into a CTF file that can be read by Babeltrace or other CTF trace readers.The algorithm you want to end up doing (offline, on a vmcore) is pretty much
the same as grabbing an online snapshot (iterate from the consumer position up to
the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
You will need an extra trick to handle the sub-buffer that was being written to
at the time of the crash, by using the
lib/ringbuffer/frontend_types.hstructcommit_counters_hot"seq" fieldwhich is designed to track the contiguously committed data within the currently
written buffer. This can be used at any point in time (whenever a crash occurs) to
populate the last sub-buffer's content size, packet size, see:
lttng-ring-buffer-client.h:client_buffer_end()and find out how much of the last sub-buffer needs to be copied into the output
CTF trace.Thanks,
Mathieu
Quoted 4 lines
Thanks,
-corey
# gdb macros for extracting an LTT trace from a kernel coredump # Copyright (C) 2014 Montavista Software <[email protected]> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
# The layout of the LTT buffers in memory is rather complex. But here's
# the data structures involved: # # sessions - a list of running sessions of type struct lttng_session. # A session consists of a some generic information and a list of # channels for the session. # # session->chan - a list of channels of type struct lttng_channel. The # main information of interest is in chan.backend. # # This code parses the sessions and channels until it finds the first # active non-metadata session/channel running in snapshot # (relay-overwrite-mmap) mode. # # The chan.backend has some useful information about data sizes and a # percpu array of ring buffers. Each CPU logs to its ring buffer and # the downstream tools pull all the data together. The dump process # creates a separate file for each CPU, just like the normal tools do. # # chan.backend->buf - A percpu array of structures of type struct # lib_ring_buffer. This is more or less the heart of the data # structure. Like channels, lib_ring_buffer has a backend field that # holds most of the useful data. # # buf->backend.array - An array of pointers to subbuffers. A # subbuffer itself is an array of pointers to pages, and each page in # order holds data. LTT elements do not span subbuffers, so the # actual size of the data in the subbuffer may be smaller than the # full subbuffer size. subbuf->data_size holds the actual data size in # the subbuffer. # # buf->backend.buf_wsb - The subbuffers may not be in order. This array # holds the order of the subbuffers. # # chan->backend.num_subbuf - The number of subbuffers. # # chan->backend.subbuf_size - The size of each subbuffer, in bytes. # # So at this point, you can trace down any index by taking the index, # dividing it by num_subbuf to get the subbuffer, taking that and # doing a modulo on num_subbuf(*), indexing into buf->backend.buf_wsb # to get the actual subbuffer number, then indexing into # buf->backend.array to get a pointer to the subbuffer. Then take the # original index, modulo it by the number of subbuffers to get a # subbuffer offset. Dividing the subbuffer offset by the number of # pages in a subbuffer gives the page in the subbuffer, and modulo the # subbuffer offset by the page size to get the offset into the page. # The page data structure has a virtual pointer so the data can be # directly accessed. # # (*) the offset and consumed values do not wrap at the total output size, # so you must modulo the subbuffer index to get the proper value. # # buf->offset - The index of the end of the data. # buf->consumed - The index to the start of data. # # So the basic algorithm is to index down to buf->consumed and start dumping # pages until buf->offset is reached. # # The command is invoked as: # dumpltt <lttdir> <outputdir> # where lttdir is the place where the ltt modules are compiled and outputdir # is the place where you want to put the trace output. # # The lttdir is required because this code loads the symbol tables # from the ltt modules so it can access the various data structures in # ltt. It will not work without that.
import os
def calc_offset(struct, member):
"""Calculate the offset in the struct to the given member.""" return gdb.parse_and_eval("((int)&((%s *)0).%s)" % (struct, member))
def fetch_container(ptr, struct, lmember):
"""Fetch a pointer to the given structure given a pointer to one of
its members. This is generally used to get the container of a list
element."""
offset = calc_offset(struct, lmember)
s = gdb.parse_and_eval("(void *) (((unsigned long) %s) - %s)" %
(ptr, offset))
return "((%s *) %s)" % (struct, s)def get_ptr(ptr, struct):
"""Return a string that can be used to reference the given type (struct) at the given address (ptr)."""
s = gdb.parse_and_eval("(void *) %s" % ptr)
return "((%s *) %s)" % (struct, s)
def get_str(p):
"""Get a string value from the given target address"""
s = gdb.parse_and_eval("(char *) %s" % p)
return str(s).split('"')[1]def align(v, alignment):
"""Align the given integer to the given alignment""" return (v + (alignment - 1)) & ~(alignment - 1)
def get_file_size(fname):
"""Return the size of the given file"""
f = open(fname, "a")
size = f.tell()
f.close()
return sizedef value_to_binary(tmpdir, typestr, v):
"""Convert an expression to a binary value. The size depends on the typestr given. Return a string with the raw binary value in target-endian order"""
fn = "%s/dummy" % tmpdir
gdb.execute("append value %s ((%s) %ld)" % (fn, typestr, long(v))) f = open(fn, "r")
str = f.read()
f.close()
os.unlink(fn)
return str
def get_cpu_list():
"""Return a list of online CPUs on the target."""
try:
ulongsize = gdb.parse_and_eval("sizeof(unsigned long)") nr_cpumask = gdb.parse_and_eval("sizeof(*cpu_online_mask)") nr_cpumask = int(nr_cpumask) / int(ulongsize)
cpu = 0;
cpus = []
for i in range(0, nr_cpumask):
mask = gdb.parse_and_eval("cpu_online_mask->bits[%d]" % i) for j in range(0, ulongsize * 8):
if ((mask & 1) != 0):
cpus.append(cpu)
mask = mask >> 1
cpu = cpu + 1
except e:
cpus = [ 0 ]
return cpus
def get_percpu_ptr(p, cpu, typename):
"""Convert a percpu pointer 'p' on the given cpu to a pointer to the
given type."""
try:
offset = gdb.parse_and_eval("__per_cpu_offset[%d]" % cpu)
except:
offset = "0";
v = gdb.parse_and_eval("(void *) %s + %s" % (p, offset))
return "((%s *) %s)" % (typename, v)def get_member_ptr(p, membtype, member):
"""Given a pointer to a structure, get the given member as a pointer to another structure."""
v = gdb.parse_and_eval("%s->%s" % (p, member))
return "((%s *) %s)" % (membtype, str(v))
def get_member_long(p, member):
"""Given a pointer to a structure, get the given member that is an
integer."""
v = gdb.parse_and_eval("(long) (%s->%s)" % (p, member))
return long(v)def find_module(modname):
"""Return a pointer to the given module name's data structure"""
offset = gdb.parse_and_eval("((int)&((struct module *)0).list)")
head = gdb.parse_and_eval("modules").address
head = head.cast(gdb.lookup_type("unsigned long"))
current = gdb.parse_and_eval("((struct list_head *) %s)->next" % head)
while (current != head):
modp = gdb.parse_and_eval("((struct module *) (((char *) %s) - %s))"
% (current, offset))
modp = "((struct module *) %s)" % modp
cname = get_str("%s->name" % modp)
if (cname == modname):
return modp
current = gdb.parse_and_eval("((struct list_head *) %s)->next" %
current)
return None class dumpltt(gdb.Command):
"""Dump an LTT coredump from the kernel image.""" def __init__(self):
super(dumpltt, self).__init__("dumpltt", gdb.COMMAND_USER)
def find_channel(self, head):
"""Find a channel in the list that is a per-cpu channel that is in snapshot mode (relay-overwrite-mmap) and return the first one found or None if not found."""
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % head) while (p != head):
chan = fetch_container(p, "struct lttng_channel", "list") enabled = gdb.parse_and_eval("%s->enabled" % chan)
if (int(enabled)):
chantype = gdb.parse_and_eval("%s->channel_type" % chan) if (chantype == self.PER_CPU_CHANNEL):
backend = gdb.parse_and_eval("&(%s->chan->backend)" % chan) backend = "((struct channel_backend *) %s)" % str(backend) tpname = get_str("%s->name" % backend)
if (tpname == "relay-overwrite-mmap"):
return backend
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return None
def find_session_channel(self):
"""Search the channels in the given session for one in snapshot mode."""
head = gdb.parse_and_eval("&sessions")
p = gdb.parse_and_eval("sessions.next")
while (p != head):
session = fetch_container(p, "struct lttng_session", "list") active = gdb.parse_and_eval("%s->active" % session) if (not int(active)):
continue
chanlist = gdb.parse_and_eval("&(%s->chan)" % session) chan = self.find_channel(chanlist)
if (chan is not None):
return (session, chan)
p = gdb.parse_and_eval("((struct list_head *) %s)->next" % p)
return (None, None)
def add_alignment (self, cpu, size):
"""Dump 'size' bytes to the per-CPU output file.""" f = open("%s/channel0_%d" % (self.dumpdir, cpu), "ab") f.write("".ljust(size, chr(0)))
f.close()
return
def dump_page(self, cpu, page, start, size):
"""Dump 'size' bytes from the page starting at offset 'start' to the per-CPU output file."""
addr = long(gdb.parse_and_eval("%s->virt" % page)) + start end = addr + size - start
gdb.execute("append memory %s/channel0_%d %ld %ld" %
(self.dumpdir, cpu, addr, end))
return
def dump_subbuffer(self, cpu, rbbep, start, size):
"""Dump 'size' bytes from offset 'start' in the given subbuffer 'rbbep' to the per-CPU output file."""
start_page = start / self.page_size
start_offset = start % self.page_size
written = 0
for i in range(start_page, self.num_pages_per_subbuf):
bepage = get_ptr("&(%s->p[%d])" % (rbbep, i),
"struct lib_ring_buffer_backend_page")
if (size > self.page_size - start_offset):
self.dump_page(cpu, bepage, start_offset,
self.page_size - start_offset)
size -= self.page_size - start_offset written += self.page_size - start_offset start_offset = 0
else:
self.dump_page(cpu, bepage, start_offset, size) written += size
break
return written
def handle_subbuffer(self, cpu, buf, rbbe, cidx, coffset, length):
"""Extract necessary information from the subbuffer and dump it to a file.
buf is the ring buffer.
rbbe is a pointer to the buf->backend.buf_wsb array. cidx is the subbuffer index (for indexing into cidx). coffset is the offset within the subbuffer to start at. length is the number of bytes to dump from the subbuffer.
Note that each subbuffer has a header, and it may not be completely filled out. So add the subbuffer size and the size padded to a page to the header after writing it out.
"""
sid = long(gdb.parse_and_eval("%s[%d].id" % (rbbe, cidx))) sbidx = sid & self.idmask
rbbep = get_ptr("%s->backend.array[%d]" % (buf, sbidx),
"struct lib_ring_buffer_backend_pages")
size = get_member_long(buf, "commit_hot[%d].seq.v" % (cidx)) if ((size & (self.subbuf_size - 1)) == 0):
# Hot value is a multiple of the subbuffer size, use the # buffer data size.
size = int(gdb.parse_and_eval("%s->data_size" % rbbep))
else:
size = size % self.subbuf_size
last_subbuf_start = get_file_size("%s/channel0_%d"
% (self.dumpdir, cpu))
if (length > 0 and length < size):
size = length
written_size = self.dump_subbuffer(cpu, rbbep, coffset, size); padded_size = align(written_size, self.page_size) self.add_alignment(cpu, padded_size - written_size)
print " subbuf size = %d (pad %d) %d" % (padded_size,
padded_size - written_size, size)
# Now fix the values in the header
f = open("%s/channel0_%d" % (self.dumpdir, cpu), "r+") f.seek(last_subbuf_start + 40, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", written_size * 8) f.write(val)
f.seek(last_subbuf_start + 48, os.SEEK_SET)
val = value_to_binary(self.dumpdir, "uint64_t", padded_size * 8) f.write(val)
f.close()
def handle_buffer(self, cpu, buf):
"""Dump the given ring buffer to the given cpu's output file.""" print "Handling cpu %d buffer %s" % (cpu, buf)
offset = int(gdb.parse_and_eval("%s->offset.v" % buf)) oidx = (offset / self.subbuf_size) % self.num_subbuf ooffset = offset % self.subbuf_size
consumed = int(gdb.parse_and_eval("%s->consumed.counter" % buf)) cidx = (consumed / self.subbuf_size) % self.num_subbuf coffset = consumed % self.subbuf_size
rbbe = get_ptr("%s->backend.buf_wsb" % buf,
"struct lib_ring_buffer_backend_subbuffer")
subbuf_num = 1;
while (oidx != cidx):
print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, 0) cidx = (cidx + 1) % self.num_subbuf
coffset = 0
subbuf_num += 1
print " subbuffer %d" % subbuf_num self.handle_subbuffer(cpu, buf, rbbe, cidx, coffset, ooffset - coffset) return
def dump_metadata(self, session):
f = open("%s/metadata" % (self.dumpdir), "w") f.write("/* CTF 1.8 */\n")
f.close()
mdcache = get_member_ptr(session, "struct lttng_metadata_cache",
"metadata_cache")
data = get_member_long(mdcache, "data")
size = get_member_long(mdcache, "metadata_written") gdb.execute("append memory %s/metadata %ld %ld" %
(self.dumpdir, data, data + size))
def invoke(self, args, from_tty):
argv = gdb.string_to_argv(args)
if (len(argv) != 2):
raise gdb.GdbError("dumpltt <lttdir> <dumpdir>")
lttdir = argv[0]
self.dumpdir = argv[1]
os.system("mkdir -p %s" % self.dumpdir)
os.system("rm -rf %s/channel*" % self.dumpdir) os.system("rm -rf %s/dummy" % self.dumpdir)
modp = find_module("lttng_tracer")
if (modp is None):
raise gdb.GdbError("Unable to find lttng_tracer module, are you "
"sure lttng is loaded in this kernel dump?")
gdb.execute("add-symbol-file %s/lttng-tracer.o %s -s .data %s" %
(lttdir,
gdb.parse_and_eval("(unsigned long) (%s->module_core_rx)"
% modp),
gdb.parse_and_eval("(unsigned long) (%s->module_core_rw)"
% modp)))
self.PER_CPU_CHANNEL = gdb.parse_and_eval("PER_CPU_CHANNEL")
(session, backend) = self.find_session_channel()
if (backend is None):
raise gdb.GdbError("Unable to find operating session, are you sure "
"lttng was running with --snapshot?")
self.dump_metadata(session)
self.num_subbuf = gdb.parse_and_eval("%s->num_subbuf" % backend); self.subbuf_size = gdb.parse_and_eval("%s->subbuf_size" % backend); p = get_percpu_ptr("%s->buf" % backend, 0, "struct lib_ring_buffer"); self.num_pages_per_subbuf = gdb.parse_and_eval(
"%s->backend.num_pages_per_subbuf" % p);
self.page_size = self.subbuf_size / self.num_pages_per_subbuf self.idshift = int(gdb.parse_and_eval("sizeof(long)")) * 8 / 2 self.idmask = (1 << self.idshift) - 1
cpus = get_cpu_list()
for cpu in cpus:
p = get_percpu_ptr("%s->buf" % backend, cpu,
"struct lib_ring_buffer");
self.handle_buffer(cpu, p)dumpltt()
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
----- Original Message -----
Quoted 8 lines
From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpI have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.
Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.
Quoted 6 lines
I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.
Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).
You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at
/lttng-events.h
struct lttng_session {
...
struct lttng_metadata_cache *metadata_cache;
...
};
Which has the metadata string:
struct lttng_metadata_cache {
char *data; /* Metadata cache */
unsigned int cache_alloc; /* Metadata allocated size (bytes) */
unsigned int metadata_written; /* Number of bytes written in metadata cache */
struct kref refcount; /* Metadata cache usage */ struct list_head metadata_stream; /* Metadata stream list */
};
You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.
If you happen to core dump while the metadata cache is being resized, you might encounter a corrupted metadata. It should not happen frequently, but it's possible. We could eventually change lttng_metadata_printf() so it makes sure to always keep a readable metadata around (by e.g. using kcalloc rather than krealloc and dealing carefully with cache_alloc vs data fields updates).
Thoughts ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 8 lines
From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpI have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.
Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.
Yes, I'm going through that and the code I have to get things clean.
Quoted 35 lines
Quoted 5 lines
I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.
Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).
You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at
/lttng-events.h
struct
lttng_session{
...
structlttng_metadata_cache*metadata_cache;
...
};Which has the metadata string:
struct
lttng_metadata_cache{char *data; /* Metadata cache */ unsigned int cache_alloc; /* Metadata allocated size (bytes) */ unsigned int metadata_written; /* Number of bytes written in metadata cache */ struct kref refcount; /* Metadata cache usage */struct
list_headmetadata_stream; /* Metadata stream list */
};You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.
Quoted 7 lines
If you happen to core dump while the metadata cache is being resized, you might encounter a corrupted metadata. It should not happen frequently, but it's possible. We could eventually change
lttng_metadata_printf()so it makes sure to always keep a readable metadata around (by e.g. using kcalloc rather than krealloc and dealing carefully withcache_allocvs data fields updates).Thoughts ?
Perfect, that's easy. I'll do that.
Thanks again.
-corey
----- Original Message -----
Quoted 57 lines
From: "Mathieu Desnoyers" <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Tuesday, February 18, 2014 4:02:15 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump----- Original Message -----
Quoted 8 lines
From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpI have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.
Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.
Quoted 6 lines
I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.
Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).
You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at
/lttng-events.h
struct
lttng_session{
...
structlttng_metadata_cache*metadata_cache;
...
};Which has the metadata string:
struct
lttng_metadata_cache{char *data; /* Metadata cache */ unsigned int cache_alloc; /* Metadata allocated size (bytes) */ unsigned int metadata_written; /* Number of bytes written inmetadata cache */
struct kref refcount; /* Metadata cache usage */
structlist_headmetadata_stream; /* Metadata stream list */
};You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.
Quoted 8 lines
If you happen to core dump while the metadata cache is being resized, you might
encounter a corrupted metadata. It should not happen frequently, but it's possible.
We could eventually changelttng_metadata_printf()so it makes sure to always keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
dealing carefully withcache_allocvs data fields updates).
Oops, I meant "kzalloc".
I'm attaching a patch that should take care of making sure it is always in a valid state when a core dump occurs. You should be able to safely read "metadata_flushed" bytes of data at any time. Whatever has been flushed should be parseable by Babeltrace.
Thoughts ?
Thanks,
Mathieu
Quoted 17 lines
Thoughts ?
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
commit 875fbbbe6167b3847cdf4de925722be9dcf685cc
Author: Mathieu Desnoyers <[email protected]> Date: Tue Feb 18 16:35:05 2014 -0500
Ensure metadata can be extracted from core dumps at any time
lttng-events.c +65 -18
@@ -603,7 +603,7 @@ int lttng_metadata_output_channel(struct lttng_metadata_stream *stream,if (stream->metadata_in != stream->metadata_out)return 0 ; - len = stream->metadata_cache->metadata_written - + len = stream->metadata_cache->metadata_flushed - stream->metadata_in;if (!len)return 0 ;
@@ -631,17 +631,41 @@ end:return ret; } +static +void lttng_metadata_flush (struct lttng_session *session) +{ +struct lttng_metadata_cache *cache = session->metadata_cache; +struct lttng_metadata_stream *stream; + +if (cache->metadata_written == cache->metadata_flushed) +return ; + +/* +* Append to metadata before incrementing "metadata flushed", +* ensuring that core dumps always read up to "metadata flushed" +* bytes of valid metadata. +*/ +barrier (); + cache->metadata_flushed = cache->metadata_written; + +list_for_each_entry (stream, &cache->metadata_stream, list) +wake_up_interruptible (&stream->read_wait); +} +/* * Write the metadata to the metadata cache. * Must be called with sessions_mutex held. +* Grow the metadata cache with allocation, copy, populating the new +* structure, and then free the old structure, to ensure that there is +* always a consistent state to grab from kernel core dump. */ +static int lttng_metadata_printf (struct lttng_session *session,const char *fmt, ...) {char *str;size_t len;va_list ap; -struct lttng_metadata_stream *stream;WARN_ON_ONCE (!ACCESS_ONCE (session->active));
@@ -654,18 +678,33 @@ int lttng_metadata_printf(struct lttng_session *session, len =strlen (str);if (session->metadata_cache->metadata_written + len > session->metadata_cache->cache_alloc) { -char *tmp_cache_realloc; -unsigned int tmp_cache_alloc_size; - - tmp_cache_alloc_size = max_t(unsigned int, - session->metadata_cache->cache_alloc + len, - session->metadata_cache->cache_alloc <<1 ); - tmp_cache_realloc =krealloc (session->metadata_cache->data, - tmp_cache_alloc_size,GFP_KERNEL ); -if (!tmp_cache_realloc) +char *new_cache, *old_cache; +unsigned int new_cache_alloc_size, old_cache_alloc_size; + + old_cache_alloc_size = session->metadata_cache->cache_alloc; + new_cache_alloc_size = max_t(unsigned int, + old_cache_alloc_size + len, + old_cache_alloc_size <<1 ); + new_cache =kzalloc (new_cache_alloc_size,GFP_KERNEL ); +if (!new_cache) goto err; - session->metadata_cache->cache_alloc = tmp_cache_alloc_size; - session->metadata_cache->data = tmp_cache_realloc; + session->metadata_cache->cache_alloc = new_cache_alloc_size; + old_cache = session->metadata_cache->data; +memcpy (new_cache, old_cache, old_cache_alloc_size); +/* +* Copy the old cache into the new one before storing +* the pointer to new cache into "data", so core dumps +* read valid metadata. +*/ +barrier (); + session->metadata_cache->data = new_cache; +/* +* Store "data" before freeing the old cache, thus +* ensuring kernel core dumps always read up to +* "metadata_flushed" bytes of valid metadata. +*/ +barrier (); +kfree (old_cache); } memcpy(session->metadata_cache->data + session->metadata_cache->metadata_written,
@@ -673,9 +712,6 @@ int lttng_metadata_printf(struct lttng_session *session, session->metadata_cache->metadata_written += len;kfree (str); -list_for_each_entry (stream, &session->metadata_cache->metadata_stream, list) -wake_up_interruptible (&stream->read_wait); -return 0 ; err:
@@ -906,6 +942,7 @@ int _lttng_event_metadata_statedump(struct lttng_session *session, event->metadata_dumped =1 ; end: +lttng_metadata_flush (session);return ret; }
@@ -958,6 +995,7 @@ int _lttng_channel_metadata_statedump(struct lttng_session *session, chan->metadata_dumped =1 ; end: +lttng_metadata_flush (session);return ret; }
@@ -967,7 +1005,9 @@ end:static int _lttng_stream_packet_context_declare (struct lttng_session *session) { -return lttng_metadata_printf (session, +int ret; + + ret =lttng_metadata_printf (session,"struct packet_context {\n" " uint64_clock_monotonic_t timestamp_begin;\n" " uint64_clock_monotonic_t timestamp_end;\n"
@@ -977,6 +1017,8 @@ int _lttng_stream_packet_context_declare(struct lttng_session *session)" uint32_t cpu_id;\n" "};\n\n" ); +lttng_metadata_flush (session); +return ret; }/*
@@ -993,7 +1035,9 @@ int _lttng_stream_packet_context_declare(struct lttng_session *session)static int _lttng_event_header_declare (struct lttng_session *session) { -return lttng_metadata_printf (session, +int ret; + + ret =lttng_metadata_printf (session,"struct event_header_compact {\n" " enum : uint5_t { compact = 0 ... 30, extended = 31 } id;\n" " variant <id> {\n"
@@ -1022,6 +1066,8 @@ int _lttng_event_header_declare(struct lttng_session *session)lttng_alignof (uint32_t) *CHAR_BIT ,lttng_alignof (uint16_t) *CHAR_BIT ); +lttng_metadata_flush (session); +return ret; }/*
@@ -1209,6 +1255,7 @@ skip_session: } session->metadata_dumped =1 ; end: +lttng_metadata_flush (session);return ret; }
lttng-events.h +1 -0
@@ -326,6 +326,7 @@ struct lttng_metadata_cache {char *data;/* Metadata cache */ unsigned int cache_alloc;/* Metadata allocated size (bytes) */ unsigned int metadata_written;/* Number of bytes written in metadata cache */ +unsigned int metadata_flushed;/* Number of bytes flushed in metadata cache */ struct kref refcount;/* Metadata cache usage */ struct list_head metadata_stream;/* Metadata stream list */ };
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
----- Original Message -----
Quoted 75 lines
From: "Mathieu Desnoyers" <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Tuesday, February 18, 2014 4:42:06 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump----- Original Message -----
Quoted 58 lines
From: "Mathieu Desnoyers" <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Tuesday, February 18, 2014 4:02:15 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump----- Original Message -----
Quoted 8 lines
From: "Corey Minyard" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "David Goulet" <[email protected]>, [email protected]
Sent: Tuesday, February 18, 2014 2:29:23 PM
Subject: Re: [lttng-dev] Extract lttng trace from kernel coredumpI have attached some gdb macros for dumping LTT buffers from a kernel coredump. I haven't done a lot of testing, though.
Very cool! I'll have a look when things get less busy here. I hope my earlier reply helps clearing things out.
Quoted 6 lines
I do have one question. I've been getting the metadata by taking a snapshot right when the trace starts and saving that to stick into the trace output from the coredump. Is that the best way, or is there some other way I can extract it? There doesn't seem to be a metadata channel running in snapshot mode.
Actually, it's better if you grab the metadata near when you grab the coredump. Grabbing the metadata at the beginning of your tracing session will not have the info about other modules that might be coming and going while tracing is active (which might contain tracepoints).
You could grab the metadata from the coredump actually, this would be the best approach. Looking at lttng modules 2.4 rc, you will want to look at
/lttng-events.h
struct
lttng_session{
...
structlttng_metadata_cache*metadata_cache;
...
};Which has the metadata string:
struct
lttng_metadata_cache{
char *data; /* Metadata cache */
unsigned intcache_alloc; /* Metadata allocated size (bytes)
*/
unsigned intmetadata_written; /* Number of bytes written in metadata cache */
struct kref refcount; /* Metadata cache usage */
structlist_headmetadata_stream; /* Metadata stream list */
};You simply need to dump this metadata string into a "metadata" file, add a "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only CTF metadata", which can be read by babeltrace.
Quoted 9 lines
If you happen to core dump while the metadata cache is being resized, you might
encounter a corrupted metadata. It should not happen frequently, but it's possible.
We could eventually changelttng_metadata_printf()so it makes sure to always
keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
dealing carefully withcache_allocvs data fields updates).
Quoted 6 lines
Oops, I meant "kzalloc".
I'm attaching a patch that should take care of making sure it is always in a valid state when a core dump occurs. You should be able to safely read "
metadata_flushed" bytes of data at any time. Whatever has been flushed should be parseable by Babeltrace.
Updated patch (fixing poll).
Thanks,
Mathieu
Quoted 35 lines
Thoughts ?
Thanks,
Mathieu
Quoted 12 lines
Thoughts ?
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.comQuoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
commit 94d1517096f2bcf01cdfdccbfb4030dd62d8bed4
Author: Mathieu Desnoyers <[email protected]> Date: Tue Feb 18 16:35:05 2014 -0500
Ensure metadata can be extracted from core dumps at any time
lttng-abi.c +1 -1
@@ -541,7 +541,7 @@ unsigned int lttng_metadata_ring_buffer_poll(struct file *filp,if (finalized) mask |=POLLHUP ; -if (stream->metadata_cache->metadata_written > +if (stream->metadata_cache->metadata_flushed > stream->metadata_out) mask |=POLLIN ; }
lttng-events.c +65 -18
160 lines not on this page · raw message
lttng-events.h +1 -0
@@ -326,6 +326,7 @@ struct lttng_metadata_cache {char *data;/* Metadata cache */ unsigned int cache_alloc;/* Metadata allocated size (bytes) */ unsigned int metadata_written;/* Number of bytes written in metadata cache */ +unsigned int metadata_flushed;/* Number of bytes flushed in metadata cache */ struct kref refcount;/* Metadata cache usage */ struct list_head metadata_stream;/* Metadata stream list */ };
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev