"No space left on device" results in corrupt trace data set
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Paul,
The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...
To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)
Cheers!
David
Woegerer, Paul:
Quoted 33 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Thanks,
Paul
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · * David Goulet ([email protected]) wrote:
Hi Paul,
The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...
To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)
live support will eventually allow to side-track this whole issue by flushing metadata to disk periodically during tracing,
Thanks,
Mathieu
Quoted 43 lines
Cheers!
DavidWoegerer, Paul:
Quoted 24 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
Quoted 9 lines
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Thanks,
Paul_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): 20121116153029.GB23798@Krystal
Quoted 8 lines · Mathieu Desnoyers
Quoted 7 lines · * David Goulet ([email protected]) wrote:
Hi Paul,
The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...
Quoted 7 lines
Quoted 4 lines
To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)
live support will eventually allow to side-track this whole issue by flushing metadata to disk periodically during tracing,
Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?
Thanks
Paul
Quoted 49 lines
Thanks,
Mathieu
Quoted 43 lines
Cheers!
DavidWoegerer, Paul:
Quoted 24 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
Quoted 9 lines
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Thanks,
Paul_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 10 lines
Paul Woegerer | SW Development Engineer http://go.mentor.com/sourceryanalyzer Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 14 lines · Paul
Quoted 13 lines · Mathieu Desnoyers
Quoted 11 lines · * David Goulet ([email protected]) wrote:
Hi Paul,
The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...
To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)
Quoted 5 lines
Quoted 2 lines
live support will eventually allow to side-track this whole issue by flushing metadata to disk periodically during tracing,
Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?
The main change of lttng-tools 2.2 is the live tracing feature. The way it is implemented in the lttngtop-live branch is more a proof-of-concept, we will be starting the clean implementation discussions here as soon as we have lttng-tools 2.1 released (which should be around the end of this month).
In the mean time, to solve the problem you are currently experiencing, you could manually add a flush on the metadata channel after the session is started to make sure it is written on disk quickly (that's what is done in lttngtop-live at the beginning). Of course like David said, if you have new events/applications enabled during tracing, the metadata needs to be flushed again.
Thanks,
Julien
Quoted 55 lines
Thanks
PaulQuoted 49 lines
Thanks,
Mathieu
Quoted 43 lines
Cheers!
DavidWoegerer, Paul:
Quoted 24 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
Quoted 9 lines
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Thanks,
Paul_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Julien Desfossez
Quoted 2 lines · Paul
Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?
The main change of lttng-tools 2.2 is the live tracing feature. The way it is implemented in the lttngtop-live branch is more a proof-of-concept, we will be starting the clean implementation discussions here as soon as we have lttng-tools 2.1 released (which should be around the end of this month).
Quoted 4 lines
In the mean time, to solve the problem you are currently experiencing, you could manually add a flush on the metadata channel after the session is started to make sure it is written on disk quickly (that's what is done in lttngtop-live at the beginning). Of course like David said, if
I will try that. Thanks !
--
Paul
Quoted 63 lines
you have new events/applications enabled during tracing, the metadata needs to be flushed again.
Thanks,
Julien
Quoted 55 lines
Thanks
PaulQuoted 49 lines
Thanks,
Mathieu
Quoted 43 lines
Cheers!
DavidWoegerer, Paul:
Quoted 24 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
Quoted 9 lines
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Thanks,
Paul_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 10 lines
Paul Woegerer | SW Development Engineer http://go.mentor.com/sourceryanalyzer Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
Quoted 8 lines · Julien Desfossez
Quoted 2 lines
Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?
The main change of lttng-tools 2.2 is the live tracing feature. The way it is implemented in the lttngtop-live branch is more a proof-of-concept, we will be starting the clean implementation discussions here as soon as we have lttng-tools 2.1 released (which should be around the end of this month).
Hi Julien,
I looked for "live tracing" related commits in the git repos but could find anything. There is also no RFC yet under:
http://git.lttng.org/?p=lttng-tools.git;a=tree;f=doc/proposals;hb=HEAD
Does development happen on some branch that I missed or did it simply not start yet.
Thanks,
Paul
Signature · 10 lines
Paul Woegerer | SW Development Engineer http://go.mentor.com/sourceryanalyzer Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
It could be interesting to at least reserve the size for the first packet, so we have at least a little bit of information on the trace.
Quoted 53 lines · David Goulet
Hi Paul,
The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...
To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)
Cheers!
DavidWoegerer, Paul:
Quoted 24 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
Quoted 9 lines
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Thanks,
Paul_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Yannick Brosseau
It could be interesting to at least reserve the size for the first packet, so we have at least a little bit of information on the trace.
I tried that with fallocate but unfortunately tmpfs does not seem to support fallocating files :(
--
Paul
Quoted 62 lines · David Goulet
Quoted 45 lines
Hi Paul,
The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...
To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)
Cheers!
DavidWoegerer, Paul:
Quoted 29 lines
Hi,
embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).
Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:
PERROR: Error in file write: No space left on device [in
lttng_consumer_on_read_subbuffer_mmap()at consumer.c:1367]The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).
(quoted diff) +0
-3-rwxrwxrwx 1 testuser users 0 Nov 16 11:22 metadata -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0 -rwxrwxrwx 1 testuser users 4096 Nov 16 11:22 myevents_1Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?
I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?
Any ideas ?
Quoted 8 lines
Quoted 4 lines
Thanks,
Paul_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 10 lines
Paul Woegerer | SW Development Engineer http://go.mentor.com/sourceryanalyzer Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.