Skip to contentSkip to conversation

"No space left on device" results in corrupt trace data set

DISCUSSION David Goulet 5 participants 8 messages Nov 16, 2012 – Nov 29, 2012 6 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi Paul,

The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...

To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)

Cheers!
David

Woegerer, Paul:

Quoted 33 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Thanks,
Paul

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 11 lines · * David Goulet ([email protected]) wrote:

Hi Paul,

The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...

To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)

live support will eventually allow to side-track this whole issue by flushing metadata to disk periodically during tracing,

Thanks,

Mathieu

Quoted 43 lines

Cheers!
David

Woegerer, Paul:

Quoted 24 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

Quoted 9 lines

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Thanks,
Paul

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): 20121116153029.GB23798@Krystal

#5 lore raw
Quoted 8 lines · Mathieu Desnoyers
Quoted 7 lines · * David Goulet ([email protected]) wrote:

Hi Paul,

The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...

Quoted 7 lines
Quoted 4 lines

To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)

live support will eventually allow to side-track this whole issue by flushing metadata to disk periodically during tracing,

Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?

Thanks
Paul

Quoted 49 lines

Thanks,

Mathieu

Quoted 43 lines

Cheers!
David

Woegerer, Paul:

Quoted 24 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

Quoted 9 lines

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Thanks,
Paul

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 10 lines
Paul Woegerer | SW Development Engineer
http://go.mentor.com/sourceryanalyzer

Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria
Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject
to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and
other countries.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 14 lines · Paul
Quoted 13 lines · Mathieu Desnoyers
Quoted 11 lines · * David Goulet ([email protected]) wrote:

Hi Paul,

The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...

To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)

Quoted 5 lines
Quoted 2 lines

live support will eventually allow to side-track this whole issue by flushing metadata to disk periodically during tracing,

Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?

The main change of lttng-tools 2.2 is the live tracing feature. The way it is implemented in the lttngtop-live branch is more a proof-of-concept, we will be starting the clean implementation discussions here as soon as we have lttng-tools 2.1 released (which should be around the end of this month).

In the mean time, to solve the problem you are currently experiencing, you could manually add a flush on the metadata channel after the session is started to make sure it is written on disk quickly (that's what is done in lttngtop-live at the beginning). Of course like David said, if you have new events/applications enabled during tracing, the metadata needs to be flushed again.

Thanks,

Julien

Quoted 55 lines

Thanks
Paul

Quoted 49 lines

Thanks,

Mathieu

Quoted 43 lines

Cheers!
David

Woegerer, Paul:

Quoted 24 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

Quoted 9 lines

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Thanks,
Paul

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 10 lines · Julien Desfossez
Quoted 2 lines · Paul

Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?

The main change of lttng-tools 2.2 is the live tracing feature. The way it is implemented in the lttngtop-live branch is more a proof-of-concept, we will be starting the clean implementation discussions here as soon as we have lttng-tools 2.1 released (which should be around the end of this month).

Quoted 4 lines

In the mean time, to solve the problem you are currently experiencing, you could manually add a flush on the metadata channel after the session is started to make sure it is written on disk quickly (that's what is done in lttngtop-live at the beginning). Of course like David said, if

I will try that. Thanks !

--
Paul

Quoted 63 lines

you have new events/applications enabled during tracing, the metadata needs to be flushed again.

Thanks,

Julien

Quoted 55 lines

Thanks
Paul

Quoted 49 lines

Thanks,

Mathieu

Quoted 43 lines

Cheers!
David

Woegerer, Paul:

Quoted 24 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

Quoted 9 lines

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Thanks,
Paul

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 10 lines
Paul Woegerer | SW Development Engineer
http://go.mentor.com/sourceryanalyzer

Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria
Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject
to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and
other countries.
#10 lore raw
In reply to a message not in the archive
Quoted 8 lines · Julien Desfossez
Quoted 2 lines

Does that mean that the live tracing feature that currently lives in the lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?

The main change of lttng-tools 2.2 is the live tracing feature. The way it is implemented in the lttngtop-live branch is more a proof-of-concept, we will be starting the clean implementation discussions here as soon as we have lttng-tools 2.1 released (which should be around the end of this month).

Hi Julien,

I looked for "live tracing" related commits in the git repos but could find anything. There is also no RFC yet under:
http://git.lttng.org/?p=lttng-tools.git;a=tree;f=doc/proposals;hb=HEAD

Does development happen on some branch that I missed or did it simply not start yet.

Thanks,
Paul

Signature · 10 lines
Paul Woegerer | SW Development Engineer
http://go.mentor.com/sourceryanalyzer

Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria
Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject
to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and
other countries.
#11 lore raw
In reply to a message not in the archive

It could be interesting to at least reserve the size for the first packet, so we have at least a little bit of information on the trace.

Quoted 53 lines · David Goulet

Hi Paul,

The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...

To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)

Cheers!
David

Woegerer, Paul:

Quoted 24 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

Quoted 9 lines

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Thanks,
Paul

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw
Quoted 2 lines · Yannick Brosseau

It could be interesting to at least reserve the size for the first packet, so we have at least a little bit of information on the trace.

I tried that with fallocate but unfortunately tmpfs does not seem to support fallocating files :(

--
Paul

Quoted 62 lines · David Goulet
Quoted 45 lines

Hi Paul,

The problem here would be that metadata can grow arbitrarily over time when let say new events are enabled during tracing or a new lib is dynamically loaded in the application you are tracing. Same goes for the kernel for loaded module or CPU hotplug...

To achieve a static size of metadata during the whole life of a tracing session, I guess we need somekind of flag or option set to tell the tracers to just deny new events once started... which from there, we need Mathieu's opinion :)

Cheers!
David

Woegerer, Paul:

Quoted 29 lines

Hi,

embedded users sometimes have to trace to tmpfs (e.g. streaming in not an option due to missing network connectivity).

Ramdisks of 16M to 128M size are created and as long as the ram disk does not get full, tracing works fine. Unfortunately if the disk gets full things fall apart:

PERROR: Error in file write: No space left on device [in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]

The consumer daemon will not be able to write the metadata channel and therefore the trace data set will be unusable (cannot be read with babeltrace).

(quoted diff) +0 -3
-rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
-rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
-rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1

Is it possible to somehow reserve the file size (fallocate) for metadata so that in case the disk gets full at least the metadata gets written and the resulting trace set is readable with babeltrace ?

I wonder if that is possible at all. The consumer daemon doesn't know upfront how much would need to be reserved for metadata, right ?

Any ideas ?

Quoted 8 lines
Quoted 4 lines

Thanks,
Paul

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 10 lines
Paul Woegerer | SW Development Engineer
http://go.mentor.com/sourceryanalyzer

Mentor Embedded(tm) | Prinz Eugen Straße 72/2/4, Vienna, 1040 Austria
Nucleus® | Linux® | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject
to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and
other countries.