Sessions are disconnected when disk is full on remote host
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 29 lines · Mikael Beckius
Hello!
I am trying to verify the expected behaviour in the event a machine hosting an lttng relay daemon runs out of disk space.
Currently it appears from testing and inspecting the source code that whenever the lttng relay daemon hits an error caused by the lack of disk space all failing sessions will be disconnected. It also appears that the only way to recover from this situation once disk space has been restored is to recreate all the disconnected sessions.
I have also played around with different combinations of the below lttng enable-channel options:
--discard
--overwrite
--tracefile-count <trace file count>
--tracefile-size <trace file size>The initial channel configuration was using the options --tracefile-count --tracefile-size but both were abandoned as Babeltrace faced constant streaming errors during heavy live tracing and now the trace files are growing continuously until there is no available disk space.
I assume that --overwrite would not make any difference compared to --discard with regards to a disk full scenario. Even if you specify --tracefile-count and --tracefile-size to limit the amount of trace data from a single session you may still face the same issue with disconnected sessions if you for some reason run out of disk space.
Quoted 7 lines
Testing was carried out on the latest 2.6 releases and all sessions were live sessions.
So my questions are:
- Is there any way to keep sessions alive even if you run out of disk space?
- How can you clean up old traces once you have collected them without recreating sessions?
Hi Mikael,
LTTng currently doesn't support these recovery modes.
However, the file rotation issues you mentioned should be resolved in lttng-tools 2.6.1. Any chance you can give it a try?
The website will be updated later today. In the meantime, here's a direct link to the release:
https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2
Regards,
Jérémie
Quoted 8 lines
Micke
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 8 lines
Jérémie Galarneau EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): CA+jJMxsAFJ+MgYYRxhcJUuoyUT8bUWfuiH6WJXSPyVg65zLGWw@mail.gmail.com
________________________________________
Från: [email protected] [[email protected]] för Jérémie Galarneau [[email protected]]
Skickat: den 25 september 2015 21:50
Till: Mikael Beckius
Kopia: [email protected]
Ämne: Re: [lttng-dev] Sessions are disconnected when disk is full on remote host
Quoted 29 lines · Mikael Beckius
Hello!
I am trying to verify the expected behaviour in the event a machine hosting an lttng relay daemon runs out of disk space.
Currently it appears from testing and inspecting the source code that whenever the lttng relay daemon hits an error caused by the lack of disk space all failing sessions will be disconnected. It also appears that the only way to recover from this situation once disk space has been restored is to recreate all the disconnected sessions.
I have also played around with different combinations of the below lttng enable-channel options:
--discard
--overwrite
--tracefile-count <trace file count>
--tracefile-size <trace file size>The initial channel configuration was using the options --tracefile-count --tracefile-size but both were abandoned as Babeltrace faced constant streaming errors during heavy live tracing and now the trace files are growing continuously until there is no available disk space.
I assume that --overwrite would not make any difference compared to --discard with regards to a disk full scenario. Even if you specify --tracefile-count and --tracefile-size to limit the amount of trace data from a single session you may still face the same issue with disconnected sessions if you for some reason run out of disk space.
Quoted 7 lines
Testing was carried out on the latest 2.6 releases and all sessions were live sessions.
So my questions are:
- Is there any way to keep sessions alive even if you run out of disk space?
- How can you clean up old traces once you have collected them without recreating sessions?
Hi Mikael,
LTTng currently doesn't support these recovery modes.
However, the file rotation issues you mentioned should be resolved in lttng-tools 2.6.1. Any chance you can give it a try?
The website will be updated later today. In the meantime, here's a direct link to the release:
https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2
Regards,
Jérémie
Hello Jérémie!
Thanks for the hint. I will give the new 2.6.1 and 2.7.0 releases of lttng-tools a try and see if they work better than 2.6.0.
Micke
Quoted 8 lines
Micke
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
After trying with 2.6.1 and 2.7.0 releases of lttng-tools I cannot say that it makes much of a difference compared with the older versions previously tried. Once the file size for a CPU is reached Babeltrace will output:
[error] get_next_index: error
[error] get_next_index failed
and Babeltrace output from that CPU will cease. Restarting Babeltrace will output some traces again before output ceases almost immediately and the problem remains until the session has been recreated.
This was with --tracefile-count set to "1" and --tracefile-size of various sizes up to a megabyte BUT if I increase the trace file count to "2" it actually appears to be working.
"LTTng currently doesn't support these recovery modes."
Are there any plans for LTTng to support these "recovery modes" or similar in a future release? Or is there perhaps a way using the LTTng API to detect that a session has lost its connection?
Micke