Skip to contentSkip to conversation

On lost events

DISCUSSION Mathieu Desnoyers 3 participants 7 messages Jul 4, 2013 – Jul 8, 2013 6 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 11 lines · * Matthew Khouzam ([email protected]) wrote:

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 31 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 9 lines

Hope this clarifies some issues, it did for me!

Matthew

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#3 lore raw
In reply to a message not in the archive

Quick question. Some traces have overlapping packets, so in that case, the lost events are between the last event of a previous packet and the first event of the next packet, which just happens to be the previous packet's end time?

Quoted 76 lines · Mathieu Desnoyers
Quoted 11 lines · * Matthew Khouzam ([email protected]) wrote:

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 30 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 8 lines

Hope this clarifies some issues, it did for me!

Matthew

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#5 lore raw
In reply to a message not in the archive
Quoted 4 lines · Matthew Khouzam

Quick question. Some traces have overlapping packets, so in that case, the lost events are between the last event of a previous packet and the first event of the next packet, which just happens to be the previous packet's end time?

Babeltrace say that the losts are between the beginning of the next and the end of the previous.

Quoted 82 lines · Mathieu Desnoyers
Quoted 62 lines · * Matthew Khouzam ([email protected]) wrote:
Quoted 11 lines

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 30 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

Quoted 14 lines

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 8 lines

Hope this clarifies some issues, it did for me!

Matthew

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#7 lore raw
In reply to a message not in the archive
Quoted 7 lines · Yannick Brosseau
Quoted 4 lines · Matthew Khouzam

Quick question. Some traces have overlapping packets, so in that case, the lost events are between the last event of a previous packet and the first event of the next packet, which just happens to be the previous packet's end time?

Babeltrace say that the losts are between the beginning of the next and the end of the previous.

To quote the green lantern, "I KNOW RIGHT?!!?!?!". jokes aside, that implies that all packets should always overlap and if they don't there should be anti-time. I think the safest way to go about it is "last event of the last packet to first event of the next one".

Quoted 87 lines
Quoted 74 lines · Mathieu Desnoyers
Quoted 72 lines · * Matthew Khouzam ([email protected]) wrote:
Quoted 11 lines

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 30 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 4 lines

Hope this clarifies some issues, it did for me!

Matthew

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#8 lore raw
In reply to a message not in the archive
Quoted 4 lines · * Matthew Khouzam ([email protected]) wrote:

Quick question. Some traces have overlapping packets, so in that case, the lost events are between the last event of a previous packet and the first event of the next packet, which just happens to be the previous packet's end time?

"Some traces have overlapping packets"

do you mean

"Some traces have overlapping packets"

or

"Some streams have overlapping packets"

?

Quoted 78 lines · Mathieu Desnoyers
Quoted 76 lines · * Matthew Khouzam ([email protected]) wrote:
Quoted 11 lines

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 30 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 8 lines

Hope this clarifies some issues, it did for me!

Matthew

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#10 lore raw
In reply to a message not in the archive

Some streams have overlapping packets. I was shocked.

Quoted 94 lines · Mathieu Desnoyers
Quoted 4 lines · * Matthew Khouzam ([email protected]) wrote:

Quick question. Some traces have overlapping packets, so in that case, the lost events are between the last event of a previous packet and the first event of the next packet, which just happens to be the previous packet's end time?

"Some traces have overlapping packets"

do you mean

"Some traces have overlapping packets"

or

"Some streams have overlapping packets"

?

Quoted 77 lines · Mathieu Desnoyers
Quoted 76 lines · * Matthew Khouzam ([email protected]) wrote:
Quoted 11 lines

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 30 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 8 lines

Hope this clarifies some issues, it did for me!

Matthew

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#12 lore raw
In reply to a message not in the archive
Quoted 1 line · * Matthew Khouzam ([email protected]) wrote:

Some streams have overlapping packets. I was shocked.

Ah! yes. The time stamp begin is sampled at event space reservation, and time stamp end is sampled at commit.

So you can have, with packets A and B in a stream:

- reserve space at end of A
- interrupted
- reserve space at beginning of B (get timestamp begin)
- commit at end of A (get timestamp end)
- commit at beginning of B

So yes, the timestamps can very slightly overlap indeed.

Thanks,

Mathieu

Quoted 98 lines · Mathieu Desnoyers
Quoted 94 lines · * Matthew Khouzam ([email protected]) wrote:
Quoted 4 lines

Quick question. Some traces have overlapping packets, so in that case, the lost events are between the last event of a previous packet and the first event of the next packet, which just happens to be the previous packet's end time?

"Some traces have overlapping packets"

do you mean

"Some traces have overlapping packets"

or

"Some streams have overlapping packets"

?

Quoted 77 lines · Mathieu Desnoyers
Quoted 76 lines · * Matthew Khouzam ([email protected]) wrote:
Quoted 11 lines

Hi tracing werewolves,

I want to clear up some stuff about lost events.

Lost events are caused, in lttng 2.x when the tracer does not have enough memory available to write an event. This can be due to:
* events coming in too fast (faster than we can write them)
* a given event being too large
* we have an event that does a nested wrap around

LTTng will not write the whole packet that is lost.

LTTng does not write an _event_ discarded at all. However, a packet "lost" is a different thing: it only happens in overwrite mode, when we overwrite a packet.

Quoted 3 lines

It uses a ring
buffer and the packet will just be overwritten in the buffer, but a counter will be incremented.

For an _event_ discarded, we're not even writing it into the buffer.

Quoted 1 line

The counter is written in the packet header in the CTF trace.

Indeed, the event lost counters are written in the packet in the ctf trace.

Quoted 30 lines

When reading a trace, if you just read the packet headers, you will see something as follows:

Packet 1:
ts_begin: 100
ts_end: 200
discarded: 0

Packet 2:
ts_begin: 300
ts_end: 400
discarded: 100

Packet 3:
ts_begin: 500
ts_end: 600
discarded: 100

Packet 4:
ts_begin: 650
ts_end: 750
discarded: 200

With this example, the lost events are located between packets 1, 2 and packets 3,4.

If the events are too large, they are discarded in the current packet, lttng does not split events into packets (yet?), and this is lost during a current packet.

Quoted 7 lines

That means if the events are lost between 1 and 2, they are potentially in the range of [200-400] but probably in [200-300]
The next lost event are between 3 and 4 (events discarded is a total count, not local to that packet), so 100 events are also lost between [600-750] but likely in between [600-650]. If you can confirm that all events WITH CONTEXTS are smaller than the packet sizes, I think it's a shoe in that it's between the packets and not containing the next packet.

"and not contained in the next packet".

Thanks,

Mathieu

Quoted 8 lines

Hope this clarifies some issues, it did for me!

Matthew

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com