Skip to contentSkip to conversation

Common Trace Format Updates (upcoming 1.8.3)

RFC PATCH In review Mathieu Desnoyers via lttng-dev 3 participants 12 messages Apr 23, 2020 – Apr 29, 2020

lore
Share & tools
lore
#0 lore raw

cover Common Trace Format Updates (upcoming 1.8.3)

Hi,

We noticed a few unspecified areas in the CTF 1.8.2 specification which require some clarifications. We therefore plan to do a 1.8.3 release of the specification including those changes.

Review is welcome,

Thanks,

Mathieu

Geneviève Bastien (1):
Clarify that unlisted enum values are implementation-defined

Mathieu Desnoyers (2):
Clarify monotonicity requirement on timestamp begin Clarify that timestamp begin/end need to be complete clock values

 common-trace-format-specification.md | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
Signature · 6 lines
2.11.0

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#1 lore raw

RFC PATCH 1/3 Clarify that unlisted enum values are implementation-defined

From: Geneviève Bastien <[email protected]>

---
 common-trace-format-specification.md | 3 +++
 1 file changed, 3 insertions(+)
common-trace-format-specification.md +3 -0
index fd49e59..f5fea51 100644
@@ -464,6 +464,9 @@ enum {
 }
 ~~~
 
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+
 ### 4.2 Compound types
 
 Compound are aggregation of type declarations. Compound types include
Signature · 6 lines
2.11.0

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#2 lore raw

Re: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

Quoted 14 lines · Mathieu Desnoyers

From: Geneviève Bastien <[email protected]>

---
 common-trace-format-specification.md | 3 +++
 1 file changed, 3 insertions(+)
common-trace-format-specification.md +0 -0
index fd49e59..f5fea51 100644

common-trace-format-specification.md enum { · line 469

Quoted 6 lines
common-trace-format-specification.md +3 -0
@@ -464,6 +464,9 @@ enum {
 }
 ~~~
 
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+

This is too vague to be useful knowing that the main rationale for this change is to allow enums to express some type of bitfield of flags in the tracer and readers [1].

What is the meaning of an unmapped value? This section should at least describe the correct interpretation of unmapped values as flags and when it is appropriate to do so.

Thanks,
Jérémie

[1] https://review.lttng.org/c/babeltrace/+/3045

Quoted 6 lines
common-trace-format-specification.md +0 -1
 ### 4.2 Compound types
 
 Compound are aggregation of type declarations. Compound types include
--

2.11.0

Signature · 7 lines
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#3 lore raw

Re: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

Quoted 28 lines · ----- On Apr 23, 2020, at 6:51 PM, Jeremie Galarneau…
Quoted 15 lines · Mathieu Desnoyers

From: Geneviève Bastien <[email protected]>

---
 common-trace-format-specification.md | 3 +++
 1 file changed, 3 insertions(+)
common-trace-format-specification.md +0 -0
index fd49e59..f5fea51 100644

common-trace-format-specification.md enum { · line 469

Quoted 6 lines
common-trace-format-specification.md +3 -0
@@ -464,6 +464,9 @@ enum {
 }
 ~~~
 
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+

This is too vague to be useful knowing that the main rationale for this change is to allow enums to express some type of bitfield of flags in the tracer and readers [1].

Quoted 3 lines

What is the meaning of an unmapped value? This section should at least describe the correct interpretation of unmapped values as flags and when it is appropriate to do so.

Considering that this is a patchlevel update to CTF, I would not expect that we introduce new features in the specification. Only clarifications to parts of the specification that were unclear.

Specifying a new behavior related to unmapped values would fall IMO into the realm of "new feature", and would belong to a CTF 1.9 or CTF 2.0. As we all know, there is a CTF 2.0 in the making, but it does not solve the immediate problem of LTTng 2.12 which produces those unmapped enum values within traces identified as CTF 1.8.

Moreover, to add to the problem, Babeltrace 2 has a strict match on CTF 1.8 in the CTF source plugin, and won't accept a CTF 1.9 trace (unlike Babeltrace 1 which would emit a warning about possibly unsupported features, but would accept a 1.9 CTF trace nevertheless).

So this is why I am proposing this minimal clarification to the CTF 1.8
specification: that unmapped enum values are implementation defined (rather than saying nothing about them). Considering that tracers can generate this kind of trace data anyway, it's really a consideration that should have been explicitly expressed in the specification from the start and was an involuntary omission.

So perhaps we need to state something more than just "implementation defined", but it's unclear what without ending up adding features into a patchlevel update.

Thanks,

Mathieu

Quoted 17 lines

Thanks,
Jérémie

[1] https://review.lttng.org/c/babeltrace/+/3045

Quoted 6 lines
common-trace-format-specification.md +0 -1
 ### 4.2 Compound types
 
 Compound are aggregation of type declarations. Compound types include
--

2.11.0

--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#4 lore raw

Re: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

In reply to Mathieu Desnoyers via lttng-dev

----- Original Message -----

Quoted 24 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>, [email protected], [email protected], "Jeremie Galarneau" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Sent: Thursday, 23 April, 2020 16:52:24
Subject: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

From: Geneviève Bastien <[email protected]>

---
common-trace-format-specification.md | 3 +++

1 file changed, 3 insertions(+)
common-trace-format-specification.md +0 -0
index fd49e59..f5fea51 100644
@@ -464,6 +464,9 @@ enum {
 }

~~~

common-trace-format-specification.md enum { · line 469

Quoted 3 lines
common-trace-format-specification.md +3 -0
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+

Why not just:

An enumeration field can have an integral value which its type does not map to a string.

?

Phil

Quoted 5 lines

### 4.2 Compound types

Compound are aggregation of type declarations. Compound types include --
2.11.0

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#5 lore raw

Re: [diamon-discuss] [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

Quoted 35 lines · ----- On Apr 28, 2020, at 2:40 PM, Philippe Proulx [email protected] wrote:

----- Original Message -----

Quoted 28 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>, [email protected], [email protected], "Jeremie Galarneau" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Sent: Thursday, 23 April, 2020 16:52:24
Subject: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

From: Geneviève Bastien <[email protected]>

---
common-trace-format-specification.md | 3 +++

1 file changed, 3 insertions(+)
common-trace-format-specification.md +0 -0
index fd49e59..f5fea51 100644
@@ -464,6 +464,9 @@ enum {
 }

~~~

common-trace-format-specification.md enum { · line 469

Quoted 3 lines
common-trace-format-specification.md +3 -0
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+

Why not just:

Quoted 4 lines

An enumeration field can have an integral value which its type does not map to a string.

?

Good point, I will use that wording.

Thanks,

Mathieu

Quoted 8 lines

Phil

Quoted 5 lines

### 4.2 Compound types

Compound are aggregation of type declarations. Compound types include --
2.11.0

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#6 lore raw

Re: [diamon-discuss] [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

Quoted 37 lines · ----- On Apr 28, 2020, at 2:51 PM, Mathieu Desnoyers…
Quoted 35 lines · ----- On Apr 28, 2020, at 2:40 PM, Philippe Proulx [email protected] wrote:

----- Original Message -----

Quoted 28 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>, [email protected], [email protected], "Jeremie Galarneau" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Sent: Thursday, 23 April, 2020 16:52:24
Subject: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

From: Geneviève Bastien <[email protected]>

---
common-trace-format-specification.md | 3 +++

1 file changed, 3 insertions(+)
common-trace-format-specification.md +0 -0
index fd49e59..f5fea51 100644
@@ -464,6 +464,9 @@ enum {
 }

~~~

common-trace-format-specification.md enum { · line 469

Quoted 3 lines
common-trace-format-specification.md +3 -0
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+

Why not just:

Quoted 6 lines
Quoted 4 lines

An enumeration field can have an integral value which its type does not map to a string.

?

Good point, I will use that wording.

Geneviève pointed out on IRC that the sentence above is weird. Would the following convey the right meaning ?

"An enumeration field can have an integral value for which the associated enumeration type does not map to a string."

Thanks,

Mathieu

Quoted 18 lines

Thanks,

Mathieu

Quoted 8 lines

Phil

Quoted 5 lines

### 4.2 Compound types

Compound are aggregation of type declarations. Compound types include --
2.11.0

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#7 lore raw

Re: [diamon-discuss] [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

----- Original Message -----

Quoted 53 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "Philippe Proulx" <[email protected]>
Cc: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>, "Jeremie Galarneau" <[email protected]>, "lttng-dev" <[email protected]>, "diamon-discuss" <[email protected]>
Sent: Wednesday, 29 April, 2020 08:08:04
Subject: Re: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

----- On Apr 28, 2020, at 2:51 PM, Mathieu Desnoyers

Quoted 42 lines · [email protected] wrote:
Quoted 39 lines · ----- On Apr 28, 2020, at 2:40 PM, Philippe Proulx [email protected] wrote:

----- Original Message -----

Quoted 28 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>, [email protected], [email protected], "Jeremie Galarneau" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Sent: Thursday, 23 April, 2020 16:52:24
Subject: [RFC PATCH CTF 1/3] Clarify that unlisted enum values are implementation-defined

From: Geneviève Bastien <[email protected]>

---
common-trace-format-specification.md | 3 +++

1 file changed, 3 insertions(+)
common-trace-format-specification.md +0 -0
index fd49e59..f5fea51 100644
@@ -464,6 +464,9 @@ enum {
 }

~~~

common-trace-format-specification.md enum { · line 469

Quoted 3 lines
common-trace-format-specification.md +3 -0
+The mappings in the enumeration type do not have to be exhaustive.
+Unlisted values are implementation defined.
+

Why not just:

An enumeration field can have an integral value which its type does not map to a string.

?

Quoted 8 lines
Quoted 1 line

Good point, I will use that wording.

Geneviève pointed out on IRC that the sentence above is weird. Would the following convey the right meaning ?

"An enumeration field can have an integral value for which the associated enumeration
type does not map to a string."

Sure.

Phil

Quoted 28 lines

Thanks,

Mathieu

Quoted 10 lines

Thanks,

Mathieu

Quoted 5 lines

Phil

Quoted 2 lines

### 4.2 Compound types

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Compound are aggregation of type declarations. Compound types include --
2.11.0

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

#8 lore raw

RFC PATCH 2/3 Clarify monotonicity requirement on timestamp begin

In reply to Mathieu Desnoyers via lttng-dev
---
 common-trace-format-specification.md | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
common-trace-format-specification.md +6 -4
index f5fea51..53b70f4 100644
@@ -836,10 +836,12 @@ TSDL metadata):
     range between these timestamps should include all event timestamps
     assigned to events contained within the packet. The timestamp at the
     beginning of an event packet is guaranteed to be below or equal the
-    timestamp at the end of that event packet. The timestamp at the end
-    of an event packet is guaranteed to be below or equal the
-    timestamps at the end of any following packet within the same stream.
-    See [Clocks](#spec8) for more detail.
+    timestamp at the end of that event packet. The timestamp at the
+    beginning of an event packet is guaranteed to be above or equal the
+    timestamps at the beginning of any prior packet within the same
+    stream. The timestamp at the end of an event packet is guaranteed to
+    be below or equal the timestamps at the end of any following packet
+    within the same stream. See [Clocks](#spec8) for more detail.
   * **Events discarded count**. Snapshot of a per-stream
     free-running counter, counting the number of events discarded that
     were supposed to be written in the stream after the last event in
Signature · 1 line
2.11.0
#9 lore raw

Re: [RFC PATCH CTF 2/3] Clarify monotonicity requirement on timestamp begin

----- Original Message -----

Quoted 18 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>, [email protected], [email protected], "Jeremie Galarneau" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Sent: Thursday, 23 April, 2020 16:52:25
Subject: [RFC PATCH CTF 2/3] Clarify monotonicity requirement on timestamp begin

---
common-trace-format-specification.md | 10 ++++++----

1 file changed, 6 insertions(+), 4 deletions(-)
common-trace-format-specification.md +0 -0
index f5fea51..53b70f4 100644

common-trace-format-specification.md TSDL metadata): · line 840

Quoted 9 lines
common-trace-format-specification.md +2 -4
@@ -836,10 +836,12 @@ TSDL metadata):
    range between these timestamps should include all event timestamps
    assigned to events contained within the packet. The timestamp at the
    beginning of an event packet is guaranteed to be below or equal the
-    timestamp at the end of that event packet. The timestamp at the end
-    of an event packet is guaranteed to be below or equal the
-    timestamps at the end of any following packet within the same stream.
-    See [Clocks](#spec8) for more detail.
+    timestamp at the end of that event packet. The timestamp at the
+    beginning of an event packet is guaranteed to be above or equal the

"greater than or equal to"

common-trace-format-specification.md TSDL metadata): · line 843

Quoted 3 lines
common-trace-format-specification.md +3 -0
+    timestamps at the beginning of any prior packet within the same
+    stream. The timestamp at the end of an event packet is guaranteed to
+    be below or equal the timestamps at the end of any following packet

"less than or equal to"

Phil

Quoted 6 lines
common-trace-format-specification.md +1 -1
+    within the same stream. See [Clocks](#spec8) for more detail.
  * **Events discarded count**. Snapshot of a per-stream
    free-running counter, counting the number of events discarded that
    were supposed to be written in the stream after the last event in
--

2.11.0

#10 lore raw

Re: [diamon-discuss] [RFC PATCH CTF 2/3] Clarify monotonicity requirement on timestamp begin

Quoted 25 lines · ----- On Apr 28, 2020, at 2:42 PM, Philippe Proulx [email protected] wrote:

----- Original Message -----

Quoted 24 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "gbastien+lttng" <[email protected]>, "Matthew Khouzam" <[email protected]>,
[email protected], [email protected], "Jeremie Galarneau" <[email protected]>
Cc: "lttng-dev" <[email protected]>, "Mathieu Desnoyers" <[email protected]>
Sent: Thursday, 23 April, 2020 16:52:25
Subject: [RFC PATCH CTF 2/3] Clarify monotonicity requirement on timestamp begin

---
common-trace-format-specification.md | 10 ++++++----

1 file changed, 6 insertions(+), 4 deletions(-)
common-trace-format-specification.md +0 -0
index f5fea51..53b70f4 100644
@@ -836,10 +836,12 @@ TSDL metadata):
    range between these timestamps should include all event timestamps
    assigned to events contained within the packet. The timestamp at the
    beginning of an event packet is guaranteed to be below or equal the
Quoted 8 lines

common-trace-format-specification.md TSDL metadata): · line 840

Quoted 6 lines
common-trace-format-specification.md +2 -4
-    timestamp at the end of that event packet. The timestamp at the end
-    of an event packet is guaranteed to be below or equal the
-    timestamps at the end of any following packet within the same stream.
-    See [Clocks](#spec8) for more detail.
+    timestamp at the end of that event packet. The timestamp at the
+    beginning of an event packet is guaranteed to be above or equal the

"greater than or equal to"

done

Quoted 6 lines

common-trace-format-specification.md TSDL metadata): · line 843

Quoted 4 lines
common-trace-format-specification.md +3 -0
+    timestamps at the beginning of any prior packet within the same
+    stream. The timestamp at the end of an event packet is guaranteed to
+    be below or equal the timestamps at the end of any following packet

"less than or equal to"

done,

Thanks!

Mathieu

Quoted 9 lines

Phil

Quoted 6 lines
common-trace-format-specification.md +1 -1
+    within the same stream. See [Clocks](#spec8) for more detail.
  * **Events discarded count**. Snapshot of a per-stream
    free-running counter, counting the number of events discarded that
    were supposed to be written in the stream after the last event in
--

2.11.0

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#11 lore raw

RFC PATCH 3/3 Clarify that timestamp begin/end need to be complete clock values

In reply to Mathieu Desnoyers via lttng-dev
---
 common-trace-format-specification.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
common-trace-format-specification.md +4 -3
index 53b70f4..7681ce5 100644
@@ -1580,7 +1580,8 @@ stream {
 };
 ~~~
 
-For a N-bit integer type referring to a clock, if the integer overflows
+Within the stream event context, event context, and event payload,
+fields of N-bit integer type referring to a clock, if the integer overflows
 compared to the N low order bits of the clock prior value found in the
 same stream, then it is assumed that one, and only one, overflow
 occurred. It is therefore important that events encoding time on a small
@@ -1589,8 +1590,8 @@ N-bit overflow occurs.
 
 In a packet context, clock field names ending with `_begin` and `_end`
 have a special meaning: this refers to the timestamps at, respectively,
-the beginning and the end of each packet.
-
+the beginning and the end of each packet. Those are required to be
+complete representations of the clock value.
 
 ## A. Helper macros
 
Signature · 1 line
2.11.0