Modernize CTF specification using Markdown
A message that is not in the archive (sent off-list or not yet received): [email protected]
Merged, and I pushed a symlink from the .txt -> .md.
Thanks!
Mathieu
----- Original Message -----
Quoted 3,996 lines
From: "Philippe Proulx" <[email protected]>
To: [email protected]
Sent: Wednesday, January 21, 2015 11:51:16 PM
Subject: [lttng-dev] [PATCH ctf] Modernize CTF specification using MarkdownContent is unchanged, with the exception of a few
minor typos fixed here and there.Signed-off-by Philippe Proulx <[email protected]>--- common-trace-format-specification.md | 2035 +++++++++++++++++++++++++++++++++ common-trace-format-specification.txt | 1823 ----------------------------- 2 files changed, 2035 insertions(+), 1823 deletions(-) create mode 100644 common-trace-format-specification.md delete mode 100644 common-trace-format-specification.txtcommon-trace-format-specification.md new file +8
-0@@ -0,0 +1,2035 @@ +# Common Trace Format (CTF) Specification (v1.8.2) + +**Author**: Mathieu Desnoyers, [EfficiOS Inc.](http://www.efficios.com/) + +The goal of the present document is to specify a trace format that suits +the needs of the embedded, telecom, high-performance and kernel +communities. It is based on the +[Common Trace Format Requirements(v1.4)](http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-reqs.txt;hb=master)
common-trace-format-specification.md new file +20
-0+document. It is designed to allow traces to be natively generated by the +Linux kernel, Linux user space applications written in C/C++, and +hardware components. One major element of CTF is the Trace Stream +Description Language (TSDL) which flexibility enables description of +various binary trace stream layouts. + +The latest version of this document can be found at: + + * Git: `git clone git://git.efficios.com/ctf.git` + * [Gitweb](http://git.efficios.com/?p=ctf.git) + +A reference implementation of a library to read and write this trace +format is being implemented within the +[Babeltrace](http://www.efficios.com/babeltrace) project, a converter +between trace formats. The development tree is available at: + + * Git: `git clone git://git.efficios.com/babeltrace.git` + * [Gitweb](http://git.efficios.com/?p=babeltrace.git) + +The [CEWorkgroup](http://www.linuxfoundation.org/collaborate/workgroups/celf)
common-trace-format-specification.md new file +1596
-0+of the Linux Foundation, [Ericsson](http://www.ericsson.com/), and +[EfficiOS](http://www.efficios.com/) have sponsored this work. + +**Contents**: + + 1. Preliminary definitions + 2. High-level representation of a trace + 3. Event stream + 4. Types + 4.1 Basic types + 4.1.1 Type inheritance + 4.1.2 Alignment + 4.1.3 Byte order + 4.1.4 Size + 4.1.5 Integers + 4.1.6 GNU/C bitfields + 4.1.7 Floating point + 4.1.8 Enumerations + 4.2 Compound types + 4.2.1 Structures + 4.2.2 Variants (discriminated/tagged unions) + 4.2.3 Arrays + 4.2.4 Sequences + 4.2.5 Strings + 5. Event packet header + 5.1 Event packet header description + 5.2 Event packet context description + 6. Event structure + 6.1 Event header + 6.1.1 Type 1: few event IDs + 6.1.2 Type 2: many event IDs + 6.2 Stream event context and event context + 6.3 Event payload + 6.3.1 Padding + 6.3.2 Alignment + 7. Trace Stream Description Language (TSDL) + 7.1 Metadata + 7.2 Declaration vs definition + 7.3 TSDL scopes + 7.3.1 Lexical scope + 7.3.2 Static and dynamic scopes + 7.4 TSDL examples + 8. Clocks + A. Helper macros + B. Stream header rationale + C. TSDL Grammar + C.1 Lexical grammar + C.1.1 Lexical elements + C.1.2 Keywords + C.1.3 Identifiers + C.1.4 Universal character names + C.1.5 Constants + C.1.6 String literals + C.1.7 Punctuators + C.2 Phrase structure grammar + C.2.2 Declarations: + C.2.3 CTF-specific declarations + + +## 1. Preliminary definitions + + * **Event trace**: an ordered sequence of events. + * **Event stream**: an ordered sequence of events, containing a + subset of the trace event types. + * **Event packet**: a sequence of physically contiguous events within + an event stream. + * **Event**: this is the basic entry in a trace. Also known as + a _trace record_. + * An **event identifier** (ID) relates to the class (a type) of + event within an event stream, e.g. event `irq_entry`. + * An **event** (or event record) relates to a specific instance of + an event class, e.g. event `irq_entry`, at time _X_, on CPU _Y_. + * Source architecture: architecture writing the trace. + * Reader architecture: architecture reading the trace. + + +## 2. High-level representation of a trace + +A _trace_ is divided into multiple event streams. Each event stream +contains a subset of the trace event types. + +The final output of the trace, after its generation and optional +transport over the network, is expected to be either on permanent or +temporary storage in a virtual file system. Because each event stream +is appended to while a trace is being recorded, each is associated with +a distinct set of files for output. Therefore, a stored trace can be +represented as a directory containing zero, one or more files +per stream. + +Metadata description associated with the trace contains information on +trace event types expressed in the _Trace Stream Description Language_ +(TSDL). This language describes: + + * Trace version + * Types available + * Per-trace event header description + * Per-stream event header description + * Per-stream event context description + * Per-event + * Event type to stream mapping + * Event type to name mapping + * Event type to ID mapping + * Event context description + * Event fields description + + +## 3. Event stream + +An _event stream_ can be divided into contiguous event packets of +variable size. An event packet can contain a certain amount of padding +at the end. The stream header is repeated at the beginning of each +event packet. The rationale for the event stream design choices is +explained in [Stream header rationale](#specB). + +The event stream header will therefore be referred to as the +_event packet header_ throughout the rest of this document. + + +## 4. Types + +Types are organized as type classes. Each type class belong to either +of two kind of types: _basic types_ or _compound types_. + + +### 4.1 Basic types + +A basic type is a scalar type, as described in this section. It +includes integers, GNU/C bitfields, enumerations, and floating +point values. + + +#### 4.1.1 Type inheritance + +Type specifications can be inherited to allow deriving types from a +type class. For example, see the uint32_t named type derived from the +[_integer_ type](#spec4.1.5) class. Types have a precise binary +representation in the trace. A type class has methods to read and write +these types, but must be derived into a type to be usable in an event +field. + + +#### 4.1.2 Alignment + +We define _byte-packed_ types as aligned on the byte size, namely 8-bit. +We define _bit-packed_ types as following on the next bit, as defined +by the [Integers](#spec4.1.5) section. + +Each basic type must specify its alignment, in bits. Examples of +possible alignments are: bit-packed (`align = 1`), byte-packed +(`align = 8`), or word-aligned (e.g. `align = 32` or `align = 64`). +The choice depends on the architecture preference and compactness vs +performance trade-offs of the implementation. Architectures providing +fast unaligned write byte-packed basic types to save space, aligning +each type on byte boundaries (8-bit). Architectures with slow unaligned +writes align types on specific alignment values. If no specific +alignment is declared for a type, it is assumed to be bit-packed for +integers with size not multiple of 8 bits and for gcc bitfields. All +other basic types are byte-packed by default. It is however recommended +to always specify the alignment explicitly. Alignment values must be +power of two. Compound types are aligned as specified in their +individual specification. + +The base offset used for field alignment is the start of the packet +containing the field. For instance, a field aligned on 32-bit needs to +be at an offset multiple of 32-bit from the start of the packet that +contains it. + +TSDL metadata attribute representation of a specific alignment: + +~~~ tsdl +align = /* value in bits */; +~~~ + +#### 4.1.3 Byte order + +By default, byte order of a basic type is the byte order described in +the trace description. It can be overridden by specifying a +`byte_order` attribute for a basic type. Typical use-case is to specify +the network byte order (big endian: `be`) to save data captured from +the network into the trace without conversion. + +TSDL metadata representation: + +~~~ tsdl +/* network and be are aliases */ +byte_order = /* native OR network OR be OR le */; +~~~ + +The `native` keyword selects the byte order described in the trace +description. The `network` byte order is an alias for big endian. + +Even though the trace description section is not per se a type, for +sake of clarity, it should be noted that `native` and `network` byte +orders are only allowed within type declaration. The `byte_order` +specified in the trace description section only accepts `be` or `le` +values. + + +#### 4.1.4 Size + +Type size, in bits, for integers and floats is that returned by +`sizeof()` in C multiplied by `CHAR_BIT`. We require the size of `char` +and `unsigned char` types (`CHAR_BIT`) to be fixed to 8 bits for +cross-endianness compatibility. + +TSDL metadata representation: + +~~~ tsdl +size = /* value is in bits */; +~~~ + + +#### 4.1.5 Integers + +Signed integers are represented in two-complement. Integer alignment, +size, signedness and byte ordering are defined in the TSDL metadata. +Integers aligned on byte size (8-bit) and with length multiple of byte +size (8-bit) correspond to the C99 standard integers. In addition, +integers with alignment and/or size that are _not_ a multiple of the +byte size are permitted; these correspond to the C99 standard bitfields, +with the added specification that the CTF integer bitfields have a fixed +binary representation. Integer size needs to be a positive integer. +Integers of size 0 are **forbidden**. An MIT-licensed reference +implementation of the CTF portable bitfields is available +[here](http://git.efficios.com/?p=babeltrace.git;a=blob;f=include/babeltrace/bitfield.h). + +Binary representation of integers: + + * On little and big endian: + * Within a byte, high bits correspond to an integer high bits, and + low bits correspond to low bits + * On little endian: + * Integer across multiple bytes are placed from the less significant + to the most significant + * Consecutive integers are placed from lower bits to higher bits + (even within a byte) + * On big endian: + * Integer across multiple bytes are placed from the most significant + to the less significant + * Consecutive integers are placed from higher bits to lower bits + (even within a byte) + +This binary representation is derived from the bitfield implementation +in GCC for little and big endian. However, contrary to what GCC does, +integers can cross units boundaries (no padding is required). Padding +can be [explicitly added](#spec4.1.6) to follow the GCC layout if needed. + +TSDL metadata representation: + +~~~ tsdl +integer { + signed = /* true OR false */; /* default: false */ + byte_order = /* native OR network OR be OR le */; /* default: native */ + size = /* value in bits */; /* no default */ + align = /* value in bits */; + + /* base used for pretty-printing output; default: decimal */ + base = /* decimal OR dec OR d OR i OR u OR 10 OR hexadecimal OR hex + OR x OR X OR p OR 16 OR octal OR oct OR o OR 8 OR binary + OR b OR 2 */; + + /* character encoding */ + encoding = /* none or UTF8 or ASCII */; /* default: none */ +} +~~~ + +Example of type inheritance (creation of a `uint32_t` named type): + +~~~ tsdl +typealias integer { + size = 32; + signed = false; + align = 32; +} := uint32_t; +~~~ + +Definition of a named 5-bit signed bitfield: + +~~~ tsdl +typealias integer { + size = 5; + signed = true; + align = 1; +} := int5_t; +~~~ + +The character encoding field can be used to specify that the integer +must be printed as a text character when read. e.g.: + +~~~ tsdl +typealias integer { + size = 8; + align = 8; + signed = false; + encoding = UTF8; +} := utf_char; +~~~ + +#### 4.1.6 GNU/C bitfields + +The GNU/C bitfields follow closely the integer representation, with a +particularity on alignment: if a bitfield cannot fit in the current +unit, the unit is padded and the bitfield starts at the following unit. +The unit size is defined by the size of the type `unit_type`. + +TSDL metadata representation: + +~~~ tsdl +unit_type name:size; +~~~ + +As an example, the following structure declared in C compiled by GCC: + +~~~ tsdl +struct example { + short a:12; + short b:5; +}; +~~~ + +The example structure is aligned on the largest element (short). The +second bitfield would be aligned on the next unit boundary, because it +would not fit in the current unit. + + +#### 4.1.7 Floating point + +The floating point values byte ordering is defined in the TSDL metadata. + +Floating point values follow the IEEE 754-2008 standard interchange +formats. Description of the floating point values include the exponent +and mantissa size in bits. Some requirements are imposed on the +floating point values: + +* `FLT_RADIX` must be 2. +* `mant_dig` is the number of digits represented in the mantissa. It is + specified by the ISO C99 standard, section 5.2.4, as `FLT_MANT_DIG`, + `DBL_MANT_DIG` and `LDBL_MANT_DIG` as defined by `<float.h>`. +* `exp_dig` is the number of digits represented in the exponent. Given + that `mant_dig` is one bit more than its actual size in bits (leading + 1 is not needed) and also given that the sign bit always takes one + bit, `exp_dig` can be specified as: + * `sizeof(float) * CHAR_BIT - FLT_MANT_DIG` + * `sizeof(double) * CHAR_BIT - DBL_MANT_DIG` + * `sizeof(long double) * CHAR_BIT - LDBL_MANT_DIG` + +TSDL metadata representation: + +~~~ tsdl +floating_point { + exp_dig = /* value */; + mant_dig = /* value */; + byte_order = /* native OR network OR be OR le */; + align = /* value */; +} +~~~ + +Example of type inheritance: + +~~~ tsdl +typealias floating_point { + exp_dig = 8; /* sizeof(float) * CHAR_BIT - FLT_MANT_DIG */ + mant_dig = 24; /* FLT_MANT_DIG */ + byte_order = native; + align = 32; +} := float; +~~~ + +TODO: define NaN, +inf, -inf behavior. + +Bit-packed, byte-packed or larger alignments can be used for floating +point values, similarly to integers. + + +#### 4.1.8 Enumerations + +Enumerations are a mapping between an integer type and a table of +strings. The numerical representation of the enumeration follows the +integer type specified by the metadata. The enumeration mapping table +is detailed in the enumeration description within the metadata. The +mapping table maps inclusive value ranges (or single values) to strings. +Instead of being limited to simple `value -> string` mappings, these +enumerations map `[ start_value ... end_value ] -> string`, which map +inclusive ranges of values to strings. An enumeration from the C +language can be represented in this format by having the same +`start_value` and `end_value` for each mapping, which is in fact a +range of size 1. This single-value range is supported without repeating +the start and end values with the `value = string` declaration. +Enumerations need to contain at least one entry. + +~~~ tsdl +enum name : integer_type { + somestring = /* start_value1 */ ... /* end_value1 */, + "other string" = /* start_value2 */ ... /* end_value2 */, + yet_another_string, /* will be assigned to end_value2 + 1 */ + "some other string" = /* value */, + /* ... */ +} +~~~ + +If the values are omitted, the enumeration starts at 0 and increment +of 1 for each entry. An entry with omitted value that follows a range +entry takes as value the `end_value` of the previous range + 1: + +~~~ tsdl +enum name : unsigned int { + ZERO, + ONE, + TWO, + TEN = 10, + ELEVEN, +} +~~~ + +Overlapping ranges within a single enumeration are implementation +defined. + +A nameless enumeration can be declared as a field type or as part of +a `typedef`: + +~~~ tsdl +enum : integer_type { + /* ... */ +} +~~~ + +Enumerations omitting the container type `: integer_type` use the `int` +type (for compatibility with C99). The `int` type _must be_ previously +declared, e.g.: + +~~~ tsdl +typealias integer { size = 32; align = 32; signed = true; } := int; + +enum { + /* ... */ +} +~~~ + +### 4.2 Compound types + +Compound are aggregation of type declarations. Compound types include +structures, variant, arrays, sequences, and strings. + + +#### 4.2.1 Structures + +Structures are aligned on the largest alignment required by basic types +contained within the structure. (This follows the ISO/C standard for +structures) + +TSDL metadata representation of a named structure: + +~~~ tsdl +struct name { + field_type field_name; + field_type field_name; + /* ... */ +}; +~~~ + +Example: + +~~~ tsdl +struct example { + integer { /* nameless type */ + size = 16; + signed = true; + align = 16; + } first_field_name; + uint64_t second_field_name; /* named type declared in the metadata */ +}; +~~~ + +The fields are placed in a sequence next to each other. They each +possess a field name, which is a unique identifier within the structure. +The identifier is not allowed to use any [reserved keyword](#specC.1.2). +Replacing reserved keywords with underscore-prefixed field names is +**recommended**. Fields starting with an underscore should have their +leading underscore removed by the CTF trace readers. + +A nameless structure can be declared as a field type or as part of +a `typedef`: + +~~~ tsdl +struct { + /* ... */ +} +~~~ + +Alignment for a structure compound type can be forced to a minimum +value by adding an `align` specifier after the declaration of a +structure body. This attribute is read as: `align(value)`. The value is +specified in bits. The structure will be aligned on the maximum value +between this attribute and the alignment required by the basic types +contained within the structure. e.g. + +~~~ tsdl +struct { + /* ... */ +} align(32) +~~~ + +#### 4.2.2 Variants (discriminated/tagged unions) + +A CTF variant is a selection between different types. A CTF variant must +always be defined within the scope of a structure or within fields +contained within a structure (defined recursively). A _tag_ enumeration +field must appear in either the same static scope, prior to the variant +field (in field declaration order), in an upper static scope, or in an +upper dynamic scope (see [Static and dynamic scopes](#spec7.3.2)). +The type selection is indicated by the mapping from the enumeration +value to the string used as variant type selector. The field to use as +tag is specified by the `tag_field`, specified between `< >` after the +`variant` keyword for unnamed variants, and after _variant name_ for +named variants. It is not required that each enumeration mapping appears +as variant type tag field. It is also not required that each variant +type tag appears as enumeration mapping. However, it is required that +any enumeration mapping encountered within a stream has a matching +variant type tag field. + +The alignment of the variant is the alignment of the type as selected +by the tag value for the specific instance of the variant. The size of +the variant is the size as selected by the tag value for the specific +instance of the variant. + +The alignment of the type containing the variant is independent of the +variant alignment. For instance, if a structure contains two fields, a +32-bit integer, aligned on 32 bits, and a variant, which contains two +choices: either a 32-bit field, aligned on 32 bits, or a 64-bit field, +aligned on 64 bits, the alignment of the outmost structure will be +32-bit (the alignment of its largest field, disregarding the alignment +of the variant). The alignment of the variant will depend on the +selector: if the variant's 32-bit field is selected, its alignment will +be 32-bit, or 64-bit otherwise. It is important to note that variants +are specifically tailored for compactness in a stream. Therefore, the +relative offsets of compound type fields can vary depending on the +offset at which the compound type starts if it contains a variant +that itself contains a type with alignment larger than the largest field +contained within the compound type. This is caused by the fact that the +compound type may contain the enumeration that select the variant's +choice, and therefore the alignment to be applied to the compound type +cannot be determined before encountering the enumeration. + +Each variant type selector possess a field name, which is a unique +identifier within the variant. The identifier is not allowed to use any +[reserved keyword](#C.1.2). Replacing reserved keywords with +underscore-prefixed field names is recommended. Fields starting with an +underscore should have their leading underscore removed by the CTF trace +readers. + +A named variant declaration followed by its definition within a +structure declaration: + +~~~ tsdl +variant name { + field_type sel1; + field_type sel2; + field_type sel3; + /* ... */ +}; + +struct { + enum : integer_type { sel1, sel2, sel3, /* ... */ } tag_field; + /* ... */ + variant name <tag_field> v; +} +~~~ + +An unnamed variant definition within a structure is expressed by the +following TSDL metadata: + +~~~ tsdl +struct { + enum : integer_type { sel1, sel2, sel3, /* ... */ } tag_field; + /* ... */ + variant <tag_field> { + field_type sel1; + field_type sel2; + field_type sel3; + /* ... */ + } v; +} +~~~ + +Example of a named variant within a sequence that refers to a single +tag field: + +~~~ tsdl +variant example { + uint32_t a; + uint64_t b; + short c; +}; + +struct { + enum : uint2_t { a, b, c } choice; + unsigned int seqlen; + variant example <choice> v[seqlen]; +} +~~~ + +Example of an unnamed variant: + +~~~ tsdl +struct { + enum : uint2_t { a, b, c, d } choice; + + /* Unrelated fields can be added between the variant and its tag */ + int32_t somevalue; + variant <choice> { + uint32_t a; + uint64_t b; + short c; + struct { + unsigned int field1; + uint64_t field2; + } d; + } s; +} +~~~ + +Example of an unnamed variant within an array: + +~~~ tsdl +struct { + enum : uint2_t { a, b, c } choice; + variant <choice> { + uint32_t a; + uint64_t b; + short c; + } v[10]; +} +~~~ + +Example of a variant type definition within a structure, where the +defined type is then declared within an array of structures. This +variant refers to a tag located in an upper static scope. This example +clearly shows that a variant type definition referring to the tag `x` +uses the closest preceding field from the static scope of the type +definition. + +~~~ tsdl +struct { + enum : uint2_t { a, b, c, d } x; + + /* + * "x" refers to the preceding "x" enumeration in the + * static scope of the type definition. + */ + typedef variant <x> { + uint32_t a; + uint64_t b; + short c; + } example_variant; + + struct { + enum : int { x, y, z } x; /* This enumeration is not used by "v". */ + + /* "v" uses the "enum : uint2_t { a, b, c, d }" tag. */ + example_variant v; + } a[10]; +} +~~~ + + +#### 4.2.3 Arrays + +Arrays are fixed-length. Their length is declared in the type +declaration within the metadata. They contain an array of _inner type_ +elements, which can refer to any type not containing the type of the +array being declared (no circular dependency). The length is the number +of elements in an array. + +TSDL metadata representation of a named array: + +~~~ tsdl +typedef elem_type name[/* length */]; +~~~ + +A nameless array can be declared as a field type within a +structure, e.g.: + +~~~ tsdl +uint8_t field_name[10]; +~~~ + +Arrays are always aligned on their element alignment requirement. + + +#### 4.2.4 Sequences + +Sequences are dynamically-sized arrays. They refer to a _length_ +unsigned integer field, which must appear in either the same static +scope, prior to the sequence field (in field declaration order), +in an upper static scope, or in an upper dynamic scope +(see [Static and dynamic scopes](#spec7.3.2)). This length field represents +the number of elements in the sequence. The sequence per se is an +array of _inner type_ elements. + +TSDL metadata representation for a sequence type definition: + +~~~ tsdl +struct { + unsigned int length_field; + typedef elem_type typename[length_field]; + typename seq_field_name; +} +~~~ + +A sequence can also be declared as a field type, e.g.: + +~~~ tsdl +struct { + unsigned int length_field; + long seq_field_name[length_field]; +} +~~~ + +Multiple sequences can refer to the same length field, and these length +fields can be in a different upper dynamic scope, e.g., assuming the +`stream.event.header` defines: + +~~~ tsdl +stream { + /* ... */ + id = 1; + event.header := struct { + uint16_t seq_len; + }; +}; + +event { + /* ... */ + stream_id = 1; + fields := struct { + long seq_a[stream.event.header.seq_len]; + char seq_b[stream.event.header.seq_len]; + }; +}; +~~~ + +The sequence elements follow the [array](#spec4.2.3) specifications. + + +#### 4.2.5 Strings + +Strings are an array of _bytes_ of variable size and are terminated by +a `'\0'` "NULL" character. Their encoding is described in the TSDL +metadata. In absence of encoding attribute information, the default +encoding is UTF-8. + +TSDL metadata representation of a named string type: + +~~~ tsdl +typealias string { + encoding = /* UTF8 OR ASCII */; +} := name; +~~~ + +A nameless string type can be declared as a field type: + +~~~ tsdl +string field_name; /* use default UTF8 encoding */ +~~~ + +Strings are always aligned on byte size. + + +## 5. Event packet header + +The event packet header consists of two parts: the +_event packet header_ is the same for all streams of a trace. The +second part, the _event packet context_, is described on a per-stream +basis. Both are described in the TSDL metadata. + +Event packet header (all fields are optional, specified by +TSDL metadata): + + * **Magic number** (CTF magic number: 0xC1FC1FC1) specifies that this is + a CTF packet. This magic number is optional, but when present, it + should come at the very beginning of the packet. + * **Trace UUID**, used to ensure the event packet match the metadata used. + Note: we cannot use a metadata checksum in every cases instead of a + UUID because metadata can be appended to while tracing is active. + This field is optional. + * **Stream ID**, used as reference to stream description in metadata. + This field is optional if there is only one stream description in + the metadata, but becomes required if there are more than one + stream in the TSDL metadata description. + +Event packet context (all fields are optional, specified by +TSDL metadata): + + * Event packet **content size** (in bits). + * Event packet **size** (in bits, includes padding). + * Event packet content checksum. Checksum excludes the event packet + header. + * Per-stream event **packet sequence count** (to deal with UDP packet + loss). The number of significant sequence counter bits should also + be present, so wrap-arounds are dealt with correctly. + * Time-stamp at the beginning and timestamp at the end of the event + packet. Both timestamps are written in the packet header, but + sampled respectively while (or before) writing the first event and + while (or after) writing the last event in the packet. The inclusive + range between these timestamps should include all event timestamps + assigned to events contained within the packet. The timestamp at the + beginning of an event packet is guaranteed to be below or equal the + timestamp at the end of that event packet. The timestamp at the end + of an event packet is guaranteed to be below or equal the + timestamps at the end of any following packet within the same stream. + See [Clocks](#spec8) for more detail. + * **Events discarded count**. Snapshot of a per-stream + free-running counter, counting the number of events discarded that + were supposed to be written in the stream after the last event in + the event packet. Note: producer-consumer buffer full condition can + fill the current event packet with padding so we know exactly where + events have been discarded. However, if the buffer full condition + chooses not to fill the current event packet with padding, all we + know about the timestamp range in which the events have been + discarded is that it is somewhere between the beginning and the end + of the packet. + * Lossless **compression scheme** used for the event packet content. + Applied directly to raw data. New types of compression can be added + in following versions of the format. + * 0: no compression scheme + * 1: bzip2 + * 2: gzip + * 3: xz + * **Cypher** used for the event packet content. Applied after + compression. + * 0: no encryption + * 1: AES + * **Checksum scheme** used for the event packet content. Applied after + encryption. + * 0: no checksum + * 1: md5 + * 2: sha1 + * 3: crc32 + + +### 5.1 Event packet header description + +The event packet header layout is indicated by the +`trace.packet.header` field. Here is a recommended structure type for +the packet header with the fields typically expected (although these +fields are each optional): + +~~~ tsdl +struct event_packet_header { + uint32_t magic; + uint8_t uuid[16]; + uint32_t stream_id; +}; + +trace { + /* ... */ + packet.header := struct event_packet_header; +}; +~~~ + +If the magic number is not present, tools such as `file` will have no +mean to discover the file type. + +If the uuid is not present, no validation that the metadata actually +corresponds to the stream is performed. + +If the stream_id packet header field is missing, the trace can only +contain a single stream. Its `id` field can be left out, and its events +don't need to declare a `stream_id` field. + + +### 5.2 Event packet context description + +Event packet context example. These are declared within the stream +declaration in the metadata. All these fields are optional. If the +packet size field is missing, the whole stream only contains a single +packet. If the content size field is missing, the packet is filled +(no padding). The content and packet sizes include all headers. + +An example event packet context type: + +~~~ tsdl +struct event_packet_context { + uint64_t timestamp_begin; + uint64_t timestamp_end; + uint32_t checksum; + uint32_t stream_packet_count; + uint32_t events_discarded; + uint32_t cpu_id; + uint64_t content_size; + uint64_t packet_size; + uint8_t compression_scheme; + uint8_t encryption_scheme; + uint8_t checksum_scheme; +}; +~~~ + + +## 6. Event Structure + +The overall structure of an event is: + + 1. Event header (as specified by the stream metadata) + 2. Stream event context (as specified by the stream metadata) + 3. Event context (as specified by the event metadata) + 4. Event payload (as specified by the event metadata) + +This structure defines an implicit dynamic scoping, where variants +located in inner structures (those with a higher number in the listing +above) can refer to the fields of outer structures (with lower number +in the listing above). See [TSDL scopes](#spec7.3) for more detail. + +The total length of an event is defined as the difference between the +end of its event payload and the end of the previous event's event +payload. Therefore, it includes the event header alignment padding, and +all its fields and their respective alignment padding. Events of length +0 are forbidden. + + +### 6.1 Event header + +Event headers can be described within the metadata. We hereby propose, +as an example, two types of events headers. Type 1 accommodates streams +with less than 31 event IDs. Type 2 accommodates streams with 31 or +more event IDs. + +One major factor can vary between streams: the number of event IDs +assigned to a stream. Luckily, this information tends to stay +relatively constant (modulo event registration while trace is being +recorded), so we can specify different representations for streams +containing few event IDs and streams containing many event IDs, so we +end up representing the event ID and timestamp as densely as possible +in each case. + +The header is extended in the rare occasions where the information +cannot be represented in the ranges available in the standard event +header. They are also used in the rare occasions where the data +required for a field could not be collected: the flag corresponding to +the missing field within the `missing_fields` array is then set to 1. + +Types `uintX_t` represent an `X`-bit unsigned integer, as declared with +either: + +~~~ tsdl +typealias integer { + size = /* X */; + align = /* X */; + signed = false; +} := uintX_t; +~~~ + +or + +~~~ tsdl +typealias integer { + size = /* X */; + align = 1; + signed = false; +} := uintX_t; +~~~ + +For more information about timestamp fields, see [Clocks](#spec8). + + +#### 6.1.1 Type 1: few event IDs + + * Aligned on 32-bit (or 8-bit if byte-packed, depending on the + architecture preference) + * Native architecture byte ordering + * For `compact` selection, fixed size of 32 bits + * For "extended" selection, size depends on the architecture and + variant alignment + +~~~ tsdl +struct event_header_1 { + /* + * id: range: 0 - 30. + * id 31 is reserved to indicate an extended header. + */ + enum : uint5_t { compact = 0 ... 30, extended = 31 } id; + variant <id> { + struct { + uint27_t timestamp; + } compact; + struct { + uint32_t id; /* 32-bit event IDs */ + uint64_t timestamp; /* 64-bit timestamps */ + } extended; + } v; +} align(32); /* or align(8) */ +~~~ + + +#### 6.1.2 Type 2: many event IDs + + * Aligned on 16-bit (or 8-bit if byte-packed, depending on the + architecture preference) + * Native architecture byte ordering + * For `compact` selection, size depends on the architecture and + variant alignment + * For `extended` selection, size depends on the architecture and + variant alignment + +~~~ tsdl +struct event_header_2 { + /* + * id: range: 0 - 65534. + * id 65535 is reserved to indicate an extended header. + */ + enum : uint16_t { compact = 0 ... 65534, extended = 65535 } id; + variant <id> { + struct { + uint32_t timestamp; + } compact; + struct { + uint32_t id; /* 32-bit event IDs */ + uint64_t timestamp; /* 64-bit timestamps */ + } extended; + } v; +} align(16); /* or align(8) */ +~~~ + + +### 6.2 Stream event context and event context + +The event context contains information relative to the current event. +The choice and meaning of this information is specified by the TSDL +stream and event metadata descriptions. The stream context is applied +to all events within the stream. The stream context structure follows +the event header. The event context is applied to specific events. Its +structure follows the stream context structure. + +An example of stream-level event context is to save the event payload +size with each event, or to save the current PID with each event. +These are declared within the stream declaration within the metadata: + +~~~ tsdl +stream { + /* ... */ + event.context := struct { + uint pid; + uint16_t payload_size; + }; +}; +~~~ + +An example of event-specific event context is to declare a bitmap of +missing fields, only appended after the stream event context if the +extended event header is selected. `NR_FIELDS` is the number of fields +within the event (a numeric value). + +~~~ tsdl +event { + context := struct { + variant <id> { + struct { } compact; + struct { + /* missing event fields bitmap */ + uint1_t missing_fields[NR_FIELDS]; + } extended; + } v; + }; + /* ... */ +} +~~~ + + +### 6.3 Event payload + +An event payload contains fields specific to a given event type. The +fields belonging to an event type are described in the event-specific +metadata within a structure type. + + +#### 6.3.1 Padding + +No padding at the end of the event payload. This differs from the ISO/C +standard for structures, but follows the CTF standard for structures. +In a trace, even though it makes sense to align the beginning of a +structure, it really makes no sense to add padding at the end of the +structure, because structures are usually not followed by a structure +of the same type. + +This trick can be done by adding a zero-length `end` field at the end +of the C structures, and by using the offset of this field rather than +using `sizeof()` when calculating the size of a structure +(see [Helper macros](#specA)). + + +#### 6.3.2 Alignment + +The event payload is aligned on the largest alignment required by types +contained within the payload. This follows the ISO/C standard for +structures. + + +## 7. Trace Stream Description Language (TSDL) + +The Trace Stream Description Language (TSDL) allows expression of the +binary trace streams layout in a C99-like Domain Specific Language +(DSL). + + +### 7.1 Meta-data + +The trace stream layout description is located in the trace metadata. +The metadata is itself located in a stream identified by its name: +`metadata`. + +The metadata description can be expressed in two different formats: +text-only and packet-based. The text-only description facilitates +generation of metadata and provides a convenient way to enter the +metadata information by hand. The packet-based metadata provides the +CTF stream packet facilities (checksumming, compression, encryption, +network-readiness) for metadata stream generated and transported by a +tracer. + +The text-only metadata file is a plain-text TSDL description. This file +must begin with the following characters to identify the file as a CTF +TSDL text-based metadata file (without the double-quotes): + +~~~ text +"/* CTF" +~~~ + +It must be followed by a space, and the version of the specification +followed by the CTF trace, e.g.: + +~~~ text +" 1.8" +~~~ + +These characters allow automated discovery of file type and CTF +specification version. They are interpreted as a the beginning of a +comment by the TSDL metadata parser. The comment can be continued to +contain extra commented characters before it is closed. + +The packet-based metadata is made of _metadata packets_, which each +start with a metadata packet header. The packet-based metadata +description is detected by reading the magic number 0x75D11D57 at the +beginning of the file. This magic number is also used to detect the +endianness of the architecture by trying to read the CTF magic number +and its counterpart in reversed endianness. The events within the +metadata stream have no event header nor event context. Each event only +contains a special _sequence_ payload, which is a sequence of bits which +length is implicitly calculated by using the +`trace.packet.header.content_size` field, minus the packet header size. +The formatting of this sequence of bits is a plain-text representation +of the TSDL description. Each metadata packet start with a special +packet header, specific to the metadata stream, which contains, +exactly: + +~~~ tsdl +struct metadata_packet_header { + uint32_t magic; /* 0x75D11D57 */ + uint8_t uuid[16]; /* Unique Universal Identifier */ + uint32_t checksum; /* 0 if unused */ + uint32_t content_size; /* in bits */ + uint32_t packet_size; /* in bits */ + uint8_t compression_scheme; /* 0 if unused */ + uint8_t encryption_scheme; /* 0 if unused */ + uint8_t checksum_scheme; /* 0 if unused */ + uint8_t major; /* CTF spec version major number */ + uint8_t minor; /* CTF spec version minor number */ +}; +~~~ + +The packet-based metadata can be converted to a text-only metadata by +concatenating all the strings it contains. + +In the textual representation of the metadata, the text contained +within `/*` and `*/`, as well as within `//` and end of line, are +treated as comments. Boolean values can be represented as `true`, +`TRUE`, or `1` for true, and `false`, `FALSE`, or `0` for false. Within +the string-based metadata description, the trace UUID is represented as +a string of hexadecimal digits and dashes `-`. In the event packet +header, the trace UUID is represented as an array of bytes. + + +### 7.2 Declaration vs definition + +A declaration associates a layout to a type, without specifying where +this type is located in the event [structure hierarchy](#spec6). +This therefore includes `typedef`, `typealias`, as well as all type +specifiers. In certain circumstances (`typedef`, structure field and +variant field), a declaration is followed by a declarator, which specify +the newly defined type name (for `typedef`), or the field name (for +declarations located within structure and variants). Array and sequence, +declared with square brackets (`[` `]`), are part of the declarator, +similarly to C99. The enumeration base type is specified by +`: enum_base`, which is part of the type specifier. The variant tag +name, specified between `<` `>`, is also part of the type specifier. + +A definition associates a type to a location in the event +[structure hierarchy](#spec6). This association is denoted by `:=`, +as shown in [TSDL scopes](#spec7.3). + + +### 7.3 TSDL scopes + +TSDL uses three different types of scoping: a lexical scope is used for +declarations and type definitions, and static and dynamic scopes are +used for variants references to tag fields (with relative and absolute +path lookups) and for sequence references to length fields. + + +#### 7.3.1 Lexical Scope + +Each of `trace`, `env`, `stream`, `event`, `struct` and `variant` have +their own nestable declaration scope, within which types can be declared +using `typedef` and `typealias`. A root declaration scope also contains +all declarations located outside of any of the aforementioned +declarations. An inner declaration scope can refer to type declared +within its container lexical scope prior to the inner declaration scope. +Redefinition of a typedef or typealias is not valid, although hiding an +upper scope typedef or typealias is allowed within a sub-scope. + + +#### 7.3.2 Static and dynamic scopes + +A local static scope consists in the scope generated by the declaration +of fields within a compound type. A static scope is a local static scope +augmented with the nested sub-static-scopes it contains. + +A dynamic scope consists in the static scope augmented with the +implicit [event structure](#spec6) definition hierarchy. + +Multiple declarations of the same field name within a local static scope +is not valid. It is however valid to re-use the same field name in +different local scopes. + +Nested static and dynamic scopes form lookup paths. These are used for +variant tag and sequence length references. They are used at the variant +and sequence definition site to look up the location of the tag field +associated with a variant, and to lookup up the location of the length +field associated with a sequence. + +Variants and sequences can refer to a tag field either using a relative +path or an absolute path. The relative path is relative to the scope in +which the variant or sequence performing the lookup is located. +Relative paths are only allowed to lookup within the same static scope, +which includes its nested static scopes. Lookups targeting parent static +scopes need to be performed with an absolute path. + +Absolute path lookups use the full path including the dynamic scope +followed by a `.` and then the static scope. Therefore, variants (or +sequences) in lower levels in the dynamic scope (e.g., event context) +can refer to a tag (or length) field located in upper levels +(e.g., in the event header) by specifying, in this case, the associated +tag with `<stream.event.header.field_name>`. This allows, for instance, +the event context to define a variant referring to the `id` field of +the event header as selector. + +The dynamic scope prefixes are thus: + + * Trace environment: `<env. >` + * Trace packet header: `<trace.packet.header. >` + * Stream packet context: `<stream.packet.context. >` + * Event header: `<stream.event.header. >` + * Stream event context: `<stream.event.context. >` + * Event context: `<event.context. >` + * Event payload: `<event.fields. >` + +The target dynamic scope must be specified explicitly when referring to +a field outside of the static scope (absolute scope reference). No +conflict can occur between relative and dynamic paths, because the +keywords `trace`, `stream`, and `event` are reserved, and thus not +permitted as field names. It is recommended that field names clashing +with CTF and C99 reserved keywords use an underscore prefix to +eliminate the risk of generating a description containing an invalid +field name. Consequently, fields starting with an underscore should have +their leading underscore removed by the CTF trace readers. + +The information available in the dynamic scopes can be thought of as the +current tracing context. At trace production, information about the +current context is saved into the specified scope field levels. At trace +consumption, for each event, the current trace context is therefore +readable by accessing the upper dynamic scopes. + + +### 7.4 TSDL examples + +The grammar representing the TSDL metadata is presented in +[TSDL grammar](#specC). This section presents a rather lighter reading that +consists in examples of TSDL metadata, with template values. + +The stream ID can be left out if there is only one stream in the +trace. The event `id` field can be left out if there is only one event +in a stream. + +~~~ tsdl +trace { + major = /* value */; /* CTF spec version major number */ + minor = /* value */; /* CTF spec version minor number */ + uuid = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"; /* Trace UUID */ + byte_order = /* be OR le */; /* Endianness (required) */ + packet.header := struct { + uint32_t magic; + uint8_t uuid[16]; + uint32_t stream_id; + }; +}; + +/* + * The "env" (environment) scope contains assignment expressions. The + * field names and content are implementation-defined. + */ +env { + pid = /* value */; /* example */ + proc_name = "name"; /* example */ + /* ... */ +}; + +stream { + id = /* stream_id */; + /* Type 1 - Few event IDs; Type 2 - Many event IDs. See section 6.1. */ + event.header := /* event_header_1 OR event_header_2 */; + event.context := struct { + /* ... */ + }; + packet.context := struct { + /* ... */ + }; +}; + +event { + name = "event_name"; + id = /* value */; /* Numeric identifier within the stream */ + stream_id = /* stream_id */; + loglevel = /* value */; + model.emf.uri = "string"; + context := struct { + /* ... */ + }; + fields := struct { + /* ... */ + }; +}; + +callsite { + name = "event_name"; + func = "func_name"; + file = "myfile.c"; + line = 39; + ip = 0x40096c; +}; +~~~ + +More detail on [types](#spec4): + +~~~ tsdl +/* + * Named types: + * + * Type declarations behave similarly to the C standard. + */ + +typedef aliased_type_specifiers new_type_declarators; + +/* e.g.: typedef struct example new_type_name[10]; */ + +/* + * typealias + * + * The "typealias" declaration can be used to give a name (including + * pointer declarator specifier) to a type. It should also be used to + * map basic C types (float, int, unsigned long, ...) to a CTF type. + * Typealias is a superset of "typedef": it also allows assignment of a + * simple variable identifier to a type. + */ + +typealias type_class { + /* ... */ +} := type_specifiers type_declarator; + +/* + * e.g.: + * typealias integer { + * size = 32; + * align = 32; + * signed = false; + * } := struct page *; + * + * typealias integer { + * size = 32; + * align = 32; + * signed = true; + * } := int; + */ + +struct name { + /* ... */ +}; + +variant name { + /* ... */ +}; + +enum name : integer_type { + /* ... */ +}; +~~~ + +Unnamed types, contained within compound type fields, `typedef` or +`typealias`: + +~~~ tsdl +struct { + /* ... */ +} +~~~ + +~~~ tsdl +struct { + /* ... */ +} align(value) +~~~ + +~~~ tsdl +variant { + /* ... */ +} +~~~ + +~~~ tsdl +enum : integer_type { + /* ... */ +} +~~~ + +~~~ tsdl +typedef type new_type[length]; + +struct { + type field_name[length]; +} +~~~ + +~~~ tsdl +typedef type new_type[length_type]; + +struct { + type field_name[length_type]; +} +~~~ + +~~~ tsdl +integer { + /* ... */ +} +~~~ + +~~~ tsdl +floating_point { + /* ... */ +} +~~~ + +~~~ tsdl +struct { + integer_type field_name:size; /* GNU/C bitfield */ +} +~~~ + +~~~ tsdl +struct { + string field_name; +} +~~~ + + +## 8. Clocks + +Clock metadata allows to describe the clock topology of the system, as +well as to detail each clock parameter. In absence of clock description, +it is assumed that all fields named `timestamp` use the same clock +source, which increments once per nanosecond. + +Describing a clock and how it is used by streams is threefold: first, +the clock and clock topology should be described in a `clock` +description block, e.g.: + +~~~ tsdl +clock { + name = cycle_counter_sync; + uuid = "62189bee-96dc-11e0-91a8-cfa3d89f3923"; + description = "Cycle counter synchronized across CPUs"; + freq = 1000000000; /* frequency, in Hz */ + /* precision in seconds is: 1000 * (1/freq) */ + precision = 1000; + /* + * clock value offset from Epoch is: + * offset_s + (offset * (1/freq)) + */ + offset_s = 1326476837; + offset = 897235420; + absolute = FALSE; +}; +~~~ + +The mandatory `name` field specifies the name of the clock identifier, +which can later be used as a reference. The optional field `uuid` is +the unique identifier of the clock. It can be used to correlate +different traces that use the same clock. An optional textual +description string can be added with the `description` field. The +`freq` field is the initial frequency of the clock, in Hz. If the +`freq` field is not present, the frequency is assumed to be 1000000000 +(providing clock increment of 1 ns). The optional `precision` field +details the uncertainty on the clock measurements, in (1/freq) units. +The `offset_s` and `offset` fields indicate the offset from +POSIX.1 Epoch, 1970-01-01 00:00:00 +0000 (UTC), to the zero of value +of the clock. The `offset_s` field is in seconds. The `offset` field is +in (1/freq) units. If any of the `offset_s` or `offset` field is not +present, it is assigned the 0 value. The field `absolute` is `TRUE` if +the clock is a global reference across different clock UUID +(e.g. NTP time). Otherwise, `absolute` is `FALSE`, and the clock can +be considered as synchronized only with other clocks that have the same +UUID. + +Secondly, a reference to this clock should be added within an integer +type: + +~~~ tsdl +typealias integer { + size = 64; align = 1; signed = false; + map = clock.cycle_counter_sync.value; +} := uint64_ccnt_t; +~~~ + +Thirdly, stream declarations can reference the clock they use as a +timestamp source: + +~~~ tsdl +struct packet_context { + uint64_ccnt_t ccnt_begin; + uint64_ccnt_t ccnt_end; + /* ... */ +}; + +stream { + /* ... */ + event.header := struct { + uint64_ccnt_t timestamp; + /* ... */ + }; + packet.context := struct packet_context; +}; +~~~ + +For a N-bit integer type referring to a clock, if the integer overflows +compared to the N low order bits of the clock prior value found in the +same stream, then it is assumed that one, and only one, overflow +occurred. It is therefore important that events encoding time on a small +number of bits happen frequently enough to detect when more than one +N-bit overflow occurs. + +In a packet context, clock field names ending with `_begin` and `_end` +have a special meaning: this refers to the timestamps at, respectively, +the beginning and the end of each packet. + + +## A. Helper macros + +The two following macros keep track of the size of a GNU/C structure +without padding at the end by placing HEADER_END as the last field. +A one byte end field is used for C90 compatibility (C99 flexible arrays +could be used here). Note that this does not affect the effective +structure size, which should always be calculated with the +`header_sizeof()` helper. + +~~~ c +#define HEADER_END char end_field +#define header_sizeof(type) offsetof(typeof(type), end_field) +~~~ + +## B. Stream header rationale + +An event stream is divided in contiguous event packets of variable +size. These subdivisions allow the trace analyzer to perform a fast +binary search by time within the stream (typically requiring to index +only the event packet headers) without reading the whole stream. These +subdivisions have a variable size to eliminate the need to transfer the +event packet padding when partially filled event packets must be sent +when streaming a trace for live viewing/analysis. An event packet can +contain a certain amount of padding at the end. Dividing streams into +event packets is also useful for network streaming over UDP and flight +recorder mode tracing (a whole event packet can be swapped out of the +buffer atomically for reading). + +The stream header is repeated at the beginning of each event packet to +allow flexibility in terms of: + + * streaming support + * allowing arbitrary buffers to be discarded without making the trace + unreadable + * allow UDP packet loss handling by either dealing with missing eventpacket
common-trace-format-specification.md new file +10
-0+ or asking for re-transmission + * transparently support flight recorder mode + * transparently support crash dump + + +## C. TSDL Grammar + +~~~ c +/* + * Common Trace Format (CTF) Trace Stream Description Language (TSDL)Grammar.
+ *
+ * Inspired from the C99 grammar:
+ * http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf (Annex A)
+ * and c++1x grammar (draft)
+ * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3291.pdf (Annex
A)
+ *
+ * Specialized for CTF needs by including only constant and declarations from
+ * C99 (excluding function declarations), and by adding support for variants,
+ * sequences and CTF-specific specifiers. Enumeration container types
+ * semantic is inspired from c++1x enum-base.
+ */
+~~~common-trace-format-specification.md new file +279
-0+ + +### C.1 Lexical grammar + + +#### C.1.1 Lexical elements + +~~~ text +token: + keyword + identifier + constant + string-literal + punctuator +~~~ + +#### C.1.2 Keywords + +~~~ text +keyword: is one of + +align +callsite +const +char +clock +double +enum +env +event +floating_point +float +integer +int +long +short +signed +stream +string +struct +trace +typealias +typedef +unsigned +variant +void +_Bool +_Complex +_Imaginary +~~~ + + +#### C.1.3 Identifiers + +~~~ text +identifier: + identifier-nondigit + identifier identifier-nondigit + identifier digit + +identifier-nondigit: + nondigit + universal-character-name + any other implementation-defined characters + +nondigit: + _ + [a-zA-Z] /* regular expression */ + +digit: + [0-9] /* regular expression */ +~~~ + + +#### C.1.4 Universal character names + +~~~ text +universal-character-name: + \u hex-quad + \U hex-quad hex-quad + +hex-quad: + hexadecimal-digit hexadecimal-digit hexadecimal-digit hexadecimal-digit +~~~ + + +##### C.1.5 Constants + +~~~ text +constant: + integer-constant + enumeration-constant + character-constant + +integer-constant: + decimal-constant integer-suffix-opt + octal-constant integer-suffix-opt + hexadecimal-constant integer-suffix-opt + +decimal-constant: + nonzero-digit + decimal-constant digit + +octal-constant: + 0 + octal-constant octal-digit + +hexadecimal-constant: + hexadecimal-prefix hexadecimal-digit + hexadecimal-constant hexadecimal-digit + +hexadecimal-prefix: + 0x + 0X + +nonzero-digit: + [1-9] + +integer-suffix: + unsigned-suffix long-suffix-opt + unsigned-suffix long-long-suffix + long-suffix unsigned-suffix-opt + long-long-suffix unsigned-suffix-opt + +unsigned-suffix: + u + U + +long-suffix: + l + L + +long-long-suffix: + ll + LL + +enumeration-constant: + identifier + string-literal + +character-constant: + ' c-char-sequence ' + L' c-char-sequence ' + +c-char-sequence: + c-char + c-char-sequence c-char + +c-char: + any member of source charset except single-quote ('), backslash + (\), or new-line character. + escape-sequence + +escape-sequence: + simple-escape-sequence + octal-escape-sequence + hexadecimal-escape-sequence + universal-character-name + +simple-escape-sequence: one of + \' \" \? \\ \a \b \f \n \r \t \v + +octal-escape-sequence: + \ octal-digit + \ octal-digit octal-digit + \ octal-digit octal-digit octal-digit + +hexadecimal-escape-sequence: + \x hexadecimal-digit + hexadecimal-escape-sequence hexadecimal-digit +~~~ + + +#### C.1.6 String literals + +~~~ text +string-literal: + " s-char-sequence-opt " + L" s-char-sequence-opt " + +s-char-sequence: + s-char + s-char-sequence s-char + +s-char: + any member of source charset except double-quote ("), backslash + (\), or new-line character. + escape-sequence +~~~ + + +#### C.1.7 Punctuators + +~~~ text +punctuator: one of + [ ] ( ) { } . -> * + - < > : ; ... = , +~~~ + + +### C.2 Phrase structure grammar + +~~~ text +primary-expression: + identifier + constant + string-literal + ( unary-expression ) + +postfix-expression: + primary-expression + postfix-expression [ unary-expression ] + postfix-expression . identifier + postfix-expressoin -> identifier + +unary-expression: + postfix-expression + unary-operator postfix-expression + +unary-operator: one of + + - + +assignment-operator: + = + +type-assignment-operator: + := + +constant-expression-range: + unary-expression ... unary-expression +~~~ + + +#### C.2.2 Declarations: + +~~~ text +declaration: + declaration-specifiers declarator-list-opt ; + ctf-specifier ; + +declaration-specifiers: + storage-class-specifier declaration-specifiers-opt + type-specifier declaration-specifiers-opt + type-qualifier declaration-specifiers-opt + +declarator-list: + declarator + declarator-list , declarator + +abstract-declarator-list: + abstract-declarator + abstract-declarator-list , abstract-declarator + +storage-class-specifier: + typedef + +type-specifier: + void + char + short + int + long + float + double + signed + unsigned + _Bool + _Complex + _Imaginary + struct-specifier + variant-specifier + enum-specifier + typedef-name + ctf-type-specifier + +align-attribute: + align ( unary-expression ) + +struct-specifier: + struct identifier-opt { struct-or-variant-declaration-list-opt }align-attribute-opt
common-trace-format-specification.md new file +9
-0+ struct identifier align-attribute-opt + +struct-or-variant-declaration-list: + struct-or-variant-declaration + struct-or-variant-declaration-list struct-or-variant-declaration + +struct-or-variant-declaration: + specifier-qualifier-list struct-or-variant-declarator-list ; + declaration-specifiers-opt storage-class-specifierdeclaration-specifiers-opt declarator-list ;
+ typealias declaration-specifiers abstract-declarator-list
type-assignment-operator declaration-specifiers abstract-declarator-list ;
+ typealias declaration-specifiers abstract-declarator-list
type-assignment-operator declarator-list ;common-trace-format-specification.md new file +15
-0+ +specifier-qualifier-list: + type-specifier specifier-qualifier-list-opt + type-qualifier specifier-qualifier-list-opt + +struct-or-variant-declarator-list: + struct-or-variant-declarator + struct-or-variant-declarator-list , struct-or-variant-declarator + +struct-or-variant-declarator: + declarator + declarator-opt : unary-expression + +variant-specifier: + variant identifier-opt variant-tag-opt {struct-or-variant-declaration-list }
common-trace-format-specification.md new file +65
-0+ variant identifier variant-tag + +variant-tag: + < unary-expression > + +enum-specifier: + enum identifier-opt { enumerator-list } + enum identifier-opt { enumerator-list , } + enum identifier + enum identifier-opt : declaration-specifiers { enumerator-list } + enum identifier-opt : declaration-specifiers { enumerator-list , } + +enumerator-list: + enumerator + enumerator-list , enumerator + +enumerator: + enumeration-constant + enumeration-constant assignment-operator unary-expression + enumeration-constant assignment-operator constant-expression-range + +type-qualifier: + const + +declarator: + pointer-opt direct-declarator + +direct-declarator: + identifier + ( declarator ) + direct-declarator [ unary-expression ] + +abstract-declarator: + pointer-opt direct-abstract-declarator + +direct-abstract-declarator: + identifier-opt + ( abstract-declarator ) + direct-abstract-declarator [ unary-expression ] + direct-abstract-declarator [ ] + +pointer: + * type-qualifier-list-opt + * type-qualifier-list-opt pointer + +type-qualifier-list: + type-qualifier + type-qualifier-list type-qualifier + +typedef-name: + identifier +~~~ + + +#### C.2.3 CTF-specific declarations + +~~~ text +ctf-specifier: + clock { ctf-assignment-expression-list-opt } + event { ctf-assignment-expression-list-opt } + stream { ctf-assignment-expression-list-opt } + env { ctf-assignment-expression-list-opt } + trace { ctf-assignment-expression-list-opt } + callsite { ctf-assignment-expression-list-opt } + typealias declaration-specifiers abstract-declarator-listtype-assignment-operator declaration-specifiers abstract-declarator-list
+ typealias declaration-specifiers abstract-declarator-list
type-assignment-operator declarator-listcommon-trace-format-specification.md new file +15
-0+ +ctf-type-specifier: + floating_point { ctf-assignment-expression-list-opt } + integer { ctf-assignment-expression-list-opt } + string { ctf-assignment-expression-list-opt } + string + +ctf-assignment-expression-list: + ctf-assignment-expression ; + ctf-assignment-expression-list ctf-assignment-expression ; + +ctf-assignment-expression: + unary-expression assignment-operator unary-expression + unary-expression type-assignment-operator type-specifier + declaration-specifiers-opt storage-class-specifierdeclaration-specifiers-opt declarator-list
+ typealias declaration-specifiers abstract-declarator-list
type-assignment-operator declaration-specifiers abstract-declarator-list
+ typealias declaration-specifiers abstract-declarator-list
type-assignment-operator declarator-list
+~~~common-trace-format-specification.txt deleted +0
-6@@ -1,1823 +0,0 @@ -Common Trace Format (CTF) Specification (v1.8.2) - -Mathieu Desnoyers, EfficiOS Inc. - -The goal of the present document is to specify a trace format that suits the -needs of the embedded, telecom, high-performance and kernel communities. Itis
-based on the Common Trace Format Requirements (v1.4) document. It is designed tocommon-trace-format-specification.txt deleted +0
-11-allow traces to be natively generated by the Linux kernel, Linux user-space -applications written in C/C++, and hardware components. One major element of -CTF is the Trace Stream Description Language (TSDL) which flexibility -enables description of various binary trace stream layouts. - -The latest version of this document can be found at: - - git tree: git://git.efficios.com/ctf.git - gitweb: http://git.efficios.com/?p=ctf.git - -A reference implementation of a library to read and write this trace formatis
common-trace-format-specification.txt deleted +0
-76-being implemented within the BabelTrace project, a converter between trace -formats. The development tree is available at: - - git tree: git://git.efficios.com/babeltrace.git - gitweb: http://git.efficios.com/?p=babeltrace.git - -The CE Workgroup of the Linux Foundation, Ericsson, and EfficiOS have -sponsored this work. - - -Table of Contents - -1. Preliminary definitions -2. High-level representation of a trace -3. Event stream -4. Types - 4.1 Basic types - 4.1.1 Type inheritance - 4.1.2 Alignment - 4.1.3 Byte order - 4.1.4 Size - 4.1.5 Integers - 4.1.6 GNU/C bitfields - 4.1.7 Floating point - 4.1.8 Enumerations - 4.2 Compound types - 4.2.1 Structures - 4.2.2 Variants (Discriminated/Tagged Unions) - 4.2.3 Arrays - 4.2.4 Sequences - 4.2.5 Strings -5. Event Packet Header - 5.1 Event Packet Header Description - 5.2 Event Packet Context Description -6. Event Structure - 6.1 Event Header - 6.1.1 Type 1 - Few event IDs - 6.1.2 Type 2 - Many event IDs - 6.2 Stream Event Context and Event Context - 6.3 Event Payload - 6.3.1 Padding - 6.3.2 Alignment -7. Trace Stream Description Language (TSDL) - 7.1 Meta-data - 7.2 Declaration vs Definition - 7.3 TSDL Scopes - 7.3.1 Lexical Scope - 7.3.2 Static and Dynamic Scopes - 7.4 TSDL Examples -8. Clocks - - -1. Preliminary definitions - - - Event Trace: An ordered sequence of events. - - Event Stream: An ordered sequence of events, containing a subset of the - trace event types. - - Event Packet: A sequence of physically contiguous events within an event - stream. - - Event: This is the basic entry in a trace. (aka: a trace record). - - An event identifier (ID) relates to the class (a type) of event within - an event stream. - e.g. event: irq_entry. - - An event (or event record) relates to a specific instance of an event - class. - e.g. event: irq_entry, at time X, on CPU Y - - Source Architecture: Architecture writing the trace. - - Reader Architecture: Architecture reading the trace. - - -2. High-level representation of a trace - -A trace is divided into multiple event streams. Each event stream contains a -subset of the trace event types. - -The final output of the trace, after its generation and optional transportover
common-trace-format-specification.txt deleted +0
-2-the network, is expected to be either on permanent or temporary storage in a -virtual file system. Because each event stream is appended to while a traceis
common-trace-format-specification.txt deleted +0
-47-being recorded, each is associated with a distinct set of files for -output. Therefore, a stored trace can be represented as a directory -containing zero, one or more files per stream. - -Meta-data description associated with the trace contains information on -trace event types expressed in the Trace Stream Description Language -(TSDL). This language describes: - -- Trace version. -- Types available. -- Per-trace event header description. -- Per-stream event header description. -- Per-stream event context description. -- Per-event - - Event type to stream mapping. - - Event type to name mapping. - - Event type to ID mapping. - - Event context description. - - Event fields description. - - -3. Event stream - -An event stream can be divided into contiguous event packets of variable -size. An event packet can contain a certain amount of padding at the -end. The stream header is repeated at the beginning of each event -packet. The rationale for the event stream design choices is explained -in Appendix B. Stream Header Rationale. - -The event stream header will therefore be referred to as the "event packet -header" throughout the rest of this document. - - -4. Types - -Types are organized as type classes. Each type class belong to either of two -kind of types: basic types or compound types. - -4.1 Basic types - -A basic type is a scalar type, as described in this section. It includes -integers, GNU/C bitfields, enumerations, and floating point values. - -4.1.1 Type inheritance - -Type specifications can be inherited to allow deriving types from a -type class. For example, see the uint32_t named type derived from the"integer"
common-trace-format-specification.txt deleted +0
-2-type class below ("Integers" section). Types have a precise binary -representation in the trace. A type class has methods to read and writethese
common-trace-format-specification.txt deleted +0
-42-types, but must be derived into a type to be usable in an event field. - -4.1.2 Alignment - -We define "byte-packed" types as aligned on the byte size, namely 8-bit. -We define "bit-packed" types as following on the next bit, as defined by the -"Integers" section. - -Each basic type must specify its alignment, in bits. Examples of -possible alignments are: bit-packed (align = 1), byte-packed (align = -8), or word-aligned (e.g. align = 32 or align = 64). The choice depends -on the architecture preference and compactness vs performance trade-offs -of the implementation. Architectures providing fast unaligned write -byte-packed basic types to save space, aligning each type on byte -boundaries (8-bit). Architectures with slow unaligned writes align types -on specific alignment values. If no specific alignment is declared for a -type, it is assumed to be bit-packed for integers with size not multiple -of 8 bits and for gcc bitfields. All other basic types are byte-packed -by default. It is however recommended to always specify the alignment -explicitly. Alignment values must be power of two. Compound types are -aligned as specified in their individual specification. - -The base offset used for field alignment is the start of the packet -containing the field. For instance, a field aligned on 32-bit needs to -be at an offset multiple of 32-bit from the start of the packet that -contains it. - -TSDL meta-data attribute representation of a specific alignment: - - align = value; /* value in bits */ - -4.1.3 Byte order - -By default, byte order of a basic type is the byte order described in -the trace description. It can be overridden by specifying a -"byte_order" attribute for a basic type. Typical use-case is to specify -the network byte order (big endian: "be") to save data captured from the -network into the trace without conversion. - -TSDL meta-data representation: - - byte_order = native OR network OR be OR le; /* network and be are aliases*/
common-trace-format-specification.txt deleted +0
-12- -The "native" keyword selects the byte order described in the trace -description. The "network" byte order is an alias for big endian. - -Even though the trace description section is not per se a type, for sake -of clarity, it should be noted that "native" and "network" byte orders -are only allowed within type declaration. The byte_order specified in -the trace description section only accepts "be" or "le" values. - -4.1.4 Size - -Type size, in bits, for integers and floats is that returned by "sizeof()"in C
common-trace-format-specification.txt deleted +0
-2-multiplied by CHAR_BIT. -We require the size of "char" and "unsigned char" types (CHAR_BIT) to befixed
common-trace-format-specification.txt deleted +0
-20-to 8 bits for cross-endianness compatibility. - -TSDL meta-data representation: - - size = value; (value is in bits) - -4.1.5 Integers - -Signed integers are represented in two-complement. Integer alignment, -size, signedness and byte ordering are defined in the TSDL meta-data. -Integers aligned on byte size (8-bit) and with length multiple of byte -size (8-bit) correspond to the C99 standard integers. In addition, -integers with alignment and/or size that are _not_ a multiple of the -byte size are permitted; these correspond to the C99 standard bitfields, -with the added specification that the CTF integer bitfields have a fixed -binary representation. Integer size needs to be a positive integer. -Integers of size 0 are forbidden. A MIT-licensed reference -implementation of the CTF portable bitfields is available at: - -http://git.efficios.com/?p=babeltrace.git;a=blob;f=include/babeltrace/bitfield.h
common-trace-format-specification.txt deleted +0
-5- -Binary representation of integers: - -- On little and big endian: - - Within a byte, high bits correspond to an integer high bits, and lowbits
common-trace-format-specification.txt deleted +0
-3- correspond to low bits. -- On little endian: - - Integer across multiple bytes are placed from the less significant tothe
common-trace-format-specification.txt deleted +0
-2- most significant. - - Consecutive integers are placed from lower bits to higher bits (evenwithin
common-trace-format-specification.txt deleted +0
-3- a byte). -- On big endian: - - Integer across multiple bytes are placed from the most significant tothe
common-trace-format-specification.txt deleted +0
-2- less significant. - - Consecutive integers are placed from higher bits to lower bits (evenwithin
common-trace-format-specification.txt deleted +0
-3- a byte). - -This binary representation is derived from the bitfield implementation inGCC
common-trace-format-specification.txt deleted +0
-10-for little and big endian. However, contrary to what GCC does, integers can -cross units boundaries (no padding is required). Padding can be explicitly -added (see 4.1.6 GNU/C bitfields) to follow the GCC layout if needed. - -TSDL meta-data representation: - - integer { - signed = true OR false; /* default false */ - byte_order = native OR network OR be OR le; /* default native */ - size = value; /* value in bits, no default*/
common-trace-format-specification.txt deleted +0
-3- align = value; /* value in bits */ - /* based used for pretty-printing output, default: decimal. */ - base = decimal OR dec OR d OR i OR u OR 10 OR hexadecimal OR hex OR x ORX OR p OR 16
common-trace-format-specification.txt deleted +0
-36- OR octal OR oct OR o OR 8 OR binary OR b OR 2; - /* character encoding, default: none */ - encoding = none or UTF8 or ASCII; - } - -Example of type inheritance (creation of a uint32_t named type): - -typealias integer { - size = 32; - signed = false; - align = 32; -} := uint32_t; - -Definition of a named 5-bit signed bitfield: - -typealias integer { - size = 5; - signed = true; - align = 1; -} := int5_t; - -The character encoding field can be used to specify that the integer -must be printed as a text character when read. e.g.: - -typealias integer { - size = 8; - align = 8; - signed = false; - encoding = UTF8; -} := utf_char; - - -4.1.6 GNU/C bitfields - -The GNU/C bitfields follow closely the integer representation, with a -particularity on alignment: if a bitfield cannot fit in the current unit,the
-unit is padded and the bitfield starts at the following unit. The unit size iscommon-trace-format-specification.txt deleted +0
-15-defined by the size of the type "unit_type". - -TSDL meta-data representation: - - unit_type name:size; - -As an example, the following structure declared in C compiled by GCC: - -struct example { - short a:12; - short b:5; -}; - -The example structure is aligned on the largest element (short). The second -bitfield would be aligned on the next unit boundary, because it would notfit in
common-trace-format-specification.txt deleted +0
-8-the current unit. - -4.1.7 Floating point - -The floating point values byte ordering is defined in the TSDL meta-data. - -Floating point values follow the IEEE 754-2008 standard interchange formats. -Description of the floating point values include the exponent and mantissasize
common-trace-format-specification.txt deleted +0
-4-in bits. Some requirements are imposed on the floating point values: - -- FLT_RADIX must be 2. -- mant_dig is the number of digits represented in the mantissa. It isspecified
common-trace-format-specification.txt deleted +0
-5- by the ISO C99 standard, section 5.2.4, as FLT_MANT_DIG, DBL_MANT_DIG and - LDBL_MANT_DIG as defined by <float.h>. -- exp_dig is the number of digits represented in the exponent. Given that - mant_dig is one bit more than its actual size in bits (leading 1 is not - needed) and also given that the sign bit always takes one bit, exp_dig canbe
common-trace-format-specification.txt deleted +0
-32- specified as: - - - sizeof(float) * CHAR_BIT - FLT_MANT_DIG - - sizeof(double) * CHAR_BIT - DBL_MANT_DIG - - sizeof(long double) * CHAR_BIT - LDBL_MANT_DIG - -TSDL meta-data representation: - -floating_point { - exp_dig = value; - mant_dig = value; - byte_order = native OR network OR be OR le; - align = value; -} - -Example of type inheritance: - -typealias floating_point { - exp_dig = 8; /* sizeof(float) * CHAR_BIT - FLT_MANT_DIG */ - mant_dig = 24; /* FLT_MANT_DIG */ - byte_order = native; - align = 32; -} := float; - -TODO: define NaN, +inf, -inf behavior. - -Bit-packed, byte-packed or larger alignments can be used for floating -point values, similarly to integers. - -4.1.8 Enumerations - -Enumerations are a mapping between an integer type and a table of strings.The
-numerical representation of the enumeration follows the integer type specified
-by the meta-data. The enumeration mapping table is detailed in the enumerationcommon-trace-format-specification.txt deleted +0
-19-description within the meta-data. The mapping table maps inclusive value -ranges (or single values) to strings. Instead of being limited to simple -"value -> string" mappings, these enumerations map -"[ start_value ... end_value ] -> string", which map inclusive ranges of -values to strings. An enumeration from the C language can be represented in -this format by having the same start_value and end_value for each -mapping, which is in fact a range of size 1. This single-value range is -supported without repeating the start and end values with the value = -string declaration. Enumerations need to contain at least one entry. - -enum name : integer_type { - somestring = start_value1 ... end_value1, - "other string" = start_value2 ... end_value2, - yet_another_string, /* will be assigned to end_value2 + 1 */ - "some other string" = value, - ... -}; - -If the values are omitted, the enumeration starts at 0 and increment of 1for
common-trace-format-specification.txt deleted +0
-14-each entry. An entry with omitted value that follows a range entry takes -as value the end_value of the previous range + 1: - -enum name : unsigned int { - ZERO, - ONE, - TWO, - TEN = 10, - ELEVEN, -}; - -Overlapping ranges within a single enumeration are implementation defined. - -A nameless enumeration can be declared as a field type or as part of atypedef:
common-trace-format-specification.txt deleted +0
-25- -enum : integer_type { - ... -} - -Enumerations omitting the container type ": integer_type" use the "int" -type (for compatibility with C99). The "int" type must be previously -declared. E.g.: - -typealias integer { size = 32; align = 32; signed = true; } := int; - -enum { - ... -} - - -4.2 Compound types - -Compound are aggregation of type declarations. Compound types include -structures, variant, arrays, sequences, and strings. - -4.2.1 Structures - -Structures are aligned on the largest alignment required by basic types -contained within the structure. (This follows the ISO/C standard forstructures)
common-trace-format-specification.txt deleted +0
-29- -TSDL meta-data representation of a named structure: - -struct name { - field_type field_name; - field_type field_name; - ... -}; - -Example: - -struct example { - integer { /* Nameless type */ - size = 16; - signed = true; - align = 16; - } first_field_name; - uint64_t second_field_name; /* Named type declared in the meta-data */ -}; - -The fields are placed in a sequence next to each other. They each -possess a field name, which is a unique identifier within the structure. -The identifier is not allowed to use any reserved keyword -(see Section C.1.2). Replacing reserved keywords with -underscore-prefixed field names is recommended. Fields starting with an -underscore should have their leading underscore removed by the CTF trace -readers. - -A nameless structure can be declared as a field type or as part of atypedef:
common-trace-format-specification.txt deleted +0
-82- -struct { - ... -} - -Alignment for a structure compound type can be forced to a minimum value -by adding an "align" specifier after the declaration of a structure -body. This attribute is read as: align(value). The value is specified in -bits. The structure will be aligned on the maximum value between this -attribute and the alignment required by the basic types contained within -the structure. e.g. - -struct { - ... -} align(32) - -4.2.2 Variants (Discriminated/Tagged Unions) - -A CTF variant is a selection between different types. A CTF variant must -always be defined within the scope of a structure or within fields -contained within a structure (defined recursively). A "tag" enumeration -field must appear in either the same static scope, prior to the variant -field (in field declaration order), in an upper static scope, or in an -upper dynamic scope (see Section 7.3.2). The type selection is indicated -by the mapping from the enumeration value to the string used as variant -type selector. The field to use as tag is specified by the "tag_field", -specified between "< >" after the "variant" keyword for unnamed -variants, and after "variant name" for named variants. It is not -required that each enumeration mapping appears as variant type tag -field. It is also not required that each variant type tag appears as -enumeration mapping. However, it is required that any enumeration -mapping encountered within a stream has a matching variant type tag -field. - -The alignment of the variant is the alignment of the type as selected by -the tag value for the specific instance of the variant. The size of the -variant is the size as selected by the tag value for the specific -instance of the variant. - -The alignment of the type containing the variant is independent of the -variant alignment. For instance, if a structure contains two fields, a -32-bit integer, aligned on 32 bits, and a variant, which contains two -choices: either a 32-bit field, aligned on 32 bits, or a 64-bit field, -aligned on 64 bits, the alignment of the outmost structure will be -32-bit (the alignment of its largest field, disregarding the alignment -of the variant). The alignment of the variant will depend on the -selector: if the variant's 32-bit field is selected, its alignment will -be 32-bit, or 64-bit otherwise. It is important to note that variants -are specifically tailored for compactness in a stream. Therefore, the -relative offsets of compound type fields can vary depending on -the offset at which the compound type starts if it contains a variant -that itself contains a type with alignment larger than the largest field -contained within the compound type. This is caused by the fact that the -compound type may contain the enumeration that select the variant's -choice, and therefore the alignment to be applied to the compound type -cannot be determined before encountering the enumeration. - -Each variant type selector possess a field name, which is a unique -identifier within the variant. The identifier is not allowed to use any -reserved keyword (see Section C.1.2). Replacing reserved keywords with -underscore-prefixed field names is recommended. Fields starting with an -underscore should have their leading underscore removed by the CTF trace -readers. - - -A named variant declaration followed by its definition within a structure -declaration: - -variant name { - field_type sel1; - field_type sel2; - field_type sel3; - ... -}; - -struct { - enum : integer_type { sel1, sel2, sel3, ... } tag_field; - ... - variant name <tag_field> v; -} - -An unnamed variant definition within a structure is expressed by thefollowing
common-trace-format-specification.txt deleted +0
-14-TSDL meta-data: - -struct { - enum : integer_type { sel1, sel2, sel3, ... } tag_field; - ... - variant <tag_field> { - field_type sel1; - field_type sel2; - field_type sel3; - ... - } v; -} - -Example of a named variant within a sequence that refers to a single tagfield:
common-trace-format-specification.txt deleted +0
-42- -variant example { - uint32_t a; - uint64_t b; - short c; -}; - -struct { - enum : uint2_t { a, b, c } choice; - unsigned int seqlen; - variant example <choice> v[seqlen]; -} - -Example of an unnamed variant: - -struct { - enum : uint2_t { a, b, c, d } choice; - /* Unrelated fields can be added between the variant and its tag */ - int32_t somevalue; - variant <choice> { - uint32_t a; - uint64_t b; - short c; - struct { - unsigned int field1; - uint64_t field2; - } d; - } s; -} - -Example of an unnamed variant within an array: - -struct { - enum : uint2_t { a, b, c } choice; - variant <choice> { - uint32_t a; - uint64_t b; - short c; - } v[10]; -} - -Example of a variant type definition within a structure, where the definedtype
common-trace-format-specification.txt deleted +0
-3-is then declared within an array of structures. This variant refers to a tag -located in an upper static scope. This example clearly shows that a variant -type definition referring to the tag "x" uses the closest preceding fieldfrom
common-trace-format-specification.txt deleted +0
-142-the static scope of the type definition. - -struct { - enum : uint2_t { a, b, c, d } x; - - typedef variant <x> { /* - * "x" refers to the preceding "x" enumeration in the - * static scope of the type definition. - */ - uint32_t a; - uint64_t b; - short c; - } example_variant; - - struct { - enum : int { x, y, z } x; /* This enumeration is not used by "v". */ - example_variant v; /* - * "v" uses the "enum : uint2_t { a, b, c, d }" - * tag. - */ - } a[10]; -} - -4.2.3 Arrays - -Arrays are fixed-length. Their length is declared in the type -declaration within the meta-data. They contain an array of "inner type" -elements, which can refer to any type not containing the type of the -array being declared (no circular dependency). The length is the number -of elements in an array. - -TSDL meta-data representation of a named array: - -typedef elem_type name[length]; - -A nameless array can be declared as a field type within a structure, e.g.: - - uint8_t field_name[10]; - -Arrays are always aligned on their element alignment requirement. - -4.2.4 Sequences - -Sequences are dynamically-sized arrays. They refer to a "length" -unsigned integer field, which must appear in either the same static scope, -prior to the sequence field (in field declaration order), in an upper -static scope, or in an upper dynamic scope (see Section 7.3.2). This -length field represents the number of elements in the sequence. The -sequence per se is an array of "inner type" elements. - -TSDL meta-data representation for a sequence type definition: - -struct { - unsigned int length_field; - typedef elem_type typename[length_field]; - typename seq_field_name; -} - -A sequence can also be declared as a field type, e.g.: - -struct { - unsigned int length_field; - long seq_field_name[length_field]; -} - -Multiple sequences can refer to the same length field, and these length -fields can be in a different upper dynamic scope: - -e.g., assuming the stream.event.header defines: - -stream { - ... - id = 1; - event.header := struct { - uint16_t seq_len; - }; -}; - -event { - ... - stream_id = 1; - fields := struct { - long seq_a[stream.event.header.seq_len]; - char seq_b[stream.event.header.seq_len]; - }; -}; - -The sequence elements follow the "array" specifications. - -4.2.5 Strings - -Strings are an array of bytes of variable size and are terminated by a '\0' -"NULL" character. Their encoding is described in the TSDL meta-data. In -absence of encoding attribute information, the default encoding is -UTF-8. - -TSDL meta-data representation of a named string type: - -typealias string { - encoding = UTF8 OR ASCII; -} := name; - -A nameless string type can be declared as a field type: - -string field_name; /* Use default UTF8 encoding */ - -Strings are always aligned on byte size. - -5. Event Packet Header - -The event packet header consists of two parts: the "event packet header" -is the same for all streams of a trace. The second part, the "event -packet context", is described on a per-stream basis. Both are described -in the TSDL meta-data. - -Event packet header (all fields are optional, specified by TSDL meta-data): - -- Magic number (CTF magic number: 0xC1FC1FC1) specifies that this is a - CTF packet. This magic number is optional, but when present, it should - come at the very beginning of the packet. -- Trace UUID, used to ensure the event packet match the meta-data used. - (note: we cannot use a meta-data checksum in every cases instead of a - UUID because meta-data can be appended to while tracing is active) - This field is optional. -- Stream ID, used as reference to stream description in meta-data. - This field is optional if there is only one stream description in the - meta-data, but becomes required if there are more than one stream in - the TSDL meta-data description. - -Event packet context (all fields are optional, specified by TSDL meta-data): - -- Event packet content size (in bits). -- Event packet size (in bits, includes padding). -- Event packet content checksum. Checksum excludes the event packet - header. -- Per-stream event packet sequence count (to deal with UDP packet loss). The - number of significant sequence counter bits should also be present, so - wrap-arounds are dealt with correctly. -- Time-stamp at the beginning and time-stamp at the end of the event packet. - Both timestamps are written in the packet header, but sampled respectively - while (or before) writing the first event and while (or after) writing the - last event in the packet. The inclusive range between these timestampsshould
- include all event timestamps assigned to events contained within the packet.common-trace-format-specification.txt deleted +0
-11- The timestamp at the beginning of an event packet is guaranteed to be - below or equal the timestamp at the end of that event packet. - The timestamp at the end of an event packet is guaranteed to be below - or equal the timestamps at the end of any following packet within the - same stream. See Section 8. Clocks for more detail. -- Events discarded count - - Snapshot of a per-stream free-running counter, counting the number of - events discarded that were supposed to be written in the stream after - the last event in the event packet. - * Note: producer-consumer buffer full condition can fill the current - event packet with padding so we know exactly where events havebeen
common-trace-format-specification.txt deleted +0
-16- discarded. However, if the buffer full condition chooses not - to fill the current event packet with padding, all we know - about the timestamp range in which the events have been - discarded is that it is somewhere between the beginning and - the end of the packet. -- Lossless compression scheme used for the event packet content. Applied - directly to raw data. New types of compression can be added in following - versions of the format. - 0: no compression scheme - 1: bzip2 - 2: gzip - 3: xz -- Cypher used for the event packet content. Applied after compression. - 0: no encryption - 1: AES -- Checksum scheme used for the event packet content. Applied afterencryption.
common-trace-format-specification.txt deleted +0
-36- 0: no checksum - 1: md5 - 2: sha1 - 3: crc32 - -5.1 Event Packet Header Description - -The event packet header layout is indicated by the trace packet.header -field. Here is a recommended structure type for the packet header with -the fields typically expected (although these fields are each optional): - -struct event_packet_header { - uint32_t magic; - uint8_t uuid[16]; - uint32_t stream_id; -}; - -trace { - ... - packet.header := struct event_packet_header; -}; - -If the magic number is not present, tools such as "file" will have no -mean to discover the file type. - -If the uuid is not present, no validation that the meta-data actually -corresponds to the stream is performed. - -If the stream_id packet header field is missing, the trace can only -contain a single stream. Its "id" field can be left out, and its events -don't need to declare a "stream_id" field. - - -5.2 Event Packet Context Description - -Event packet context example. These are declared within the streamdeclaration
common-trace-format-specification.txt deleted +0
-45-in the meta-data. All these fields are optional. If the packet size field is -missing, the whole stream only contains a single packet. If the content -size field is missing, the packet is filled (no padding). The content -and packet sizes include all headers. - -An example event packet context type: - -struct event_packet_context { - uint64_t timestamp_begin; - uint64_t timestamp_end; - uint32_t checksum; - uint32_t stream_packet_count; - uint32_t events_discarded; - uint32_t cpu_id; - uint64_t/uint32_t/uint16_t content_size; - uint64_t/uint32_t/uint16_t packet_size; - uint8_t compression_scheme; - uint8_t encryption_scheme; - uint8_t checksum_scheme; -}; - - -6. Event Structure - -The overall structure of an event is: - -1 - Event Header (as specified by the stream meta-data) - 2 - Stream Event Context (as specified by the stream meta-data) - 3 - Event Context (as specified by the event meta-data) - 4 - Event Payload (as specified by the event meta-data) - -This structure defines an implicit dynamic scoping, where variants -located in inner structures (those with a higher number in the listing -above) can refer to the fields of outer structures (with lower number in -the listing above). See Section 7.3 TSDL Scopes for more detail. - -The total length of an event is defined as the difference between the -end of its Event Payload and the end of the previous event's Event -Payload. Therefore, it includes the event header alignment padding, and -all its fields and their respective alignment padding. Events of length -0 are forbidden. - -6.1 Event Header - -Event headers can be described within the meta-data. We hereby propose, asan
-example, two types of events headers. Type 1 accommodates streams with less than
-31 event IDs. Type 2 accommodates streams with 31 or more event IDs.common-trace-format-specification.txt deleted +0
-2- -One major factor can vary between streams: the number of event IDs assignedto
-a stream. Luckily, this information tends to stay relatively constant (modulo
-event registration while trace is being recorded), so we can specify differentcommon-trace-format-specification.txt deleted +0
-6-representations for streams containing few event IDs and streams containing -many event IDs, so we end up representing the event ID and time-stamp as -densely as possible in each case. - -The header is extended in the rare occasions where the information cannot be -represented in the ranges available in the standard event header. They arealso
common-trace-format-specification.txt deleted +0
-2-used in the rare occasions where the data required for a field could not be -collected: the flag corresponding to the missing field within the
missing_fieldscommon-trace-format-specification.txt deleted +0
-16-array is then set to 1. - -Types uintX_t represent an X-bit unsigned integer, as declared with -either: - - typealias integer { size = X; align = X; signed = false; } := uintX_t; - - or - - typealias integer { size = X; align = 1; signed = false; } := uintX_t; - -For more information about timestamp fields, see Section 8. Clocks. - -6.1.1 Type 1 - Few event IDs - - - Aligned on 32-bit (or 8-bit if byte-packed, depending on thearchitecture
common-trace-format-specification.txt deleted +0
-28- preference). - - Native architecture byte ordering. - - For "compact" selection - - Fixed size: 32 bits. - - For "extended" selection - - Size depends on the architecture and variant alignment. - -struct event_header_1 { - /* - * id: range: 0 - 30. - * id 31 is reserved to indicate an extended header. - */ - enum : uint5_t { compact = 0 ... 30, extended = 31 } id; - variant <id> { - struct { - uint27_t timestamp; - } compact; - struct { - uint32_t id; /* 32-bit event IDs */ - uint64_t timestamp; /* 64-bit timestamps */ - } extended; - } v; -} align(32); /* or align(8) */ - - -6.1.2 Type 2 - Many event IDs - - - Aligned on 16-bit (or 8-bit if byte-packed, depending on thearchitecture
common-trace-format-specification.txt deleted +0
-35- preference). - - Native architecture byte ordering. - - For "compact" selection - - Size depends on the architecture and variant alignment. - - For "extended" selection - - Size depends on the architecture and variant alignment. - -struct event_header_2 { - /* - * id: range: 0 - 65534. - * id 65535 is reserved to indicate an extended header. - */ - enum : uint16_t { compact = 0 ... 65534, extended = 65535 } id; - variant <id> { - struct { - uint32_t timestamp; - } compact; - struct { - uint32_t id; /* 32-bit event IDs */ - uint64_t timestamp; /* 64-bit timestamps */ - } extended; - } v; -} align(16); /* or align(8) */ - - -6.2 Stream Event Context and Event Context - -The event context contains information relative to the current event. -The choice and meaning of this information is specified by the TSDL -stream and event meta-data descriptions. The stream context is applied -to all events within the stream. The stream context structure follows -the event header. The event context is applied to specific events. Its -structure follows the stream context structure. - -An example of stream-level event context is to save the event payload sizewith
common-trace-format-specification.txt deleted +0
-22-each event, or to save the current PID with each event. These are declared -within the stream declaration within the meta-data: - - stream { - ... - event.context := struct { - uint pid; - uint16_t payload_size; - }; - }; - -An example of event-specific event context is to declare a bitmap of missing -fields, only appended after the stream event context if the extended event -header is selected. NR_FIELDS is the number of fields within the event (a -numeric value). - - event { - context := struct { - variant <id> { - struct { } compact; - struct { - uint1_t missing_fields[NR_FIELDS]; /* missing event fields bitmap*/
common-trace-format-specification.txt deleted +0
-15- } extended; - } v; - }; - ... - } - -6.3 Event Payload - -An event payload contains fields specific to a given event type. The fields -belonging to an event type are described in the event-specific meta-data -within a structure type. - -6.3.1 Padding - -No padding at the end of the event payload. This differs from the ISO/Cstandard
-for structures, but follows the CTF standard for structures. In a trace, even
-though it makes sense to align the beginning of a structure, it really makes no
-sense to add padding at the end of the structure, because structures are usuallycommon-trace-format-specification.txt deleted +0
-3-not followed by a structure of the same type. - -This trick can be done by adding a zero-length "end" field at the end of theC
common-trace-format-specification.txt deleted +0
-7-structures, and by using the offset of this field rather than using sizeof() -when calculating the size of a structure (see Appendix "A. Helper macros"). - -6.3.2 Alignment - -The event payload is aligned on the largest alignment required by types -contained within the payload. (This follows the ISO/C standard forstructures)
common-trace-format-specification.txt deleted +0
-302- - -7. Trace Stream Description Language (TSDL) - -The Trace Stream Description Language (TSDL) allows expression of the -binary trace streams layout in a C99-like Domain Specific Language -(DSL). - - -7.1 Meta-data - -The trace stream layout description is located in the trace meta-data. -The meta-data is itself located in a stream identified by its name: -"metadata". - -The meta-data description can be expressed in two different formats: -text-only and packet-based. The text-only description facilitates -generation of meta-data and provides a convenient way to enter theLines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
typealias.
- */common-trace-format-specification.txt deleted +0
-134Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
field
common-trace-format-specification.txt deleted +0
-2Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
be
common-trace-format-specification.txt deleted +0
-9Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
These
-subdivisions allow the trace analyzer to perform a fast binary search by time
-within the stream (typically requiring to index only the event packet headers)common-trace-format-specification.txt deleted +0
-2Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
filled
common-trace-format-specification.txt deleted +0
-13Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
packet
common-trace-format-specification.txt deleted +0
-9Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Grammar.
- *
- * Inspired from the C99 grammar:
- * http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf (Annex A)
- * and c++1x grammar (draft)
- * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3291.pdf (Annex
A)
- *
- * Specialized for CTF needs by including only constant and declarations from
- * C99 (excluding function declarations), and by adding support for variants,
- * sequences and CTF-specific specifiers. Enumeration container types
- * semantic is inspired from c++1x enum-base.
- */common-trace-format-specification.txt deleted +0
-255Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
align-attribute-opt
common-trace-format-specification.txt deleted +0
-9Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
declaration-specifiers-opt declarator-list ;
common-trace-format-specification.txt deleted +0
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
type-assignment-operator declaration-specifiers abstract-declarator-list ;
common-trace-format-specification.txt deleted +0
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
type-assignment-operator declarator-list ;
common-trace-format-specification.txt deleted +0
-15Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
}
common-trace-format-specification.txt deleted +0
-62Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
type-assignment-operator declaration-specifiers abstract-declarator-list
common-trace-format-specification.txt deleted +0
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
type-assignment-operator declarator-list
common-trace-format-specification.txt deleted +0
-15Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
declaration-specifiers-opt declarator-list
common-trace-format-specification.txt deleted +0
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
type-assignment-operator declaration-specifiers abstract-declarator-list
common-trace-format-specification.txt deleted +0
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
type-assignment-operator declarator-list
--
2.2.1_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Part of the diff is not shown here. View the raw message.