Skip to contentSkip to conversation

Fix: allocate ts_end in ringbuffer shared memory

PATCH Applied Sebastien Boisvert 3 participants 9 messages May 10, 2019 6 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

On 2019-05-10 11:26 a.m., Gabriel-Andrew Pollo-Guilbert wrote:

Hello Gabriel-Andrew,

Quoted 10 lines

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer

I need to killall existing lttng-sessiond daemon, right ?

Quoted 1 line

2. start a traced application

I used lttng-ust/doc/examples/demo/.

Quoted 1 line

3. lttng create "test-sesssion"

                                ^^^

There are too many s in session.

Quoted 8 lines

4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

When I build lttng-tools b14f53d4

I get this error:

make[3]: Entering directory '/home/sboisvert/open-source/lttng.org/Source/lttng-tools/src/common/ust-consumer'
CC ust-consumer.lo
ust-consumer.c: In function 'lttng_ustconsumer_recv_cmd':
ust-consumer.c:1459:7: error: 'struct ustctl_consumer_channel_attr' has no member named 'blocking_timeout'

   attr.blocking_timeout= msg.u.ask_channel.blocking_timeout;
       ^
ust-consumer.c: In function 'lttng_ustconsumer_sample_snapshot_positions':
ust-consumer.c:2225:9: warning: implicit declaration of function 'ustctl_snapshot_sample_positions'; did you mean 'ustctl_snapshot_get_produced'? [-Wimplicit-function-declaration]
  return ustctl_snapshot_sample_positions(stream->ustream);
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         ustctl_snapshot_get_produced

Makefile:548: recipe for target 'ust-consumer.lo' failed

There seems to be 2 declarations of struct ustctl_consumer_channel_attr.

In file lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h, struct ustctl_consumer_channel_attr has the attribute blocking_timeout.

In file lttng-ust/include/lttng/ust-ctl.h, struct ustctl_consumer_channel_attr does not have the attribute.

They are exactly the same, except the missing blocking_timeout attribute.

[sboisvert@GT480:Source]$ grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ diff -u <(grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h) <(grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h)
--- /dev/fd/63	2019-05-10 12:32:23.562246820 -0400
+++ /dev/fd/62	2019-05-10 12:32:23.562246820 -0400
@@ -8,5 +8,4 @@
 	enum lttng_ust_output output;		/* splice, mmap */
 	uint32_t chan_id;			/* channel ID */
 	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
-	int64_t blocking_timeout;			/* Blocking timeout (usec) */
 } LTTNG_PACKED;

My configure commands (followed by "make install"):

[sboisvert@GT480:userspace-rcu]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0
[sboisvert@GT480:lttng-ust]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib
[sboisvert@GT480:lttng-tools]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-tools/b14f53d4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/lib CPPFLAGS=-I/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/include

Thanks

Quoted 21 lines
---
 libringbuffer/ring_buffer_backend.c | 3 +++
 1 file changed, 3 insertions(+)
libringbuffer/ring_buffer_backend.c +3 -0
index a0ef7446..431b8eae 100644
@@ -341,6 +341,9 @@ int channel_backend_init(struct channel_backend *chanb,
 	shmsize += sizeof(struct commit_counters_hot) * num_subbuf;
 	shmsize += offset_align(shmsize, __alignof__(struct commit_counters_cold));
 	shmsize += sizeof(struct commit_counters_cold) * num_subbuf;
+	/* Sampled timestamp end */
+	shmsize += offset_align(shmsize, __alignof__(uint64_t));
+	shmsize += sizeof(uint64_t) * num_subbuf;
 
 	if (config->alloc == RING_BUFFER_ALLOC_PER_CPU) {
 		struct lttng_ust_lib_ring_buffer *buf;
#2 lore raw
In reply to a message not in the archive

Merged into master, 2.11, thanks!

Mathieu

Quoted 45 lines · ----- On May 10, 2019, at 11:26 AM, Gabriel-Andrew Pollo-Guilbert…

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer
2. start a traced application
3. lttng create "test-sesssion"
4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

Signed-off-by: Gabriel-Andrew Pollo-Guilbert
<[email protected]>
---
libringbuffer/ring_buffer_backend.c | 3 +++

1 file changed, 3 insertions(+)
libringbuffer/ring_buffer_backend.c +3 -1
index a0ef7446..431b8eae 100644
@@ -341,6 +341,9 @@ int channel_backend_init(struct channel_backend *chanb,
 	shmsize += sizeof(struct commit_counters_hot) * num_subbuf;
 	shmsize += offset_align(shmsize, __alignof__(struct commit_counters_cold));
 	shmsize += sizeof(struct commit_counters_cold) * num_subbuf;
+	/* Sampled timestamp end */
+	shmsize += offset_align(shmsize, __alignof__(uint64_t));
+	shmsize += sizeof(uint64_t) * num_subbuf;
 
 	if (config->alloc == RING_BUFFER_ALLOC_PER_CPU) {
 		struct lttng_ust_lib_ring_buffer *buf;
--

2.21.0

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 17 lines · ----- On May 10, 2019, at 12:33 PM, Sebastien Boisvert [email protected]…

On 2019-05-10 11:26 a.m., Gabriel-Andrew Pollo-Guilbert wrote:

Hello Gabriel-Andrew,

Quoted 10 lines

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer

I need to killall existing lttng-sessiond daemon, right ?

Quoted 7 lines
Quoted 1 line

2. start a traced application

I used lttng-ust/doc/examples/demo/.

Quoted 1 line

3. lttng create "test-sesssion"

                               ^^^

There are too many s in session.

oops, already merged.

Quoted 89 lines
Quoted 11 lines

4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

When I build lttng-tools b14f53d4

I get this error:

make[3]: Entering directory '/home/sboisvert/open-source/lttng.org/Source/lttng-tools/src/common/ust-consumer'
CC ust-consumer.lo
ust-consumer.c: In function 'lttng_ustconsumer_recv_cmd':
ust-consumer.c:1459:7: error: 'struct ustctl_consumer_channel_attr' has no member named 'blocking_timeout'

  attr.blocking_timeout= msg.u.ask_channel.blocking_timeout;
      ^

ust-consumer.c: In function 'lttng_ustconsumer_sample_snapshot_positions':
ust-consumer.c:2225:9: warning: implicit declaration of function 'ustctl_snapshot_sample_positions'; did you mean 'ustctl_snapshot_get_produced'? [-Wimplicit-function-declaration]

 return ustctl_snapshot_sample_positions(stream->ustream);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        ustctl_snapshot_get_produced

Makefile:548: recipe for target 'ust-consumer.lo' failed

There seems to be 2 declarations of struct ustctl_consumer_channel_attr.

In file lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h, struct ustctl_consumer_channel_attr has the attribute blocking_timeout.

In file lttng-ust/include/lttng/ust-ctl.h, struct ustctl_consumer_channel_attr does not have the attribute.

They are exactly the same, except the missing blocking_timeout attribute.

[sboisvert@GT480:Source]$ grep -A 10 "struct ustctl_consumer_channel_attr {"
lttng-ust/include/lttng/ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ grep -A 11 "struct ustctl_consumer_channel_attr {"
lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ diff -u <(grep -A 11 "struct
ustctl_consumer_channel_attr {"
lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h) <(grep -A 10 "struct
ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h)
dev/fd/62 renamed from dev/fd/63 +0 -1
@@ -8,5 +8,4 @@
 	enum lttng_ust_output output;		/* splice, mmap */
 	uint32_t chan_id;			/* channel ID */
 	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
-	int64_t blocking_timeout;			/* Blocking timeout (usec) */
 } LTTNG_PACKED;
Quoted 8 lines

My configure commands (followed by "make install"):

[sboisvert@GT480:userspace-rcu]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0
[sboisvert@GT480:lttng-ust]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4
LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib

I think you need a lttng-ust which is more recent than 2.9.4.

Thanks,

Mathieu

Quoted 36 lines
[sboisvert@GT480:lttng-tools]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-tools/b14f53d4
LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/lib
CPPFLAGS=-I/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/include

Thanks

Quoted 14 lines

Signed-off-by: Gabriel-Andrew Pollo-Guilbert
<[email protected]>

---
 libringbuffer/ring_buffer_backend.c | 3 +++
 1 file changed, 3 insertions(+)
libringbuffer/ring_buffer_backend.c +0 -0
index a0ef7446..431b8eae 100644

libringbuffer/ring_buffer_backend.c int channel_backend_init(struct channel_backend *chanb, · line 349

Quoted 9 lines
libringbuffer/ring_buffer_backend.c +3 -0
@@ -341,6 +341,9 @@ int channel_backend_init(struct channel_backend *chanb,
 	shmsize += sizeof(struct commit_counters_hot) * num_subbuf;
 	shmsize += offset_align(shmsize, __alignof__(struct commit_counters_cold));
 	shmsize += sizeof(struct commit_counters_cold) * num_subbuf;
+	/* Sampled timestamp end */
+	shmsize += offset_align(shmsize, __alignof__(uint64_t));
+	shmsize += sizeof(uint64_t) * num_subbuf;
 
 	if (config->alloc == RING_BUFFER_ALLOC_PER_CPU) {
 		struct lttng_ust_lib_ring_buffer *buf;

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#5 lore raw
In reply to a message not in the archive

Hi,

Quoted 9 lines · Sebastien Boisvert

On 2019-05-10 11:26 a.m., Gabriel-Andrew Pollo-Guilbert wrote:

Hello Gabriel-Andrew,

Quoted 5 lines

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

Quoted 7 lines
Quoted 5 lines

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer

I need to killall existing lttng-sessiond daemon, right ?

No, since we are in userspace this can be done with a root lttng-sessiond present. But in real life -> yes.

Quoted 4 lines
Quoted 2 lines

2. start a traced application

I used lttng-ust/doc/examples/demo/.

Make sure that the app is long lived (easy-ust with a while loop for example). This is only to force the allocation on lttng start.

Quoted 4 lines
Quoted 2 lines

3. lttng create "test-sesssion"

                                ^^^

There are too many s in session.

Does not matter since it is the session name here and subsequent call are done in the context on that session. Still a good catch.

Quoted 19 lines
Quoted 10 lines

4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

When I build lttng-tools b14f53d4

I get this error:

make[3]: Entering directory '/home/sboisvert/open-source/lttng.org/Source/lttng-tools/src/common/ust-consumer'
CC ust-consumer.lo
ust-consumer.c: In function 'lttng_ustconsumer_recv_cmd':
ust-consumer.c:1459:7: error: 'struct ustctl_consumer_channel_attr' has no member named 'blocking_timeout'

Quoted 8 lines
   attr.blocking_timeout= msg.u.ask_channel.blocking_timeout;
       ^
ust-consumer.c: In function 'lttng_ustconsumer_sample_snapshot_positions':
ust-consumer.c:2225:9: warning: implicit declaration of function 'ustctl_snapshot_sample_positions'; did you mean 'ustctl_snapshot_get_produced'? [-Wimplicit-function-declaration]
  return ustctl_snapshot_sample_positions(stream->ustream);
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         ustctl_snapshot_get_produced

Makefile:548: recipe for target 'ust-consumer.lo' failed

Are you sure you are building lttng-tools against lttng-ust 6c737d0594cac0d969e1948ea1ed55c15be9cec8 or master?

In any case, this does not seems related to this particular issue. Please start a new email thread.

Quoted 9 lines

There seems to be 2 declarations of struct ustctl_consumer_channel_attr.

In file lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h, struct ustctl_consumer_channel_attr has the attribute blocking_timeout.

In file lttng-ust/include/lttng/ust-ctl.h, struct ustctl_consumer_channel_attr does not have the attribute.

Currently on 6c737d0594cac0d969e1948ea1ed55c15be9cec8 (almost master):

struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;

This field was introduced in b2c5f61a9. Suggesting that you lttng-ust tree is out of date.

Quoted 43 lines

They are exactly the same, except the missing blocking_timeout attribute.

[sboisvert@GT480:Source]$ grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ diff -u <(grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h) <(grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h)
dev/fd/62 renamed from dev/fd/63 +0 -1
@@ -8,5 +8,4 @@
 	enum lttng_ust_output output;		/* splice, mmap */
 	uint32_t chan_id;			/* channel ID */
 	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
-	int64_t blocking_timeout;			/* Blocking timeout (usec) */
 } LTTNG_PACKED;
Quoted 8 lines

My configure commands (followed by "make install"):

[sboisvert@GT480:userspace-rcu]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0
[sboisvert@GT480:lttng-ust]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib
[sboisvert@GT480:lttng-tools]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-tools/b14f53d4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/lib CPPFLAGS=-I/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/include

This is probably an artefact but all your path have 2.9 in them. This work is done against master of lttng-tools and lttng-ust.

Our CI does not indicate any failure for the master branches. [1]

[1]

Cheers

A message that is not in the archive (sent off-list or not yet received): 20190510180851.GG15644@joraj-alpa

#7 lore raw
Quoted 156 lines · Jonathan Rajotte-Julien

Hi,

Quoted 9 lines · Sebastien Boisvert

On 2019-05-10 11:26 a.m., Gabriel-Andrew Pollo-Guilbert wrote:

Hello Gabriel-Andrew,

Quoted 5 lines

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

Quoted 7 lines
Quoted 5 lines

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer

I need to killall existing lttng-sessiond daemon, right ?

No, since we are in userspace this can be done with a root lttng-sessiond present. But in real life -> yes.

Quoted 4 lines
Quoted 2 lines

2. start a traced application

I used lttng-ust/doc/examples/demo/.

Make sure that the app is long lived (easy-ust with a while loop for example). This is only to force the allocation on lttng start.

Quoted 4 lines
Quoted 2 lines

3. lttng create "test-sesssion"

                                ^^^

There are too many s in session.

Does not matter since it is the session name here and subsequent call are done in the context on that session. Still a good catch.

Quoted 19 lines
Quoted 10 lines

4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

When I build lttng-tools b14f53d4

I get this error:

make[3]: Entering directory '/home/sboisvert/open-source/lttng.org/Source/lttng-tools/src/common/ust-consumer'
CC ust-consumer.lo
ust-consumer.c: In function 'lttng_ustconsumer_recv_cmd':
ust-consumer.c:1459:7: error: 'struct ustctl_consumer_channel_attr' has no member named 'blocking_timeout'

Quoted 8 lines
   attr.blocking_timeout= msg.u.ask_channel.blocking_timeout;
       ^
ust-consumer.c: In function 'lttng_ustconsumer_sample_snapshot_positions':
ust-consumer.c:2225:9: warning: implicit declaration of function 'ustctl_snapshot_sample_positions'; did you mean 'ustctl_snapshot_get_produced'? [-Wimplicit-function-declaration]
  return ustctl_snapshot_sample_positions(stream->ustream);
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         ustctl_snapshot_get_produced

Makefile:548: recipe for target 'ust-consumer.lo' failed

Are you sure you are building lttng-tools against lttng-ust 6c737d0594cac0d969e1948ea1ed55c15be9cec8 or master?

In any case, this does not seems related to this particular issue. Please start a new email thread.

Quoted 9 lines

There seems to be 2 declarations of struct ustctl_consumer_channel_attr.

In file lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h, struct ustctl_consumer_channel_attr has the attribute blocking_timeout.

In file lttng-ust/include/lttng/ust-ctl.h, struct ustctl_consumer_channel_attr does not have the attribute.

Currently on 6c737d0594cac0d969e1948ea1ed55c15be9cec8 (almost master):

struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;

This field was introduced in b2c5f61a9. Suggesting that you lttng-ust tree is out of date.

Quoted 43 lines

They are exactly the same, except the missing blocking_timeout attribute.

[sboisvert@GT480:Source]$ grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ diff -u <(grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h) <(grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h)
dev/fd/62 renamed from dev/fd/63 +0 -1
@@ -8,5 +8,4 @@
 	enum lttng_ust_output output;		/* splice, mmap */
 	uint32_t chan_id;			/* channel ID */
 	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
-	int64_t blocking_timeout;			/* Blocking timeout (usec) */
 } LTTNG_PACKED;
Quoted 8 lines

My configure commands (followed by "make install"):

[sboisvert@GT480:userspace-rcu]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0
[sboisvert@GT480:lttng-ust]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib
[sboisvert@GT480:lttng-tools]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-tools/b14f53d4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/lib CPPFLAGS=-I/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/include

This is probably an artefact but all your path have 2.9 in them. This work is done against master of lttng-tools and lttng-ust.

Quoted 3 lines

Our CI does not indicate any failure for the master branches. [1]

[1]

A bit quick on the send email on that one.

Our ci currently indicate failure for the master branch but nothing during the build process (at make check).

[1] https://ci.lttng.org/job/lttng-tools_master_build/

Quoted 6 lines

Cheers
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 2 lines
Jonathan Rajotte-Julien
EfficiOS

A message that is not in the archive (sent off-list or not yet received): 20190510181626.GA24050@joraj-alpa

#9 lore raw
Quoted 4 lines
Quoted 4 lines

This is probably an artefact but all your path have 2.9 in them. This work is done against master of lttng-tools and lttng-ust.

Quoted 8 lines
Quoted 3 lines

Our CI does not indicate any failure for the master branches. [1]

[1]

A bit quick on the send email on that one.

Our ci currently indicate failure for the master branch but nothing during the build process (at make check).

Again not true, failures are related to urcu.

I'll go get some coffee clearly need it.

Cheers

#10 lore raw
In reply to a message not in the archive
Quoted 32 lines · Jonathan Rajotte-Julien

Hi,

Quoted 9 lines · Sebastien Boisvert

On 2019-05-10 11:26 a.m., Gabriel-Andrew Pollo-Guilbert wrote:

Hello Gabriel-Andrew,

Quoted 5 lines

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

Quoted 7 lines
Quoted 5 lines

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer

I need to killall existing lttng-sessiond daemon, right ?

No, since we are in userspace this can be done with a root lttng-sessiond present. But in real life -> yes.

Quoted 4 lines
Quoted 2 lines

2. start a traced application

I used lttng-ust/doc/examples/demo/.

Make sure that the app is long lived (easy-ust with a while loop for example). This is only to force the allocation on lttng start.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

3. lttng create "test-sesssion"

                                ^^^

There are too many s in session.

Does not matter since it is the session name here and subsequent call are done in the context on that session. Still a good catch.

Well, I thought that this typo was going to make it into the "git log"..

Quoted 29 lines
Quoted 20 lines
Quoted 11 lines

4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

When I build lttng-tools b14f53d4

I get this error:

make[3]: Entering directory '/home/sboisvert/open-source/lttng.org/Source/lttng-tools/src/common/ust-consumer'
CC ust-consumer.lo
ust-consumer.c: In function 'lttng_ustconsumer_recv_cmd':
ust-consumer.c:1459:7: error: 'struct ustctl_consumer_channel_attr' has no member named 'blocking_timeout'

Quoted 8 lines
   attr.blocking_timeout= msg.u.ask_channel.blocking_timeout;
       ^
ust-consumer.c: In function 'lttng_ustconsumer_sample_snapshot_positions':
ust-consumer.c:2225:9: warning: implicit declaration of function 'ustctl_snapshot_sample_positions'; did you mean 'ustctl_snapshot_get_produced'? [-Wimplicit-function-declaration]
  return ustctl_snapshot_sample_positions(stream->ustream);
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         ustctl_snapshot_get_produced

Makefile:548: recipe for target 'ust-consumer.lo' failed

Quoted 2 lines

Are you sure you are building lttng-tools against lttng-ust 6c737d0594cac0d969e1948ea1ed55c15be9cec8 or master?

My email said that I was using lttng-ust, git tag v2.9.4.

Quoted 4 lines

In any case, this does not seems related to this particular issue. Please start a new email thread.

I wanted to test the change "Fix: allocate ts_end in ringbuffer shared memory" on my computer.

Building userspace-rcu, lttng-ust, and lttng-tools was a prerequisite.

But, fair enough, I should have started a new thread.

Quoted 89 lines
Quoted 9 lines

There seems to be 2 declarations of struct ustctl_consumer_channel_attr.

In file lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h, struct ustctl_consumer_channel_attr has the attribute blocking_timeout.

In file lttng-ust/include/lttng/ust-ctl.h, struct ustctl_consumer_channel_attr does not have the attribute.

Currently on 6c737d0594cac0d969e1948ea1ed55c15be9cec8 (almost master):

struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;

This field was introduced in b2c5f61a9. Suggesting that you lttng-ust tree is out of date.

Quoted 43 lines

They are exactly the same, except the missing blocking_timeout attribute.

[sboisvert@GT480:Source]$ grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ diff -u <(grep -A 11 "struct ustctl_consumer_channel_attr {" lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h) <(grep -A 10 "struct ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h)
dev/fd/62 renamed from dev/fd/63 +0 -1
@@ -8,5 +8,4 @@
 	enum lttng_ust_output output;		/* splice, mmap */
 	uint32_t chan_id;			/* channel ID */
 	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
-	int64_t blocking_timeout;			/* Blocking timeout (usec) */
 } LTTNG_PACKED;
Quoted 8 lines

My configure commands (followed by "make install"):

[sboisvert@GT480:userspace-rcu]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0
[sboisvert@GT480:lttng-ust]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib
[sboisvert@GT480:lttng-tools]$ ./configure --prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-tools/b14f53d4 LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/lib CPPFLAGS=-I/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/include

This is probably an artefact but all your path have 2.9 in them. This work is done against master of lttng-tools and lttng-ust.

Our CI does not indicate any failure for the master branches. [1]

[1]

Cheers

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 120 lines · Mathieu Desnoyers
Quoted 17 lines · ----- On May 10, 2019, at 12:33 PM, Sebastien Boisvert [email protected]…

On 2019-05-10 11:26 a.m., Gabriel-Andrew Pollo-Guilbert wrote:

Hello Gabriel-Andrew,

Quoted 10 lines

This patch allocates the memory used by the ts_end field added by commit 6c737d05. When allocating lots of subbuffer for a channel (512 or more), zalloc_shm() will fail to allocate all the objects because the allocated memory map didn't take account the newly added field.

lttng-tools version: b14f53d4 (2.12.0-pre)

Steps to reproduce the bug:

1. lttng-sessiond -vvv --verbose-consumer

I need to killall existing lttng-sessiond daemon, right ?

Quoted 7 lines
Quoted 1 line

2. start a traced application

I used lttng-ust/doc/examples/demo/.

Quoted 1 line

3. lttng create "test-sesssion"

                               ^^^

There are too many s in session.

oops, already merged.

Quoted 91 lines
Quoted 11 lines

4. lttng enable-channel --userspace --num-subbuf 512 --subbuf-size 8k --overwrite channel
5. lttng enable-event -u -a -c channel
6. lttng start

After these steps, the following error message show should be thrown:

Error: ask_channel_creation consumer command failed
Error: Error creating UST channel "channel" on the consumer daemon

When I build lttng-tools b14f53d4

I get this error:

make[3]: Entering directory '/home/sboisvert/open-source/lttng.org/Source/lttng-tools/src/common/ust-consumer'
CC ust-consumer.lo
ust-consumer.c: In function 'lttng_ustconsumer_recv_cmd':
ust-consumer.c:1459:7: error: 'struct ustctl_consumer_channel_attr' has no member named 'blocking_timeout'

  attr.blocking_timeout= msg.u.ask_channel.blocking_timeout;
      ^

ust-consumer.c: In function 'lttng_ustconsumer_sample_snapshot_positions':
ust-consumer.c:2225:9: warning: implicit declaration of function 'ustctl_snapshot_sample_positions'; did you mean 'ustctl_snapshot_get_produced'? [-Wimplicit-function-declaration]

 return ustctl_snapshot_sample_positions(stream->ustream);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        ustctl_snapshot_get_produced

Makefile:548: recipe for target 'ust-consumer.lo' failed

There seems to be 2 declarations of struct ustctl_consumer_channel_attr.

In file lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h, struct ustctl_consumer_channel_attr has the attribute blocking_timeout.

In file lttng-ust/include/lttng/ust-ctl.h, struct ustctl_consumer_channel_attr does not have the attribute.

They are exactly the same, except the missing blocking_timeout attribute.

[sboisvert@GT480:Source]$ grep -A 10 "struct ustctl_consumer_channel_attr {"
lttng-ust/include/lttng/ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ grep -A 11 "struct ustctl_consumer_channel_attr {"
lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h
struct ustctl_consumer_channel_attr {
	enum lttng_ust_chan_type type;
	uint64_t subbuf_size;			/* bytes */
	uint64_t num_subbuf;			/* power of 2 */
	int overwrite;				/* 1: overwrite, 0: discard */
	unsigned int switch_timer_interval;	/* usec */
	unsigned int read_timer_interval;	/* usec */
	enum lttng_ust_output output;		/* splice, mmap */
	uint32_t chan_id;			/* channel ID */
	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
	int64_t blocking_timeout;			/* Blocking timeout (usec) */
} LTTNG_PACKED;
[sboisvert@GT480:Source]$ diff -u <(grep -A 11 "struct
ustctl_consumer_channel_attr {"
lttng-tools/src/bin/lttng-sessiond/lttng-ust-ctl.h) <(grep -A 10 "struct
ustctl_consumer_channel_attr {" lttng-ust/include/lttng/ust-ctl.h)
dev/fd/62 renamed from dev/fd/63 +0 -1
@@ -8,5 +8,4 @@
 	enum lttng_ust_output output;		/* splice, mmap */
 	uint32_t chan_id;			/* channel ID */
 	unsigned char uuid[LTTNG_UST_UUID_LEN]; /* Trace session unique ID */
-	int64_t blocking_timeout;			/* Blocking timeout (usec) */
 } LTTNG_PACKED;

My configure commands (followed by "make install"):

Quoted 8 lines
Quoted 6 lines
[sboisvert@GT480:userspace-rcu]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0
[sboisvert@GT480:lttng-ust]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4
LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib

I think you need a lttng-ust which is more recent than 2.9.4.

Got it.

I used the most recent git tag for lttng-ust: v2.9.4

Quoted 42 lines

Thanks,

Mathieu

Quoted 36 lines
[sboisvert@GT480:lttng-tools]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-tools/b14f53d4
LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/lib
CPPFLAGS=-I/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4/include

Thanks

Quoted 14 lines

Signed-off-by: Gabriel-Andrew Pollo-Guilbert
<[email protected]>

---
 libringbuffer/ring_buffer_backend.c | 3 +++
 1 file changed, 3 insertions(+)
libringbuffer/ring_buffer_backend.c +0 -0
index a0ef7446..431b8eae 100644

libringbuffer/ring_buffer_backend.c int channel_backend_init(struct channel_backend *chanb, · line 349

Quoted 9 lines
libringbuffer/ring_buffer_backend.c +3 -0
@@ -341,6 +341,9 @@ int channel_backend_init(struct channel_backend *chanb,
 	shmsize += sizeof(struct commit_counters_hot) * num_subbuf;
 	shmsize += offset_align(shmsize, __alignof__(struct commit_counters_cold));
 	shmsize += sizeof(struct commit_counters_cold) * num_subbuf;
+	/* Sampled timestamp end */
+	shmsize += offset_align(shmsize, __alignof__(uint64_t));
+	shmsize += sizeof(uint64_t) * num_subbuf;
 
 	if (config->alloc == RING_BUFFER_ALLOC_PER_CPU) {
 		struct lttng_ust_lib_ring_buffer *buf;

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw
Quoted 1 line · ----- On May 10, 2019, at 3:16 PM, Sebastien Boisvert [email protected] wrote:

On 2019-05-10 2:00 p.m., Mathieu Desnoyers wrote:

[...]

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
[sboisvert@GT480:lttng-ust]$ ./configure
--prefix=/home/sboisvert/open-source/lttng.org/Build/lttng-ust/v2.9.4
LDFLAGS=-L/home/sboisvert/open-source/lttng.org/Build/userspace-rcu/v0.11.0/lib

I think you need a lttng-ust which is more recent than 2.9.4.

Got it.

I used the most recent git tag for lttng-ust: v2.9.4

The most recent git tag stable version for lttng-ust is actually 2.10.3.

The patch in this thread addresses an issue recently introduced only in the master branch and the stable-2.11 branch (no release tags yet).

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com