Skip to contentSkip to conversation

UST app and lttng-tools compatibility

DISCUSSION Mathieu Desnoyers 3 participants 7 messages Sep 29, 2012 – Oct 1, 2012 6 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 18 lines · * Francis Giraldeau ([email protected]) wrote:

Hi,

I wanted to share my lttng-ust 2.1 update experience, maybe it will save time for others.

I updated lttng-ust recently. After this change, the app would not produce a trace anymore. No error message is displayed by the traced app to indicate that something is wrong. Even when setting LTTNG_DEBUG_UST to the app's environment variable, there is no error message. The debug output suggests that probes are registered and everything is fine, while it's not.

By running lttng-sessiond with -vvv --verbose-consumer, I finally got this message:

DEBUG2: UST app PID 8112 is not compatible with major version 3 (supporting <= 2) [in ust_app_validate_version() at ust-app.c:2633]

Quoted 6 lines

Updating lttng-tools to 2.1 solved the issue. Seems that it's mandatory to update lttng-tools to support latest lttng-ust. It may be obvious for developers, but it should be clear for users that they must upgrade both.

IMHO, It would be nice if the app side log could tell if the session/consumer refused the registration.

I agree we should do better.

Regarding lttng-tools, I think changing this DBG2 message to a WARN message would help, so sessiond would show the warning, except in the case where it is started with "-d".

On the application side, this is a bit tricky. It has no way to find out that it has been rejected by the sessiond. The application registers at startup, and then the sessiond keeps the connexion active, but flags it as incompatible internally. The reason we do that is because we don't want the application to retry endlessly.

Moreover, I cannot change the code for the existing 2.0 UST libs, so adding a new message is not possible.

One thing we have in mind for 2.2 or 2.3 is to add syslog support within the sessiond. This would provide a nice centralized place to look at those logs.

Thoughts ?

Thanks,

Mathieu

Quoted 12 lines

Cheers,

Francis Giraldeau

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): 20120929182800.GA19994@Krystal

#3 lore raw
Quoted 23 lines · Mathieu Desnoyers
Quoted 22 lines · * Francis Giraldeau ([email protected]) wrote:

Hi,

I wanted to share my lttng-ust 2.1 update experience, maybe it will save time for others.

I updated lttng-ust recently. After this change, the app would not produce a trace anymore. No error message is displayed by the traced app to indicate that something is wrong. Even when setting LTTNG_DEBUG_UST to the app's environment variable, there is no error message. The debug output suggests that probes are registered and everything is fine, while it's not.

By running lttng-sessiond with -vvv --verbose-consumer, I finally got this message:

DEBUG2: UST app PID 8112 is not compatible with major version 3 (supporting <= 2) [in ust_app_validate_version() at ust-app.c:2633]

Updating lttng-tools to 2.1 solved the issue. Seems that it's mandatory to update lttng-tools to support latest lttng-ust. It may be obvious for developers, but it should be clear for users that they must upgrade both.

Quoted 8 lines
Quoted 2 lines

IMHO, It would be nice if the app side log could tell if the session/consumer refused the registration.

I agree we should do better.

Regarding lttng-tools, I think changing this DBG2 message to a WARN message would help, so sessiond would show the warning, except in the case where it is started with "-d".

Excellent idea.

Quoted 5 lines

On the application side, this is a bit tricky. It has no way to find out that it has been rejected by the sessiond. The application registers at startup, and then the sessiond keeps the connexion active, but flags it as incompatible internally. The reason we do that is because we don't want the application to retry endlessly.

Could the registration process block until the sessiond returns some status? I understand that in commercial setup, the application should not be prevented to start and run normally if tracing is not available or misconfigured. But for developers, some env var like "LTTNG_ABORT_ON_ERROR" could help to diagnose this king of problem.

Quoted 2 lines

Moreover, I cannot change the code for the existing 2.0 UST libs, so adding a new message is not possible.

Of course! ;)

Quoted 3 lines

One thing we have in mind for 2.2 or 2.3 is to add syslog support within the sessiond. This would provide a nice centralized place to look at those logs.

I think it would be great. It's a bit less hidden, but certainly address concerns for production, and still can be used by developers. An error message is better than no one! ;)

Cheers,

Francis

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 27 lines · * Francis Giraldeau ([email protected]) wrote:
Quoted 26 lines · Mathieu Desnoyers
Quoted 24 lines · * Francis Giraldeau ([email protected]) wrote:

Hi,

I wanted to share my lttng-ust 2.1 update experience, maybe it will save time for others.

I updated lttng-ust recently. After this change, the app would not produce a trace anymore. No error message is displayed by the traced app to indicate that something is wrong. Even when setting LTTNG_DEBUG_UST to the app's environment variable, there is no error message. The debug output suggests that probes are registered and everything is fine, while it's not.

By running lttng-sessiond with -vvv --verbose-consumer, I finally got this message:

DEBUG2: UST app PID 8112 is not compatible with major version 3 (supporting <= 2) [in ust_app_validate_version() at ust-app.c:2633]

Updating lttng-tools to 2.1 solved the issue. Seems that it's mandatory to update lttng-tools to support latest lttng-ust. It may be obvious for developers, but it should be clear for users that they must upgrade both.

IMHO, It would be nice if the app side log could tell if the session/consumer refused the registration.

Quoted 7 lines
Quoted 5 lines

I agree we should do better.

Regarding lttng-tools, I think changing this DBG2 message to a WARN message would help, so sessiond would show the warning, except in the case where it is started with "-d".

Excellent idea.

David, can you make this change ?

Quoted 9 lines
Quoted 6 lines

On the application side, this is a bit tricky. It has no way to find out that it has been rejected by the sessiond. The application registers at startup, and then the sessiond keeps the connexion active, but flags it as incompatible internally. The reason we do that is because we don't want the application to retry endlessly.

Could the registration process block until the sessiond returns some status?

The session is not "returning" anything to the application. The application is registering to the sessiond, and all the sessiond can do is to keep the socket alive or close it.

Even if we added a "command" that the sessiond could use to tell the application it has a wrong version, that would not be 2.0 material, and the application wouldn't know about it. Moreover, given we have changed the communication protocol, not sure we would like to have this extra command as entirely fixed for now, as it would be part of the communication protocol (which is not the case for the initial app registration, with is part of a lower level protocol).

Quoted 3 lines

I understand that in commercial setup, the application should not be prevented to start and run normally if tracing is not available or misconfigured.

By default, we only block the application for up to 3 seconds at registration. I think that if we start the application with LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block forever (see man lttng-ust).

Thanks!

Mathieu

Quoted 22 lines

But for developers, some env var like "LTTNG_ABORT_ON_ERROR" could help to diagnose this king of problem.

Quoted 2 lines

Moreover, I cannot change the code for the existing 2.0 UST libs, so adding a new message is not possible.

Of course! ;)

Quoted 3 lines

One thing we have in mind for 2.2 or 2.3 is to add syslog support within the sessiond. This would provide a nice centralized place to look at those logs.

I think it would be great. It's a bit less hidden, but certainly address concerns for production, and still can be used by developers. An error message is better than no one! ;)

Cheers,

Francis

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): 20121001145619.GB13423@Krystal

#7 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Mathieu Desnoyers:

Quoted 33 lines · * Francis Giraldeau ([email protected]) wrote:
Quoted 32 lines · Mathieu Desnoyers
Quoted 31 lines · * Francis Giraldeau ([email protected]) wrote:
Quoted 20 lines

Hi,

I wanted to share my lttng-ust 2.1 update experience, maybe it will save time for others.

I updated lttng-ust recently. After this change, the app would not produce a trace anymore. No error message is displayed by the traced app to indicate that something is wrong. Even when setting LTTNG_DEBUG_UST to the app's environment variable, there is no error message. The debug output suggests that probes are registered and everything is fine, while it's not.

By running lttng-sessiond with -vvv --verbose-consumer, I finally got this message:

DEBUG2: UST app PID 8112 is not compatible with major version 3 (supporting <= 2) [in ust_app_validate_version() at ust-app.c:2633]

Quoted 7 lines

Updating lttng-tools to 2.1 solved the issue. Seems that it's mandatory to update lttng-tools to support latest lttng-ust. It may be obvious for developers, but it should be clear for users that they must upgrade both.

IMHO, It would be nice if the app side log could tell if the session/consumer refused the registration.

I agree we should do better.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Regarding lttng-tools, I think changing this DBG2 message to a WARN message would help, so sessiond would show the warning, except in the case where it is started with "-d".

Excellent idea.

David, can you make this change ?

Yes!

David

Quoted 28 lines
Quoted 10 lines
Quoted 7 lines

On the application side, this is a bit tricky. It has no way to find out that it has been rejected by the sessiond. The application registers at startup, and then the sessiond keeps the connexion active, but flags it as incompatible internally. The reason we do that is because we don't want the application to retry endlessly.

Could the registration process block until the sessiond returns some status?

The session is not "returning" anything to the application. The application is registering to the sessiond, and all the sessiond can do is to keep the socket alive or close it.

Even if we added a "command" that the sessiond could use to tell the application it has a wrong version, that would not be 2.0 material, and the application wouldn't know about it. Moreover, given we have changed the communication protocol, not sure we would like to have this extra command as entirely fixed for now, as it would be part of the communication protocol (which is not the case for the initial app registration, with is part of a lower level protocol).

Quoted 3 lines

I understand that in commercial setup, the application should not be prevented to start and run normally if tracing is not available or misconfigured.

Quoted 34 lines

By default, we only block the application for up to 3 seconds at registration. I think that if we start the application with LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block forever (see man lttng-ust).

Thanks!

Mathieu

Quoted 22 lines

But for developers, some env var like "LTTNG_ABORT_ON_ERROR" could help to diagnose this king of problem.

Quoted 2 lines

Moreover, I cannot change the code for the existing 2.0 UST libs, so adding a new message is not possible.

Of course! ;)

Quoted 3 lines

One thing we have in mind for 2.2 or 2.3 is to add syslog support within the sessiond. This would provide a nice centralized place to look at those logs.

I think it would be great. It's a bit less hidden, but certainly address concerns for production, and still can be used by developers. An error message is better than no one! ;)

Cheers,

Francis

-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQabGBAAoJEELoaioR9I024c0IAJ8i7hIUthhQdbeESSV6T65L VgxPYtjUr2aKbOaeInS1FXiW39Gpg5w1Xut9gTaAmlxP+Dsgs1TXLVx0YJFCsqi0 QYrkPLbpdafmCjbXKpWa6G5hNCFa684MorL6hCqhSOQEBXD0N/aKGQmUVz+pxGXp xR/l/hwHadm4v6VH75N5dr5JvGNHQbcZDYz0gzokemyo1edw0NGa3jQGe0WYcVcp /K6hybbgRvkZObMB2hhTESkLwAFk/OO4WEP6zTZCSPLCTAdiGLaWNQQ9PdxxkcT+ crT83mDSW8K1yQkZDXOk5U9XSmdBLjBuQW6094EaCF9Nb5K6d+y2U01dii4eIU4= =6aHd
-----END PGP SIGNATURE-----

#8 lore raw
In reply to a message not in the archive
Quoted 1 line · * Mathieu Desnoyers ([email protected]) wrote:

* Francis Giraldeau ([email protected]) wrote:

[...]

Quoted 15 lines
Quoted 2 lines

Could the registration process block until the sessiond returns some status?

The session is not "returning" anything to the application. The application is registering to the sessiond, and all the sessiond can do is to keep the socket alive or close it.

Even if we added a "command" that the sessiond could use to tell the application it has a wrong version, that would not be 2.0 material, and the application wouldn't know about it. Moreover, given we have changed the communication protocol, not sure we would like to have this extra command as entirely fixed for now, as it would be part of the communication protocol (which is not the case for the initial app registration, with is part of a lower level protocol).

Quoted 8 lines
Quoted 3 lines

I understand that in commercial setup, the application should not be prevented to start and run normally if tracing is not available or misconfigured.

By default, we only block the application for up to 3 seconds at registration. I think that if we start the application with LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block forever (see man lttng-ust).

Let me take this last part back. That was a pre-morning-coffee statement. ;) The application will receive a "registration done" message when the version is found to be incompatible. Therefore, it will never wait for the incompatible sessiond.

One thing we could do to make transition smoother between future versions would be to add a new command to let the sessiond send its own version info to the application. In this command's handler, the application could show a warning on stderr. This could not apply to 2.0, but we could do it starting from 2.1.

Thoughts ?

Thanks,

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): 20121001152435.GA13628@Krystal

#10 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Mathieu Desnoyers:

Quoted 33 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 1 line · * Francis Giraldeau ([email protected]) wrote:

[...]

Quoted 16 lines
Quoted 2 lines

Could the registration process block until the sessiond returns some status?

The session is not "returning" anything to the application. The application is registering to the sessiond, and all the sessiond can do is to keep the socket alive or close it.

Even if we added a "command" that the sessiond could use to tell the application it has a wrong version, that would not be 2.0 material, and the application wouldn't know about it. Moreover, given we have changed the communication protocol, not sure we would like to have this extra command as entirely fixed for now, as it would be part of the communication protocol (which is not the case for the initial app registration, with is part of a lower level protocol).

Quoted 8 lines
Quoted 3 lines

I understand that in commercial setup, the application should not be prevented to start and run normally if tracing is not available or misconfigured.

By default, we only block the application for up to 3 seconds at registration. I think that if we start the application with LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block forever (see man lttng-ust).

Let me take this last part back. That was a pre-morning-coffee statement. ;) The application will receive a "registration done" message when the version is found to be incompatible. Therefore, it will never wait for the incompatible sessiond.

Quoted 7 lines

One thing we could do to make transition smoother between future versions would be to add a new command to let the sessiond send its own version info to the application. In this command's handler, the application could show a warning on stderr. This could not apply to 2.0, but we could do it starting from 2.1.

Thoughts ?

Why sending version infos instead of a "incompatible" command ?

David

Quoted 6 lines

Thanks,

Mathieu

-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQabZVAAoJEELoaioR9I02RYcIALJ7ySGiU6nKDD4F4ffAEIFr ZSQC/1mhr8x66TfTdB9eGaAxA6qmESEBpg6bjyWSmWCn1dI1i5lOeXhxszM1Z8Oz q6DKSSlRVRZFrCaZBf3Xy7QNbxTdUkwrcCjd8Y9Ffu1BswXIzzUhsYSHbJRQAHVF QAuNFy5AeHRlE55H94Gs/ydfiQb3jqVIalVnG5LmeDPyO58sdA+RAWkwaJypgL8e 5m6r3K0IK5ufSIjWWb6G8vmf3HFV3zzdEwGpkEyaVRZjUfDBekRnJ27Bb2XqJUxC /xe2S519VLnsCWE+j1pfN5BcchI0TKUOeTcaLzzaf/VdkHdlOmZlifSZ6ubRusU= =OVEY
-----END PGP SIGNATURE-----

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 45 lines · * David Goulet ([email protected]) wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Mathieu Desnoyers:

Quoted 39 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 1 line · * Francis Giraldeau ([email protected]) wrote:

[...]

Quoted 16 lines
Quoted 2 lines

Could the registration process block until the sessiond returns some status?

The session is not "returning" anything to the application. The application is registering to the sessiond, and all the sessiond can do is to keep the socket alive or close it.

Even if we added a "command" that the sessiond could use to tell the application it has a wrong version, that would not be 2.0 material, and the application wouldn't know about it. Moreover, given we have changed the communication protocol, not sure we would like to have this extra command as entirely fixed for now, as it would be part of the communication protocol (which is not the case for the initial app registration, with is part of a lower level protocol).

Quoted 8 lines
Quoted 3 lines

I understand that in commercial setup, the application should not be prevented to start and run normally if tracing is not available or misconfigured.

By default, we only block the application for up to 3 seconds at registration. I think that if we start the application with LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block forever (see man lttng-ust).

Let me take this last part back. That was a pre-morning-coffee statement. ;) The application will receive a "registration done" message when the version is found to be incompatible. Therefore, it will never wait for the incompatible sessiond.

One thing we could do to make transition smoother between future versions would be to add a new command to let the sessiond send its own version info to the application. In this command's handler, the application could show a warning on stderr. This could not apply to 2.0, but we could do it starting from 2.1.

Quoted 3 lines
Quoted 1 line

Thoughts ?

Why sending version infos instead of a "incompatible" command ?

It could very well be an "incompatible" command, but it would need to contain the version info, so the application can print the versions.

The main difference between sending version info (all the time) and the "incompatible" command is that the version info would be sent all the time. It could provide better debugging logs in the app when using LTTNG_UST_DEBUG=1, but would require one extra message at app registration. Not sure which is best.

Thoughts ?

Thanks,

Mathieu

Quoted 19 lines

David

Quoted 6 lines

Thanks,

Mathieu

-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQabZVAAoJEELoaioR9I02RYcIALJ7ySGiU6nKDD4F4ffAEIFr ZSQC/1mhr8x66TfTdB9eGaAxA6qmESEBpg6bjyWSmWCn1dI1i5lOeXhxszM1Z8Oz q6DKSSlRVRZFrCaZBf3Xy7QNbxTdUkwrcCjd8Y9Ffu1BswXIzzUhsYSHbJRQAHVF QAuNFy5AeHRlE55H94Gs/ydfiQb3jqVIalVnG5LmeDPyO58sdA+RAWkwaJypgL8e 5m6r3K0IK5ufSIjWWb6G8vmf3HFV3zzdEwGpkEyaVRZjUfDBekRnJ27Bb2XqJUxC /xe2S519VLnsCWE+j1pfN5BcchI0TKUOeTcaLzzaf/VdkHdlOmZlifSZ6ubRusU= =OVEY
-----END PGP SIGNATURE-----

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com