Skip to contentSkip to conversation

Machine Interface

DISCUSSION Woegerer, Paul 4 participants 7 messages Feb 6, 2014 6 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 13 lines · Jonathan Rajotte

Hello all,

After speaking with Michel Dagenais, Geneviève Bastien, folks over at EfficiOs and Ericsson, a machine interface for lttng-tool would be a nice feature to have. Olivier Cotte and me will be working on MI for the
next few weeks.

The main objective would be to reduce parsing complexity
in control tools.
ex:Regex used in TMF control service. (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)

Multiples solutions come to mind:

Review

Quoted 3 lines
(quoted diff) +0 -3
-XML
-JSON
-YAML

YAML sounds very attractive in this context.

It's easily parseable even from a shell script. If you need to automate LTTng tracing tasks on a small embedded target busybox is all you have!

Best,
Paul

Quoted 21 lines

-Custom syntax (like GDB Machine interface* as proposed by Alexandre
Montplaisir)

XML/JSON/YAML promote extensibility and ease of development for new tools dependant on lttng-tools by being established standards.
On the other hand, they add dependencies to lttng-tools where a custom syntax would not.

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

We are focusing on output right now but we may eventually add a 'mi' command for input if necessary.

What do you think ?

Thanks

*https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html

Signature · 3 lines
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 13 lines · Paul
Quoted 12 lines · Jonathan Rajotte

Hello all,

After speaking with Michel Dagenais, Geneviève Bastien, folks over at EfficiOs and Ericsson, a machine interface for lttng-tool would be a nice feature to have. Olivier Cotte and me will be working on MI for the
next few weeks.

The main objective would be to reduce parsing complexity
in control tools.
ex:Regex used in TMF control service. (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)

Quoted 6 lines

Review

Quoted 4 lines

Multiples solutions come to mind:

(quoted diff) +0 -3
-XML
-JSON
-YAML

YAML sounds very attractive in this context.

I am also fond of the yaml syntax. It is very easy to read/write by a human. Not sure how much this matters though, since is a machine interface.

Quoted 2 lines

It's easily parseable even from a shell script. If you need to automate LTTng tracing tasks on a small embedded target busybox is all you have!

This point is relevant for me, since I actually intend to use this to enhance bash completion, which is done with a bash script.

I never thought I would say that one day, but I think that XML could have an advantage here. Parsing yaml with regexp (such as in a shell script) is easier than XML, but it's still not bulletproof. Consider this eventual yaml structure I just made up for the output of the session list.

sessions:
- name: my_session
domain: kernel
events:
- name: sched_switch
filter: prev_pid == 1234
- name: sched_migrate_task
otherproperty: "something"
- name: another_session
domain: ust
events:
- name: event_hello
- name: event_bonjour
loglevel: 3

If you want to get the list of sessions, you'll probably want to grep the "name: " entries, but then event names would match as well. Of course, names could be different, or you could grep for the appropriate amount of spaces/tabs at the beginning of the line. But then it also depends if the name dictionary entry is the first, then it will have a "-" in front... It's possible, but it will also be very easy to break scripts.

With XML, there are tools (xmlstarlet, xmllint, probably others) that allow you to evaluate XPath expressions and thus not depend of the formatting. For example, an equivalent XML schema could be

<lttng>
<session name="my_session" domain="kernel">
<event name="sched_switch" filter="prev_pid == 1234" /> <event name="sched_migrate_task" otherproperty="something" />
</session>
<session name="another_session" domain="ust">
<event name="event_hello" />
<event name="event_bonjour" loglevel="3" />
</session>
</lttng>

To get a plain list of sessions, you could then do

$ lttng list --mi | xmlstarlet sel -T -t -v "/lttng/session/@name" -n
my_session
another_session

Maybe such tools exist for yaml?

Also, I understand your point for embedded devices though, where space is limited and you don't want to load an XML lib/tool just for that. If the design of the MI is somewhat modular though, it could be possible to support more than one output format, with the different options available at configure time.

Quoted 5 lines

Best,
Paul

Quoted 2 lines

-Custom syntax (like GDB Machine interface* as proposed by Alexandre
Montplaisir)

I would suggest not going for a custom syntax... that sounds like more work for more chances of bugs.

Quoted 4 lines
Quoted 4 lines

XML/JSON/YAML promote extensibility and ease of development for new tools dependant on lttng-tools by being established standards.
On the other hand, they add dependencies to lttng-tools where a custom syntax would not.

I would suggest making the dependency optional. If you don't have the lib, or don't want to depend on the lib, have a --disable-something at configure time.

Quoted 25 lines
Quoted 6 lines

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

We are focusing on output right now but we may eventually add a 'mi' command for input if necessary.

Quoted 7 lines

What do you think ?

Thanks

*https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html

--
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CAFXXi0=enX3ERpH1Q3R-P4ii2wU7TE2wX_NEcXYF0iEXKCW6yA@mail.gmail.com

#5 lore raw

Hi Simon,

Quoted 19 lines · Simon Marchi

script) is easier than XML, but it's still not bulletproof. Consider this eventual yaml structure I just made up for the output of the session list.

sessions:
- name: my_session
domain: kernel
events:
- name: sched_switch
filter: prev_pid == 1234
- name: sched_migrate_task
otherproperty: "something"
- name: another_session
domain: ust
events:
- name: event_hello
- name: event_bonjour
loglevel: 3

Quoted 7 lines

If you want to get the list of sessions, you'll probably want to grep the "name: " entries, but then event names would match as well. Of course, names could be different, or you could grep for the appropriate amount of spaces/tabs at the beginning of the line. But then it also depends if the name dictionary entry is the first, then it will have a "-" in front... It's possible, but it will also be very easy to break scripts.

The following simple script below would e.g. give you the list of sessions for you example above. To demonstrate the flexibility it also generates the event list for session my_session. This is just a quick prototype but it should make clear that YAML is considerably easier to parse from a shell script than any other option (and also reasonably robust). Also notice that no external helpers (like grep) are needed.

list_of_sessions=''
current_session=''
my_session_events=''

while IFS= read -r; do

	line=$REPLY
	case "${line}" in 
		"      - "*)
			;&
		"        "*)
			L2_key=${line:8}
			L2_key=${L2_key%:*}
			let vpos=2+8+${#L2_key}
			L2_value=${line:${vpos}}
			;;
		"  - "*)
			# New L1 list elem -> reset L2 state
			L2_key=''
			L2_value=''
			;&
		"    "*)
			L1_key=${line:4}
			L1_key=${L1_key%:*}
			let vpos=2+4+${#L1_key}
			L1_value=${line:${vpos}}
			;;
	esac
	echo "($L1_key,$L1_value), ($L2_key,$L2_value)"

	# Collect the session names
	if [ "$L1_key" = "name" ]; then
		current_session="$L1_value"
		list_of_sessions="$list_of_sessions $current_session"
	fi
	# Which events are in session my_session ?
	if [ "$current_session" = "my_session" ] && [ "$L1_key" = "events" ] && [ "$L2_key" = "name" ]; then
		my_session_events="$my_session_events $L2_value"
	fi

done
echo
echo List of sessions: $list_of_sessions
echo my_session events: $my_session_events
echo

Thanks,
Paul

Signature · 3 lines
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 30 lines · Paul

Hi Simon,

Quoted 19 lines · Simon Marchi

script) is easier than XML, but it's still not bulletproof. Consider this eventual yaml structure I just made up for the output of the session list.

sessions:
- name: my_session
domain: kernel
events:
- name: sched_switch
filter: prev_pid == 1234
- name: sched_migrate_task
otherproperty: "something"
- name: another_session
domain: ust
events:
- name: event_hello
- name: event_bonjour
loglevel: 3

Quoted 7 lines

If you want to get the list of sessions, you'll probably want to grep the "name: " entries, but then event names would match as well. Of course, names could be different, or you could grep for the appropriate amount of spaces/tabs at the beginning of the line. But then it also depends if the name dictionary entry is the first, then it will have a "-" in front... It's possible, but it will also be very easy to break scripts.

Quoted 7 lines

The following simple script below would e.g. give you the list of sessions for you example above. To demonstrate the flexibility it also generates the event list for session my_session. This is just a quick prototype but it should make clear that YAML is considerably easier to parse from a shell script than any other option (and also reasonably robust). Also notice that no external helpers (like grep) are needed.

This is impressive scripting! Although I can see this growing a little bit out of control for more complex yaml trees. I agree with you that yaml is the easiest of them to parse in a pure shell script since the formatting is more restricted. But if you're on a machine that can take it, using a solution based on an XPath tool (or equivalent) will make it so much more maintainable.

Quoted 52 lines

list_of_sessions=''
current_session=''
my_session_events=''

while IFS= read -r; do

        line=$REPLY
        case "${line}" in
                "      - "*)
                        ;&
                "        "*)
                        L2_key=${line:8}
                        L2_key=${L2_key%:*}
                        let vpos=2+8+${#L2_key}
                        L2_value=${line:${vpos}}
                        ;;
                "  - "*)
                        # New L1 list elem -> reset L2 state
                        L2_key=''
                        L2_value=''
                        ;&
                "    "*)
                        L1_key=${line:4}
                        L1_key=${L1_key%:*}
                        let vpos=2+4+${#L1_key}
                        L1_value=${line:${vpos}}
                        ;;
        esac
        echo "($L1_key,$L1_value), ($L2_key,$L2_value)"

        # Collect the session names
        if [ "$L1_key" = "name" ]; then
                current_session="$L1_value"
                list_of_sessions="$list_of_sessions $current_session"
        fi
        # Which events are in session my_session ?
        if [ "$current_session" = "my_session" ] && [ "$L1_key" = "events" ] && [ "$L2_key" = "name" ]; then
                my_session_events="$my_session_events $L2_value"
        fi

done
echo
echo List of sessions: $list_of_sessions
echo my_session events: $my_session_events
echo

Thanks,
Paul

--
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

#8 lore raw
In reply to a message not in the archive

Hi Jonathan,

Review

Quoted 6 lines

Multiples solutions come to mind:

(quoted diff) +0 -4
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
                                Montplaisir)

I would advocate for the use of XML for the machine interface. Two reasons come to mind:

1 - The expression of hierarchy (session, events, etc.) is much easier with XML.

2 - In a recent post [1], it was decided that the configuration load and restore (for sessions) would also use XML. This will introduce a new dependency on libxml. If we were to choose yet another format for the machine interface, we would probably need to introduce another dependency to some library.

Quoted 2 lines

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

Perhaps adding --xml option to the relevant command? This would be similar to svn machine interface [2].

Thanks,

Christian

[1] - http://lists.lttng.org/pipermail/lttng-dev/2013-December/022068.html
[2] - http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn.sw

Quoted 43 lines · Jonathan Rajotte

Hello all,

After speaking with Michel Dagenais, Geneviève Bastien, folks over at EfficiOs and Ericsson, a machine interface for lttng-tool would be a nice feature to have. Olivier Cotte and me will be working on MI for the
next few weeks.

The main objective would be to reduce parsing complexity
in control tools.
ex:Regex used in TMF control service. (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)

Multiples solutions come to mind:

(quoted diff) +0 -4
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
                                Montplaisir)

XML/JSON/YAML promote extensibility and ease of development for new tools dependant on lttng-tools by being established standards.
On the other hand, they add dependencies to lttng-tools where a custom syntax would not.

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

We are focusing on output right now but we may eventually add a 'mi' command for input if necessary.

What do you think ?

Thanks

*https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html

--
Jonathan Rajotte Julien
Chargé de laboratoire INF1995
Membre MD6
Polytechnique Montréal

Quoted 4 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CAGDH53mHYJg-uVd+4Cv=X6ST5Ly+nmE_BQSVmdN3Hb4k7ZKeSw@mail.gmail.com

#10 lore raw
Quoted 12 lines · Christian Babeux

Hi Jonathan,

Review

Quoted 6 lines

Multiples solutions come to mind:

(quoted diff) +0 -4
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
                                Montplaisir)

I would advocate for the use of XML for the machine interface. Two reasons come to mind:

Quoted 7 lines

1 - The expression of hierarchy (session, events, etc.) is much easier with XML.

2 - In a recent post [1], it was decided that the configuration load and restore (for sessions) would also use XML. This will introduce a new dependency on libxml. If we were to choose yet another format for the machine interface, we would probably need to introduce another dependency to some library.

Good point
+1 for XML

Quoted 6 lines
Quoted 3 lines

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

Perhaps adding --xml option to the relevant command? This would be similar to svn machine interface [2].

How about --machine-interface <output-type>? By default <output-type> would be the chosen format.

This open opportunities for other MI type and have an unified syntax.

--xml option is clear and efficient.We could add an option for each type (--json,--yaml) if needed.

Quoted 57 lines

Thanks,

Christian

[1] - http://lists.lttng.org/pipermail/lttng-dev/2013-December/022068.html
[2] - http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn.sw

Quoted 47 lines · Jonathan Rajotte

Hello all,

After speaking with Michel Dagenais, Geneviève Bastien, folks over at EfficiOs and Ericsson, a machine interface for lttng-tool would be a nice feature to have. Olivier Cotte and me will be working on MI for the
next few weeks.

The main objective would be to reduce parsing complexity
in control tools.
ex:Regex used in TMF control service. (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)

Multiples solutions come to mind:

(quoted diff) +0 -4
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
                                Montplaisir)

XML/JSON/YAML promote extensibility and ease of development for new tools dependant on lttng-tools by being established standards.
On the other hand, they add dependencies to lttng-tools where a custom syntax would not.

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

We are focusing on output right now but we may eventually add a 'mi' command for input if necessary.

What do you think ?

Thanks

*https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html

--
Jonathan Rajotte Julien
Chargé de laboratoire INF1995
Membre MD6
Polytechnique Montréal

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 4 lines
Jonathan Rajotte Juliana
Chargé de laboratoire INF1995
Membre MD6
Polytechnique Montréal

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 7 lines

How about --machine-interface <output-type>? By default <output-type> would be the chosen format.

This open opportunities for other MI type and have an unified syntax.

--xml option is clear and efficient.We could add an option for each type (--json,--yaml) if needed.

IMO, supporting multiple output formats would be a tad bit overkill. Moreover, the issue of bringing in additional library dependencies to support multiple output format remains.

Thanks,

Christian

Quoted 92 lines · Jonathan Rajotte
Quoted 12 lines · Christian Babeux

Hi Jonathan,

Review

Quoted 6 lines

Multiples solutions come to mind:

(quoted diff) +0 -4
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
                                Montplaisir)

I would advocate for the use of XML for the machine interface. Two reasons come to mind:

Quoted 7 lines

1 - The expression of hierarchy (session, events, etc.) is much easier with XML.

2 - In a recent post [1], it was decided that the configuration load and restore (for sessions) would also use XML. This will introduce a new dependency on libxml. If we were to choose yet another format for the machine interface, we would probably need to introduce another dependency to some library.

Good point
+1 for XML

Quoted 6 lines
Quoted 3 lines

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

Perhaps adding --xml option to the relevant command? This would be similar to svn machine interface [2].

How about --machine-interface <output-type>? By default <output-type> would be the chosen format.

This open opportunities for other MI type and have an unified syntax.

--xml option is clear and efficient.We could add an option for each type (--json,--yaml) if needed.

Quoted 53 lines

Thanks,

Christian

[1] - http://lists.lttng.org/pipermail/lttng-dev/2013-December/022068.html
[2] - http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn.sw

Quoted 43 lines · Jonathan Rajotte

Hello all,

After speaking with Michel Dagenais, Geneviève Bastien, folks over at EfficiOs and Ericsson, a machine interface for lttng-tool would be a nice feature to have. Olivier Cotte and me will be working on MI for the
next few weeks.

The main objective would be to reduce parsing complexity
in control tools.
ex:Regex used in TMF control service. (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)

Multiples solutions come to mind:

(quoted diff) +0 -4
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
                                Montplaisir)

XML/JSON/YAML promote extensibility and ease of development for new tools dependant on lttng-tools by being established standards.
On the other hand, they add dependencies to lttng-tools where a custom syntax would not.

Machine interface output would be generated only with --mi option and could contain additional information than regular output.

We are focusing on output right now but we may eventually add a 'mi' command for input if necessary.

What do you think ?

Thanks

*https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html

--
Jonathan Rajotte Julien
Chargé de laboratoire INF1995
Membre MD6
Polytechnique Montréal

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Jonathan Rajotte Juliana
Chargé de laboratoire INF1995
Membre MD6
Polytechnique Montréal

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev